Minecraft Building AI Agent

A player types !build a small gothic church in chat. The model designs the building, the harness turns that design into block-level detail, and the agent places the blocks in the running world over RCON.

The player can walk around the building and type !fix make this tower taller while looking at the tower. The agent reads the player's position and view direction and passes them to the model. The agent then replaces only the changed blocks in place.

LLMs can already lay out 3D massing, but a plain model produces flat walls with no detail. The agent places a harness between the model and the world. The harness has two parts. The first is a written set of a master builder's principles: odd widths, windows with depth, pillars as the skeleton, no flat walls. The second is a coded library of helpers such as window, corner_pillar, gable_roof and spire. The code of each helper reproduces block placements taken from a real builder's cathedral.

The model never writes raw block states. It writes a small script that calls these helpers. For a fix request, the model edits that script, and the agent applies only the resulting voxel diff to the world.

Both runs used the same photos and the same prompt. The plain model output a raw voxel dictionary with flat walls. With the harness, after three in-game fix turns, the church had depth, pillars, stepped arches and an odd-width footprint. Each model turn takes 1 to 4 minutes, and the world update takes seconds. This agent works inside Minecraft rather than in Slack or Teams.