Unreal Agent Harness

Drive Unreal Engine 5.8 with an AI agent (Claude) — build real 3D scenes, generate cities, and play them — through the official Unreal MCP plugin.

This is the harness behind the video: the agent gets hands (drive the editor), eyes (capture + decode the viewport), knowledge (the docs here), and a QA loop (see → act → check → fix). You point Claude Code at Unreal and it builds.

⚠️ The experimental AI tooling is real but raw — budget time for plumbing. Read docs/00-GETTING-STARTED.md first. It's the difference between "this is magic" and "why won't it connect."

What you can build with it (what's in the video)

• 🗽 Photoreal real-world cities — stream actual NYC (or anywhere) via Cesium Google 3D Tiles, flythrough-ready. → docs/cesium-for-unreal.md
• ✈️ A chase-cam aircraft over the city — import a real plane mesh, rig a third-person follow cam, fly it. → docs/plane-chase-pawn.md
• 🏙️ A procedural city that builds itself, live — PCG: shape → districts → blocks → buildings rise → roads, driven stage-by-stage on camera. → docs/PCG-GUIDE.md
• 🎮 Turn it into a game — drop a playable third-person character into a city (incl. Epic's City Sample) and roam/drive it. → docs/CITY-SAMPLE-PLAYABLE.md · docs/UE-PLAYABLE-CHARACTER.md

Quickstart (≈15 min once UE is installed)

1. Install + set up — UE 5.8, the Metal toolchain, enable the MCP plugin + auto-start server, connect Claude Code. Full steps: docs/00-GETTING-STARTED.md.
2. Verify the connection — in Claude Code: SceneTools.get_current_level should return your level (not "Unable to connect").
3. First build — ask the agent to place a cube and capture it:
   • drive: mcp__unreal__call_tool → SceneTools.add_to_scene_from_asset
   • see: EditorAppToolset.CaptureViewport → then python3 ue_qa.py decode
4. Run the loop (below). Reach for the recipes for lighting, materials, PCG, characters.

How the agent sees and builds — the loop

CaptureViewport returns a multi-MB base64 PNG that floods agent context (raw HTTP to the MCP server also returns empty for big results — they stream on an async SSE channel). So the agent triggers captures through its own MCP tool layer (the runtime auto-saves big results to tool-results/*.txt, out of context), and ue_qa.py turns that blob into a small PNG + compact JSON the agent can cheaply read.

1. Act — build via mcp__unreal__call_tool (e.g. SceneTools.add_to_scene_from_asset).
2. Capture — EditorAppToolset.CaptureViewport { captureTransform, annotations, bShowUI:false }. captureTransform shoots from a pose without moving the user's camera; annotations overlays a coordinate grid + actor labels.
3. Decode — python3 ue_qa.py decode --name NAME → /tmp/ue_qa/NAME.png + NAME.json (camera pose + labeled actors w/ positions).
4. Read — Read the PNG; cat the JSON for spatial reasoning.
5. Correct — fix and repeat.

3-angle QA sweep per build step: top-down (layout/overlap) · eye-level (aesthetic) · player-eye (gameplay feel). Also: read errors via LogsToolset; check overlaps via SceneTools.find_actors(bounds) / trace_world.

python3 ue_qa.py decode [--name NAME] [--file PATH]   # newest (or given) capture -> PNG + JSON
python3 ue_qa.py latest                               # path of newest saved capture
python3 ue_qa.py refdiff REF.png SHOT.png [--name N]  # side-by-side intent-vs-result (needs imagemagick)

Two hard constraints:

• Always go through the agent's MCP tool layer for captures (raw HTTP returns empty — async SSE).
• One editor, one game thread → serialize every scene mutation. No concurrent builders. Read-only inspection can overlap, but never two ExecuteGraphInstance/mutations at once.

Documentation

Start at the docs index — everything is grouped there (Setup · MCP+QA loop · Scene/Lighting/Material recipes · PCG city · Playable character & City Sample · Asset pipeline · Troubleshooting).

Stuck? → docs/TROUBLESHOOTING.md (camera won't move, editor crashes, "Unable to connect", version mismatch, the boot hang…).

The build narrative / "how we got here, with opinions" → AGENTIC-GAMEDEV-GUIDE.md. The running session log → BUILD-LOG.md.

Repo layout

README.md                  ← you are here (front door + the loop)
AGENTIC-GAMEDEV-GUIDE.md   ← teaching narrative + opinions
BUILD-LOG.md               ← chronological session log
ue_qa.py                   ← capture decoder (blob → PNG + JSON), refdiff
blender_jobs.py            ← headless Blender modeling jobs (bpy)
towers_jobs.py             ← procedural skyscraper generator (bpy)
ue_launch.sh / ue_crashlog.sh  ← launch UE + pull crash logs
ue_remote/                 ← Python remote-exec helpers (Cesium rebase, etc.)
splat/                     ← Gaussian-splat tile tooling
docs/                      ← all guides + recipes (see docs/README.md)
assets/                    ← (git-ignored) CC0 + generated + downloaded meshes/textures

Prerequisites (short version)

macOS (Apple Silicon, M2+ for Nanite) or Windows · UE 5.8 · Claude Code · Python 3 (+ imagemagick for refdiff) · Blender (only for the modeling jobs). Full list + install order in docs/00-GETTING-STARTED.md.