SimWorld Studio

Vibe code the physical world.

Chat with an AI coding agent to build, simulate, and control 3D environments in Unreal Engine 5 — with embodied agent support, real-time pixel streaming, and a full data visualization stack.

Overview

SimWorld Studio is an AI-native 3D scene authoring and embodied agent testbed built on Unreal Engine 5.3. It connects Claude (via Claude Code CLI) to UE through a multi-protocol bridge, letting you:

• Build scenes by chat — describe a city block and watch it materialize in real time
• Control embodied agents — spawn, command, and observe AI agents navigating the scene
• Run multi-agent experiments — testbed with auto-PIE, trajectory recording, collision tracking
• Stream the viewport — Pixel Streaming delivers the live UE viewport to any browser

Step 1 — Set Up a Coding Agent

SimWorld Studio's chat is driven by a coding-agent CLI of your choice — Claude Code (default), Codex, Gemini CLI, or OpenCode. Set one up before installing Studio. It's the same 3 steps for any of them:

1. Install the CLI:

   Agent	Install
   Claude Code	npm i -g @anthropic-ai/claude-code
   Codex	npm i -g @openai/codex
   Gemini CLI	npm i -g @google/gemini-cli
   OpenCode	npm i -g opencode-ai

2. Log in once, in your terminal — run the CLI and sign in (claude, codex login, gemini, opencode auth login), or export the provider's API key. Do this as the same user that runs the Studio server; Studio reuses that login and never asks you for keys.

3. Pick it in the UI — choose the Agent and Model from the dropdowns in the top-left of the app.

_{Advanced: change the model menus in simworld_studio_workspace/web/server/coding-agents.json, or override a CLI path / default model with the *_BIN / *_MODEL env vars.}

Now install Studio for your platform:

Quick Start (Google Colab — no local GPU needed)

Run all cells in order. Setup takes ~5 minutes. Requires a free Colab GPU runtime and an Anthropic API key.

Quick Start (Linux — One-Command Install)

For end users on a Linux machine with an NVIDIA GPU. For internal multi-user shared servers, see Linux — Shared Server below.

Prerequisites

• OS: Linux (Ubuntu 20.04+ recommended)
• GPU: NVIDIA GPU with 8GB+ VRAM (tested on L40S, T4, A100)
• NVIDIA drivers: 525+ with Vulkan support
• Node.js: 18+
• Python: 3.9+
• Disk: ~40 GB free (15 GB download + 21 GB extracted)

1. Download the Minimal SimWorld Binary

# Download (~15 GB compressed, ~21 GB extracted)
wget -O SimWorld-Studio-Minimal.tar.gz \
    https://huggingface.co/datasets/SimWorld-AI/SimWorld-Studio/resolve/main/SimWorld-Studio-Minimal.tar.gz

tar xzf SimWorld-Studio-Minimal.tar.gz

2. Install SimWorld Studio

pip install git+https://github.com/SimWorld-AI/SimWorld-Studio.git#subdirectory=packaging
npm install -g @anthropic-ai/claude-code

3. Set up your coding agent

You did this in Step 1. The default is Claude Code — authenticate it now if you haven't:

Option A — API Key:

export ANTHROPIC_API_KEY="sk-ant-..."

Get your key at console.anthropic.com.

Option B — Claude Code Login (no API key needed):

claude

This opens a browser for OAuth login. If on a headless server, use the API key option instead.

4. Launch (one command)

simworld-studio start

This will:

• Detect your GPU and authenticate with Claude
• Launch Unreal Engine (headless)
• Wait for the engine to be ready
• Start the Studio web server
• Print the URL to open in your browser

On multi-GPU systems, it will ask which GPU to use (or pass --gpu INDEX). For remote servers, it auto-detects your IP and prints SSH tunnel instructions.

Options:

--gpu INDEX    GPU to use (auto-detected if omitted)
--port PORT    Web UI port (default: 3002)
--binary PATH  Path to SimWorld-Studio-Minimal directory

Try: "Set up the environment with a sunny sky, then build a small neighborhood with 4 houses and trees"

Quick Start (Windows — Local Dev)

Prerequisites

• Unreal Engine 5.3 (Epic Games Launcher)
• Node.js 18+
• Claude Code CLI: npm install -g @anthropic-ai/claude-code

1. Configure paths

Edit SimWorld-Studio.ps1 lines 46–47:

$UeRoot    = "C:\Program Files\Epic Games\UE_5.3"
$UeProject = "E:\UE\SimWorld-copy\SimWorld.uproject"

2. Launch (one command)

.\SimWorld-Studio.ps1

The script will:

1. Kill stale port processes
2. Sync Cirrus player.js and build the React frontend
3. Start Cirrus signalling server (Pixel Streaming)
4. Launch UE Editor with MCP + Pixel Streaming flags
5. Wait for MCP port to be ready (up to 90s)
6. Start Web Server on port 3002
7. Stream all logs to terminal with [cirrus] / [server] prefixes

Open http://localhost:3002 in Chrome.

Options

.\SimWorld-Studio.ps1 -Port 3003 -McpPort 55560 -CirrusHttpPort 8687 -Gpu 1 -NoBuild

Quick Start (macOS — Web stack only, no UE)

Unreal Engine 5.3 has no native macOS build of this project, but the web stack (React UI, Node API, asset/skill/scene browsers, agent panel scaffolding) runs fine on Mac for frontend work and API exploration. UE-dependent features (viewport, scene spawn, agent control) will show ueConnected: false and are no-ops.

Prerequisites

• Node.js 18+ (brew install node)

Run

./SimWorld-Studio-Mac.sh              # installs deps, builds frontend, starts backend on :3002
./SimWorld-Studio-Mac.sh --no-build   # reuse existing dist/
./SimWorld-Studio-Mac.sh --dev        # vite dev server on :5173 with HMR + backend on :3002

Then open http://localhost:3002 (or :5173 in --dev).

The script sets UNREAL_PORT=1 / UCV_PORT=1 so the backend's UE probes fail fast and stay quiet — no Cirrus, no UE Editor, no MCP subprocess required.

Quick Start (Linux — Shared Server)

See INTERNAL_SETUP.md for port assignment table and multi-user setup.

export UE_ROOT=/data/murray/ue/UE_5.3
export UE_PROJECT_PATH=/data/user/simworld_projects

simworld-studio start \
  --port 3004 --mcp-port 55561 \
  --cirrus-http-port 8687 --cirrus-ws-port 8688 --cirrus-sfu-port 8990 \
  --gpu 0

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Browser (React UI)                    │
│  Coding Agent │ Viewport │ Embodied Agent │ Statistics   │
└──────┬────────────┬──────────────┬────────────┬─────────┘
       │ SSE/HTTP   │ WebSocket    │ SSE/HTTP   │
       ▼            ▼              ▼            ▼
┌─────────────────────────────────────────────────────────┐
│              Web Server  (Node.js / Express)             │
│  index.js  │  AgentController  │  MetricsHub  │  SSE    │
└──────┬──────────────┬──────────────────────────────────-┘
       │              │
       ▼              ▼
┌────────────┐  ┌─────────────────────────────────────────┐
│ Claude CLI │  │            Unreal Engine 5.3             │
│ (MCP tools)│  │  UnrealMCP (TCP:55557)  │  UnrealCV      │
│            │──│  spawn / delete / move  │  vget / vset   │
└────────────┘  │  Python scripting       │  camera / obj  │
                └─────────────────────────────────────────-┘
                              │
                ┌─────────────┴──────────────┐
                │  Cirrus Signalling Server   │
                │  Pixel Streaming  :8685/86  │
                └────────────────────────────┘

Component Map

Features

🏗️ Coding Agent (Scene Generation)

• Natural language → 3D scene in real time
• Pluggable backends — Claude Code, Codex, OpenCode, or Gemini CLI, each with a selectable model, chosen from the top-left of the UI (see Step 1 — Set Up a Coding Agent)
• 125 buildings (BP_Building_01–127), 6 trees, vehicles, street furniture, static meshes
• 17 marketplace packs discoverable via list_assets() (allow-AI licensed)
• Asset retrieval pipeline — semantic retrieval over the full asset library injects a setting-appropriate palette into the build and exposes a search_assets tool for on-demand lookups (default hybrid mode); see Asset Retrieval & Scene Generation
• Auto session-suffix on actor names prevents cross-map name collision crashes
• verify_scene tool: the agent evaluates a screenshot and returns PASS/NEEDS_IMPROVEMENT/FAIL

🔁 Build-Critic Loop & Asset Retrieval

• Build-critic loop: an iterative builder → critic → refine loop (text or multi-view visual critic) with a rolling intent summary, streaming round_start / critic_verdict / loop_done cards to the chat. Toggle from the chat status bar (Loop: off → text → visual) or POST /api/scene-loop {"mode":"text_loop"}. Default off — vanilla single-shot generation is unchanged.
• Asset retrieval: the search_assets(query, category, k) MCP tool runs semantic search over the full ~16k-asset library (Qdrant + Postgres + embedding service) and returns real spawn paths + dimensions ranked by relevance, so the agent places setting-appropriate real assets. Indexing pipeline + DB setup live under tools/ and docs/asset-retrieval/.

🛡️ Robustness (server-side traffic shaping)

• UE-MCP broker: every session funnels through one global serial queue at UE (concurrency=1, bounded queue + 429/Retry-After backpressure, token-bucket rate limit, dedicated execute_python_script slow lane) so request bursts can't overload/crash UE. Live status at /api/internal/ue/status.
• Opt-in UE auto-restart: UE_AUTORESTART=1 restarts a crashed UE editor in place instead of tearing the instance down — strictly bounded (max N restarts per window + cooldown) so a crash loop can't run away. Default off.
• UCV kill-switch: DISABLE_UCV_BROKER=1 makes the UnrealCV (port 9000) broker a no-op if leaking UCV connections wedge UE's game thread (scene-gen uses MCP take_screenshot, not UCV).
• PixelStreaming throttle: per-IP connection debounce (PS_CONN_MIN_GAP_MS, default 200ms) + single-flight ControlPixelStreaming to absorb reconnect storms.

👁️ Live Viewport

• Pixel Streaming via Cirrus signalling server — full UE viewport in browser
• Click to activate mouse/keyboard input
• 1920×1080 default, adaptive resolution via MatchViewportRes

🤖 Embodied Agent Panel

• Spawn and control Base_Pedestrian / Base_User_Agent / Base_Demo agents
• Per-agent session: position, rotation, heading (compass), speed, status
• Real-time state: background poller every 3s via UnrealCV
• Trajectory view: top-down SVG map with heading arrows, collision markers (red dots)
• Camera tab: 3-strategy agent POV (actor camera → camera/0 at eye-level → latest screenshot)
• Activity log: ReAct thought → tool actions → response per turn
• Hit tracking: OnActorHit plugin event → collision count, impulse, impact point
• Floating draggable detail window per agent

📊 Statistics Panels

• Embodied Agent Statistics: aggregate map (2D top-down, all agents), collision/speed time-series charts
• Coding Agent Verifier: rule-based (scene collision count) + VLM-based (Claude reads screenshot, scores 1–10)
• MetricsHub: server-side 5s sampling → real-time SVG LineChart / MultiLineChart

📁 Asset Content Drawer

• Built once at startup from live UE Python scan (EditorAssetLibrary.list_assets)
• UE Content Browser–style navigation: folder → shows subfolders + direct assets
• Left sidebar: global category filter (Buildings / Trees / Vehicles / etc.)
• Search: recursive subtree, category-aware
• Click any asset → inserts correct spawn command into chat

🧪 Multi-Agent Testbed

• Auto-start PIE (polls until confirmed active)
• Spawn N agents at configurable radius, broadcast goal
• Testbed log with per-step status

🔧 Developer Tools

• Mock mode: replay pre-recorded sessions without GPU
• Session management: slot-based with heartbeat and TTL
• Skills: pre-made prompts for city layout, weather, navigation
• Learned tools: Claude can save custom tool recipes

Note on demo assets: The featured demo showcases scenes built with high-quality commercial 3D assets (buildings, vehicles, characters, etc.) that are not included in the open-source release due to licensing restrictions. The redistributable Minimal build ships with a different set of freely licensed assets, so the visual appearance will differ from the demo. The functionality and workflow remain the same.

UE Plugin Extensions

The UnrealCV plugin has been extended with:

Recompile required after updating plugin files (Ctrl+Alt+F7 in UE Editor).

Module READMEs

For Developers

Build from Source

# Requires access to the simworld_arena source repo
./build.sh

Release

1. ./build.sh → creates dist/simworld_studio-{VERSION}.tar.gz
2. Upload to GitHub Releases
3. Update version.json

Troubleshooting

View Logs

# UE logs (Linux Minimal binary)
tail -f gym_citynav/Saved/Logs/gym_citynav.log

# Studio backend logs
tail -f simworld_studio_workspace/logs/server.log

License

Apache 2.0 — see LICENSE.