RoboLab

🌐 Website · 📄 Paper · 🏆 Leaderboard

RoboLab is a task-based evaluation benchmark for robot manipulation policies built on NVIDIA Isaac Lab. It provides 100+ manipulation tasks with automated success detection, a server-client policy architecture, and multi-environment parallel evaluation — designed for reproducible, large-scale benchmarking of generalist robot policies in simulation.

Key Features

• RoboLab-120: An initial set of 120 brand new benchmark tasks spanning pick-and-place, stacking, rearrangement, tool use, and more — each with language instructions and automated success/failure detection via composable predicates.
• Bring your own robot: Tasks are not tied to a specific robot embodiment, so you can plug in any robot compatible with IsaacLab!
• Rich Asset Libraries: See a list of objects, scenes, and curated backgrounds — everything you need to create new scenes and new tasks for your own evaluation needs.
• AI-Enabled Workflows: Generate new scenes and tasks in minutes using natural language with the /robolab-scenegen and /robolab-taskgen Claude Code skills.
• Multi-Environment Parallel Evaluation: Run multiple episodes in parallel across environments with vectorized conditionals and per-environment termination.
• Results Dashboard with Episode Videos and Cross-Experiment Analysis: A self-contained web dashboard for browsing scenes/tasks, replaying episode videos, and comparing results across experiments.

Getting Started

Requires uv and a system ffmpeg (used for video recording). Isaac Sim 5.0 and Isaac Lab 2.2.0 are installed automatically via uv sync. See Requirements for hardware.

Installation

sudo apt install ffmpeg
git clone <repo_url>
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv sync

Verify installation:

uv run pytest tests/

This runs the install-verification suite end-to-end: isaaclab importable, all task definitions valid, env factory populated, one full episode runs. The suite auto-accepts the NVIDIA Omniverse EULA so the run is fully headless with no prompts. More details at Debugging → Diagnostic Scripts.

Running without activating the venv: if you don't source .venv/bin/activate, prefix every python command with uv run (e.g. uv run pytest tests/).

EULA outside the test suite: when running other entry points (e.g. policies/pi0_family/run.py) for the first time, set export OMNI_KIT_ACCEPT_EULA=Y once. Cached after first acceptance.

Run without a policy

# Run an empty episode with random actions
python examples/run_empty.py --headless

# Playback recorded demonstration data
python examples/run_recorded.py --headless

# Toggle the gripper open/closed while holding the arm fixed (sanity-check
# the gripper action path; saves sensor + viewport video to
# output/run_gripper_toggle/<task>/)
python examples/run_gripper_toggle.py --task BananaInBowlTask --headless

Run with a policy

RoboLab uses a server-client architecture: your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. To quickly test RoboLab, try Pi0.5 via OpenPI.

Quick run after install in the RoboLab terminal, to see it working:

cd robolab
uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10 --enable-subtask

Use the dashboard to view the output written to your local folder.

Common CLI Options

# Run headlessly
python policies/pi0_family/run.py --policy pi05 --headless

# Run on specific tasks (these two are good for sanity checking)
python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask RubiksCubeAndBananaTask

# Run on a tag of tasks
python policies/pi0_family/run.py --policy pi05 --tag semantics

# Run 12 parallel episodes per task
python policies/pi0_family/run.py --policy pi05 --headless --num-envs 12

# Enable subtask progress tracking
python policies/pi0_family/run.py --policy pi05 --enable-subtask

# Resume a previous run (skips completed episodes)
python policies/pi0_family/run.py --policy pi05 --output-folder-name my_previous_run

Example Tasks

See the full Benchmark Task Library for all 120 tasks.

Dashboard

A self-contained web dashboard for browsing the benchmark (scenes and tasks) and analyzing your experiment results.

uv run robolab-dashboard
# open http://localhost:8080

See docs/dashboard.md for the full feature tour, CLI flags, and the API endpoints under the hood.

Documentation

Full documentation is at docs/README.md, covering:

• Objects, Scenes, Tasks — Creating and managing assets and benchmark tasks
• Robots, Cameras, Lighting, Backgrounds — Configuring simulation parameters
• Environment Registration — Combining tasks with robot/observation/action configs
• Inference Clients — A list of supported open-source models and clients
• Analysis and Results — Summarizing, comparing, and auditing results
• Dashboard — Interactive web viewer for benchmark, tasks, scenes, and eval results
• Subtask Checking, Conditionals, Event Tracking

Requirements

Dependency	Version
Isaac Sim	5.0
Isaac Lab	2.2.0
Python	3.11
Linux	Ubuntu 22.04+

• Disk space: ~8 GB (assets account for ~7 GB)
• GPU: NVIDIA RTX GPU required. Recommend 48GB+ VRAM. See Isaac Lab's hardware requirements for recommended GPUs and VRAM.
• Speed: 30 GPU hours / 100 tasks, 1.4 it/s (assuming ~200ms inference step)

License

The RoboLab framework is released under the Apache License 2.0.

Third-party dependency licenses are listed in THIRD_PARTY_NOTICES.md.

Citation

@inproceedings{yang2026robolab,
    author    = {Xuning Yang and Rishit Dagli and Alex Zook and Hugo Hadfield and Ankit Goyal and Stan Birchfield and Fabio Ramos and Jonathan Tremblay},
    title     = {{RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies}},
    booktitle = {Proceedings of Robotics: Science and Systems},
    year      = {2026},
    address   = {Sydney, Australia},
    month     = {July},
    url       = {https://arxiv.org/abs/2604.09860}
}

Contributing

See CONTRIBUTING.md for acknowledgements, issues, and how to contribute.