ClawArena Series: Benchmarking Language-Model Agents in Realistic, Evolving Environments

🔥 News

• [Jul 1, 2026] 📄 ClawArena-Team released on arXiv (2606.31174) — benchmarking subagent orchestration and dynamic workflows in language-model agents.
• [Jul 1, 2026] ClawArena v1.1 — adds a real-source-grounded dataset, the built-in ClawArena Native harness, and a Python SDK.
• [Apr 8, 2026] ClawArena released — paper (2604.04202), repository, and website (clawarena.cc) — a benchmark emphasizing agent reliability in evolving, multi-session information environments, with a fixed 12-scenario / 337-round test set and a comprehensive evaluation of SOTA models.

📖 Overview

The ClawArena Series studies a question most agent benchmarks sidestep: how do language-model agents behave when the environment is realistic, multi-session, and constantly changing — rather than a single static task? This repository brings together two complementary and equally-weighted benchmarks, each self-contained in its own subdirectory with its own data, code, documentation, and install path. Together they trace one axis of agent competence from solving to leading.

🏟️ ClawArena: Benchmarking AI Agents in Evolving Information Environments

Can an agent stay correct as information evolves and contradicts across sessions?

A benchmark for AI coding agents that reason over workspace files and multi-channel chat histories, then must stay correct as new files and sessions are injected mid-evaluation. A unified pipeline runs inference, scores results, and compares agents on the same realistic, multi-session scenarios.

• ✅ 12 multi-turn scenarios across retail, finance, healthcare, security, HR, education, and research integrity
• ✅ 337 evaluation rounds — multi_choice reasoning (95) + exec_check execution verification (242)
• ✅ 45 dynamic updates probing belief revision and contradiction handling
• ✅ Framework-agnostic — five frameworks in the paper (OpenClaw, Claude Code, NanoBot, PicoClaw, MetaClaw), plus a plugin system for new ones

📄 Paper | 📁 Code | 🔗 Details | 🌐 Website

🧑‍✈️ ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows

Can a single model manage a team of subagents it must delegate to?

The team-management variant. A text-only main agent must complete multi-turn, multimodal, multi-directory tasks it cannot do alone, by creating, empowering, scheduling, and integrating subagents from a fixed, locally-served pool. Every manager commands the same workers, so score differences reflect management skill, not raw capability.

• ✅ 41 scenarios · 258 rounds · 72 staged updates, spanning law, medicine, engineering, business, and science
• ✅ Fixed subagent pool (llm/vlm/omni) served locally, held identical across every evaluated manager
• ✅ Execution-based scoring, no LLM judge — every round ships a shell command whose exit code decides pass/fail
• ✅ Newly released glm-5.2 is the strongest open-weight manager (SMS 50.4, 4th overall)

📄 Paper | 📁 Code | 🔗 Details

🎯 Key Features

Both benchmarks share one design philosophy, applied to two different axes of agent competence:

• Empirically synthesized scenarios — procedurally built, version-controlled workspaces rather than hand-written toy tasks.
• Evolving, multi-session environments — new files and chat sessions are injected between rounds, so an agent must revise beliefs, not answer once.
• Machine-checkable, execution-based scoring — pass/fail is decided by rules and shell commands, with no LLM judge in the loop.
• Framework- and model-agnostic evaluation — the same pipeline compares many agents on identical scenarios.

📈 Leaderboard

For a benchmark, the leaderboard is the main result. Compact top rankings are shown below; each subproject hosts the full table and an open submission channel.

🏟️ ClawArena — Composite Reliability Score (CRS)

Agents ranked by CRS, which weighs raw correctness equally against behavioral consistency, macro-averaged across the 12 scenarios / 337 rounds.

_{Top 10 of 18. See the full leaderboard for all models and the cross-framework comparison.}

📥 Submit a result: see ClawArena/docs/submit-to-leaderboard.md. Reference layouts live under ClawArena/result_example/; contributed runs land in ClawArena/submissions/.

🧑‍✈️ ClawArena-Team — Subagent-Management Score (SMS)

Main-agent models ranked by SMS = TCR × (TPP + ROC + WPP + MCA) / 4 — task correctness discounted by a least-privilege and modality-routing factor (SMS ≤ TCR always).

_{Top 10 of 12, values in %, sorted by SMS. glm-5.2 (via the official z.ai API) is the strongest open-weight manager. See the full 12-model leaderboard.}

📥 Submit a result: see ClawArena-Team/docs/submit-to-leaderboard.md. Example runs and contributed results live in ClawArena-Team/submissions/.

📚 Citation

If you use the ClawArena Series, please cite the relevant benchmark(s).

ClawArena:

@article{ji2026clawarena,
  title={ClawArena: A Multi-Framework Benchmark for Evaluating AI Coding Agents on Realistic Multi-Session Scenarios},
  author={Ji, Haonian and Xiong, Kaiwen and Han, Siwei and Xia, Peng and Qiu, Shi and Zhou, Yiyang and Liu, Jiaqi and Li, Jinlong and Li, Bingzhou and Zheng, Zeyu and Xie, Cihang and Yao, Huaxiu},
  journal={arXiv preprint arXiv:2604.04202},
  year={2026}
}

ClawArena-Team:

@article{xiong2026clawarenateam,
  title={ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents},
  author={Xiong, Kaiwen and Ji, Haonian and Qiu, Shi and Zheng, Zeyu and Xie, Cihang and Ye, Xinyu and Yao, Huaxiu},
  journal={arXiv preprint arXiv:2606.31174},
  year={2026}
}

📄 License

This repository is licensed under the MIT License, covering both the ClawArena/ and ClawArena-Team/ subprojects.