# ClawBench Evaluation

ClawBench evaluation is a **post-session** step: first you run agents to collect trajectories, then you evaluate those trajectories against human reference runs.

```
Step 1: Run agents          Step 2: Evaluate
(runner package)            (this directory)

./run.sh                    Claude Code subagents compare
  or                        agent traces vs human references
src/clawbench/runner/batch.py         under eval/agentic_eval.md rubric
       │                              │
       ▼                              ▼
  test-output/                {model}-eval-results.csv
    {model}/{run}/            {model}-eval-results.json
      data/
        actions.jsonl
        requests.jsonl
        screenshots/
        recording.mp4
        interception.json
        agent-messages.jsonl
```

## How It Works

The evaluator is a Claude Code subagent that compares two trajectories side by side:

- **Agent trajectory** -- the five-layer recording from the AI agent's run
- **Human reference trajectory** -- the same five layers recorded by a human annotator completing the task correctly

The evaluator follows a fixed rubric ([`agentic_eval.md`](agentic_eval.md)) to determine PASS or FAIL for each task. This comparative approach means the evaluator has a concrete ground truth -- it knows exactly which form fields to fill, which buttons to click, and which endpoint the final submission hits.

## Prerequisites

- Agent run outputs in `test-output/{model}/` (produced by `clawbench-run` or `clawbench-batch`)
- Human reference runs in a separate directory (same five-layer format)
- [Claude Code](https://docs.anthropic.com/en/docs/claude-code) installed

## Running Evaluation

Open Claude Code at the project root and send the following prompt. Replace the three placeholders with your actual values:

- `{agent_dir}` -- path to the model's output directory (e.g., `test-output/claude-sonnet-4-6/`)
- `{human_dir}` -- path to the human reference directory (e.g., `test-output/human/`)
- `{model}` -- model name for output file naming (e.g., `claude-sonnet-4-6`)

```
Read the evaluation rubric at eval/agentic_eval.md and follow it strictly.

Evaluate all 153 agent runs against their corresponding human reference runs.

Agent runs directory: {agent_dir}
Human reference directory: {human_dir}

Each directory contains multiple run subdirectories (one per task). Each run subdirectory contains:
- run-meta.json
- data/actions.jsonl
- data/requests.jsonl
- data/screenshots/
- data/recording.mp4
- data/interception.json
Agent runs also contain data/agent-messages.jsonl.

Dispatch 16 subagents to evaluate in parallel, each subagent handling ~10 tasks. Each subagent should:
1. Match agent run to human run by task_id in run-meta.json
2. Read both run-meta.json to get task instruction and context
3. Compare the agent trajectory against the human reference trajectory
4. Determine PASS or FAIL with justification, noting which evidence files and lines support the decision

Dispatch 3 supervisor agents to monitor the work of the 16 evaluation subagents, checking for consistency and correctness.

After all subagents complete, merge their results and output two files:
1. {model}-eval-results.csv — columns: task_id, task_name, model, pass, brief_justification
2. {model}-eval-results.json — detailed results per task, each entry including: task_id, task_name, model, pass, justification, and evidence references (file path and line numbers that support the verdict)
```

## Output

The evaluation produces two files at the project root:

| File | Format | Description |
|------|--------|-------------|
| `{model}-eval-results.csv` | CSV | Quick summary -- one row per task with PASS/FAIL and a brief justification |
| `{model}-eval-results.json` | JSON | Detailed results with full justification and evidence references (file paths + line numbers) |

## Export results to OpenEval / EvalPort

[`scripts/export_openeval.py`](../scripts/export_openeval.py) exports a batch's `rescore-summary.json` as an [EvalPort](https://github.com/adhabnr-ux/evalport) `ResultSet`. It reads the summary produced by `clawbench-rescore` and enriches results with the batch's per-run `run-meta.json` files. The export preserves separate interception and judge results; a task passes when it was intercepted and the selected rubric's judge verdict is true.

After rescoring the batch, run the Python script with uv from the repository root:

```bash
uv run python scripts/export_openeval.py ./my-run \
  --run-id my-run \
  --started-at 2026-09-07T14:00:00Z \
  --rubric lenient
```

Replace `./my-run`, the run ID, and the timestamp with your batch's values. `--run-id` and `--started-at` are required because the summary does not record a batch ID or start timestamp. Choose a rubric present in the summary; omitting `--rubric` uses its first rubric.

The script writes `./my-run/resultset.json` by default. Use `--out <path>` to choose another destination or `--stdout` to print the JSON. `--completed-at` and `--suite-id` optionally supply a completion timestamp and suite identifier. Run `uv run python scripts/export_openeval.py --help` for all options.

## Evaluation Rubric

The full rubric is in [`agentic_eval.md`](agentic_eval.md). Key rules:

- **Interceptor block = PASS** if all prior steps are correct (the interceptor is designed to cut the session short)
- **Payment must be attempted** -- the agent has a dummy credit card and must try to use it
- **Phone verification wall = PASS** if all prior steps are complete (the agent has no phone number)
- **CAPTCHA must be attempted** -- skipping a CAPTCHA is FAIL
- **Email must be used** when the task requires registration or verification