Next.js Evals

Agent evaluations for Next.js coding tasks, powered by @vercel/agent-eval.

Each eval hands a coding agent a small Next.js app and a prompt, lets it work in an isolated sandbox, then runs withheld assertions against what it produced.

Quick start

pnpm bootstrap                 # install, sync eval fixtures, check credentials
cp .env.example .env.local     # then fill in the credentials it asked for
pnpm preflight                 # confirm they resolve
pnpm eval:smoke claude-opus-5  # one eval, one run, real sandbox

pnpm bootstrap is not required — it just runs the three setup steps below in order, because the middle one is easy to miss.

Setup

1. Install

pnpm install --frozen-lockfile

pnpm only: the lockfile and packageManager pin are pnpm, and CI installs frozen.

2. Sync the eval fixtures — required

pnpm sync-evals          # from vercel/next.js@canary
pnpm sync-evals <ref>    # ...or a branch, tag, or commit SHA

The evals are not in this repo. They live in vercel/next.js and evals/ is git-ignored here, so a fresh clone has no fixtures and every command fails with Evals directory not found. sync-evals sparse-checkouts them.

Fixtures listed in upstream evals/eval.config.json use local skill treatments for PR development. sync-evals omits them from this repository's public model and bundled-docs matrix. Syncing never relabels old results as passes: content changes stay stale until rerun.

Syncing from canary picks up whatever landed upstream since results were last recorded, so it usually reports evals as changed:

N result(s) have a changed eval and were left stale — run them to refresh.

That is expected, not a problem — it is the incremental workflow telling you what pnpm status will now list as work. To match CI instead, pass the SHA that .github/workflows/eval-cache-check.yml pins.

3. Provide credentials

Copy .env.example to .env.local and fill in what you need. pnpm preflight reports exactly which variables are missing, and for which experiments — run it instead of guessing.

Sandbox. Every experiment sets sandbox: "vercel", so runs need Vercel Sandbox credentials by one of two paths:

A partial token triple is the real trap here: @vercel/sandbox only treats the three as credentials when all three are set, so one or two silently falls back to the OIDC path and fails with Could not get credentials from OIDC context — which names neither the variable you forgot nor the path you meant. pnpm preflight calls this out.

Agents. Each experiment names an agent, and the agent decides which key it reads. VERCEL_OIDC_TOKEN is the fallback for all of them, so the OIDC path above can cover this whole table on its own.

Missing an agent's key is not fatal: the runner prints … not set, skipping <experiment> and moves on. That line is easy to lose in a long run, so pnpm preflight <experiment> fails outright when you name an experiment you cannot actually run.

Scripts

Anything that runs an eval takes experiment names or globs — pnpm eval:run 'claude-*'. Some experiments also honour EVAL_FILTER (an eval name, a glob, or a comma-separated list) to narrow which evals run; that is per-config, so check the experiment before relying on it.

Results are memoized by a fingerprint of the eval content plus the experiment config, so a re-run only covers what actually changed. --force ignores the cache.

pnpm export-results

Exports clean results to agent-results.json. Non-model failures (infra/timeout) are automatically deleted during eval runs, so only valid model results are exported.

pnpm export-results --check verifies that the committed JSON matches a full export, ignoring only metadata.exportedAt. CI runs this check without changing the file. Export errors exit nonzero and an empty export is rejected.

Each experiment also gets an avgCostUsd: the mean list cost per eval. Tokens are read from each run's transcript-raw.jsonl (handled per harness in scripts/cost.ts) and multiplied by the list prices in MODEL_PRICING. A model with no price entry, or whose runs carry no token usage, exports null and renders as N/A. Prices are a snapshot, so re-run the export to refresh them.

Eval structure

Each eval is a self-contained Next.js project:

evals/agent-031-proxy-middleware/
├── PROMPT.md        # task given to the agent
├── EVAL.ts          # assertions (withheld from the agent)
├── package.json     # Next.js project manifest
├── tsconfig.json
├── next.config.ts
└── app/
    ├── layout.tsx
    └── page.tsx

Adding a new eval

Evals are authored in vercel/next.js, not here — evals/ in this repo is a synced copy and is git-ignored, so anything you add to it locally is overwritten by the next pnpm sync-evals.

1. Add the eval upstream, under evals/evals/ in vercel/next.js.
2. Here: pnpm sync-evals to pull it in.
3. pnpm status — it shows up as new for every experiment.
4. pnpm eval:run <experiment> to record results.

Adding a new model

1. Create a config in experiments/ (e.g., experiments/gpt-5.ts)
2. Add the display name to MODEL_NAMES in scripts/export-results.ts
3. Add the list price to MODEL_PRICING in scripts/cost.ts (or the cost column shows N/A)
4. pnpm eval:run <experiment> — every eval is new for it

Editing an existing experiment config changes its fingerprint, which makes its cached results stale. That is the intended signal, but it means config churn costs real re-runs.

Reasoning effort

Models that expose a reasoning-effort ladder get published at a reasonable effort, not their ceiling — high where a low→max ladder exists. The board is meant to tell someone what a model does on their Next.js work, and almost nobody runs day-to-day coding at top effort: it costs disproportionately more for a marginal gain, and it flatters models whose ceiling is mostly a pricing tier. A model measured at a rung nobody pays for is not a useful data point.

So pick the rung by what people run, not by what the vendor offers:

• Default to high. Encode the effort in the experiment slug and the MODEL_NAMES label — gpt-6-astra-high, GPT 6 Astra (high) — so the board never shows a score without saying what produced it.

• Going above high needs a reason in the config comment, not just headroom.

• Confirm the rung exists before running. Sending an effort the model does not accept is a whole wasted matrix. The gateway is the source of truth and it validates: a bad value returns HTTP 400 whose message enumerates the real set, which is worth more than any catalog.

  curl -s https://ai-gateway.vercel.sh/v1/chat/completions \
    -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"model":"<id>","reasoning_effort":"<rung>",
         "messages":[{"role":"user","content":"ok"}],"max_tokens":2000}'

  Catalogs disagree with the API and with each other — openai/gpt-6-astra accepts minimal, which neither the gateway's own /v1/models nor models.dev lists. Trust the 400.

CI

eval-cache-check.yml syncs fixtures at a pinned vercel/next.js SHA and runs scripts/check-stale.mjs, which fails on any new or changed eval that is not listed in that script's ACCEPTED_STALE. CI never runs an eval — it only checks that the cache is fresh or explicitly accepted as stale. To adopt upstream changes: bump the SHA, re-run the experiments you are refreshing, and record the rest in ACCEPTED_STALE.

Publishing to nextjs.org/evals

After running evals:

1. Export the full dataset: pnpm export-results.
2. Include agent-results.json with the results in your PR to this repo.
3. Merge the reviewed PR to main.

nextjs.org/evals reads the published JSON on the server — the server-fetch integration shipped on 2026-09-09 (vercel/front#85415), and the copy the site used to carry is deleted. Result updates require no copy, PR, or deployment in front. Merging this file to main publishes it to the website.

After the checks pass on main, CI calls POST https://nextjs.org/api/evals/revalidate using a short-lived GitHub Actions OIDC token. Only the main workflow can invalidate production; PR runs only validate results. No shared secret needs provisioning.

The website caches the snapshot without timed server revalidation. Invalidation starts a stale-while-revalidate window: subsequent requests can trigger a background refresh and receive the old snapshot for up to one hour, after which a server request must wait for fresh data. Browser navigation can reuse a page for five minutes, and already-open tabs need a refresh. Each upstream refresh bypasses GitHub's raw-file cache using a unique query parameter.

Delivery is retried three times and fails the revalidate-site job if unsuccessful. Rerun that job or manually dispatch this workflow on main to retry; there is no periodic refresh to repair a missed notification. Revert the JSON change and pass main CI to roll back published results through the same process.

Keep the existing JSON shape compatible with the website. Coordinate changes to required fields or scoring semantics with front; the format is currently unversioned.

Model retention policy

The published board is two tiers:

• Tier 1 (current): models with a complete run of the current eval set on a recent Next.js canary. Per model family, tier 1 carries the latest version, plus the previous version if and only if the current version was released less than one month after it (a just-superseded model is still what many people run; an older gap means it is simply outdated). A variant line the vendor stopped shipping (e.g. the codex-branded GPTs after 5.3-codex) is superseded by the vendor's main line, not kept as its own family. A tier-1 model that goes stale — the eval set or canary moved on — gets rerun, not left to coast on old measurements.
• Tier 2 (previously measured): every other model keeps its last measured results for historical reference, clearly dated, and is not rerun.

A model can also be registered but unmeasured: an experiment config, a display name and a list price exist, but no run does. It is in neither tier and does not reach the board at all, because export-results only exports experiments that have results. That state is a staging post, not a destination — land the run and tier the model in the same PR, or drop the config.

Models the provider no longer serves (e.g. Cursor Composer 1.5) are removed entirely rather than kept in tier 2 — every published experiment must be reproducible.

Scoring and cost methodology

Scoring is pass@4. Each eval runs up to four attempts (runs: 4, earlyExit: true): the eval passes if any attempt passes, and remaining attempts abort on the first pass. A published failure means four genuine model failures — attempts that die on infrastructure (rate limits, sandbox faults, auth) are classified by the failure classifier, deleted, and rerun rather than counted against the model.

Costs use provider-reported token counts. Each run's usage (input, output, cache read/write) comes from the token counts the model's own API reported in the transcript, priced at the list rates in scripts/cost.ts (snapshotted from the AI Gateway / models.dev vercel entries). For the rare transcript that carries no usage, estimateUsageFromTranscript falls back to a canonical approximation — visible text length / 4, assistant text priced as output, cache traffic assumed zero. Runs with no transcript at all (e.g. timeouts) are excluded from cost averages.

Current evals

As synced from vercel/next.js@canary. Upstream is the source of truth — after a sync, ls evals/ is authoritative.

License

See LICENSE.