EnvHarness: Awakening Static Worlds for Agent Learning

Check out our paper and webpage for more details.

🔥 Updates

• [2026-08-21] We released our paper and website.

🏴󠁶󠁵󠁭󠁡󠁰󠁿 Overview

As LLMs become autonomous agents, they learn less from curated text and more from interactive environments. But those environments are expensive to build and, once built, stay static — they behave identically no matter which agent interacts with them or how much it has improved, so they can neither target a particular agent's weaknesses nor keep teaching once its tasks are solved. EnvHarness applies the agent harness idea to the other side of the interaction: just as an agent harness makes a frozen LLM capable through plug-in components (skills, memory, tools) without changing its weights, EnvHarness wraps a frozen environment with its own plug-in components to make it dynamically controllable — without touching the environment's internal code.

The layer is assembled from three plug-in components — Setup (reshape the initial state), Rule (reshape the interaction: which actions are allowed, what they do, and what the agent observes), and Link (compose in another environment's tasks) — that operate strictly at the standard reset / step interface and stack freely. They reshape only what the agent observes, what it may do, and where it starts; the goal predicate that decides success is left untouched, so every reshaped environment keeps the original benchmark's trusted, human-built verifiers — and because nothing reaches into environment-specific code, the same system works across domains.

The three EnvHarness components against the environment interface. The leftmost panel is a bare environment; each of the others plugs in one component — Setup (reshapes the initial state), Rule (reshapes the interaction: which actions are allowed, what they do, and what the agent observes), and Link (composes another environment's tasks in) — and highlights the interface calls it intercepts. All panels expose the same contract and leave the underlying implementation, tasks, and verifiers untouched.

An LLM designer agent drives a diagnostic loop: it reads the agent's trajectories to diagnose a specific weakness, writes components that reshape the environment to target it, tests the policy in the new environment, and revises until the environment can actually teach what the agent lacks. The signal is targeted (written against diagnosed flaws) and lasting (the loop repeats as the agent improves, co-evolving the two).

Across ALFWorld, WebArena, SWE-bench Verified, OfficeQA, and SpreadsheetBench, skills learned in EnvHarness environments beat both the no-skill baseline and skills learned in the original environments — more effective (up to +9 points on held-out tasks) and more efficient (~9.8% fewer interaction steps). The same dynamic environments also produce stronger policies under reinforcement learning, and repeating the designer loop compounds the gains round after round.

Key Features

• Frozen environments, no internal edits: the benchmark's task set, its dynamics and its grading stay exactly as published. Only the layer the agent acts through changes.
• Code as the Envharness: the designer emits real Python — a _Rules(Rules) subclass — not a selection from a fixed menu. It is compiled and executed in an isolated subprocess, so a bad mutation becomes a recorded trace instead of a dead run.
• Composable by construction: an EnvHarness is an ActionableEnv that wraps another one, so layers stack arbitrarily and every benchmark is driven through a single interface.
• Benchmark-agnostic: adding an environment means implementing one interface; the designer, the components, the loop and the evaluation stages need no changes.

⚡️ Quickstart Guide

0. LLM Configuration

Every stage -- corpus generation, skill induction, evaluation -- takes the same model string, so a provider is one line of config. Three families are supported:

• GPT: to use OpenAI models (gpt-4.1-mini, gpt-4.1, gpt-4o, o4-mini), set your API key:

  export OPENAI_API_KEY="your-openai-api-key"

• Claude: to use Claude (claude-sonnet-4-6) on Vertex AI, set up Application Default Credentials (unnecessary on a GCE VM with an attached service account) and name your project:

  gcloud auth application-default login
  export GOOGLE_CLOUD_PROJECT="your-project-id"
  pip install "google-cloud-aiplatform>=1.38"

• Gemini: to use Gemini models (gemini-3.5-flash, gemini-3.1-flash-lite) through the Gemini API, set your key:

  export GEMINI_API_KEY="your-gemini-api-key"

Name the model wherever a config takes one -- the provider prefix selects the backend, and envharness.infra.model supplies that provider's auth and parameter handling:

policy:
  model: openai/gpt-4.1-mini          # or gemini/... or vertex_ai/claude-...
mutator:
  type: llm
  model: vertex_ai/claude-sonnet-4-6  # the two roles can differ

A driver's MODEL env var accepts the same strings and applies to every stage of that run -- corpus policy, harness agent, skill induction and evaluation -- overriding whatever the YAML names, so a run never ends up split across providers:

MODEL=openai/gpt-4.1 python experiments/swebench/reproduce.py

The same override is available directly on the Stage 1 runner:

python scripts/run_harness.py --config <corpus.yaml> --model openai/gpt-4.1-mini

Leave MODEL unset to keep the per-role models the config names -- which is how you give the policy and the harness agent different models.

Concurrency follows your rate limit, not your CPU. Every benchmark runs its tasks through a pool of workers (CORPUS_WORKERS, EVAL_WORKERS, EVAL_CONCURRENCY, ... -- see each benchmark's README). The defaults suit a modest quota; a pool large enough to saturate your provider's tokens-per-minute tier turns into 429s that truncate episodes mid-task, which shows up as unexpectedly low success rates rather than as an error. Raise the pool when your quota allows, and on Gemini give it several keys (GEMINI_API_KEYS) -- that quota is metered per key, so the workers spread across them.

Embeddings

Skill retrieval needs an embedding model too. You do not normally pick one: each provider is paired with an embedding model the same credentials already reach, so setting MODEL is enough.

EH_EMBED_MODEL overrides the pairing for both bank building and retrieval, and takes any embedding model litellm supports (with that provider's own credentials set):

EH_EMBED_MODEL=openai/text-embedding-3-small \
MODEL=vertex_ai/claude-sonnet-4-6 python experiments/alfworld/reproduce.py

That is also why the override exists: it holds one embedding space fixed while the policy provider changes. A bank stores its vectors, and Bank.retrieve rejects a query vector of a different width, so changing the embedding model — including by changing provider — means rebuilding the bank.

1. Run a benchmark

Each benchmark has its own environment and its own one-command driver. Open the README in the experiment folder you want to run — it carries the environment setup, the run commands and the knobs for that benchmark:

• experiments/toy24
• experiments/alfworld
• experiments/swebench
• experiments/webarena
• experiments/officeqa
• experiments/spreadsheetbench

Every folder follows the same shape:

python scripts/check_env.py <benchmark>           # preflight
bash experiments/<benchmark>/reproduce_smoke.sh   # the same stages, fewer tasks
python experiments/<benchmark>/reproduce.py       # the full protocol

The experiments above distill/evaluate skills. For RL training — a policy trained with GRPO directly inside EnvHarness environments (via verl-agent) — see rl/.

📊 Results

Skills induced from EnvHarness-adapted environments transfer back to the untouched benchmark and beat both controls — no skills at all, and skills induced from the original environments. All numbers are the mean over three independent runs, with standard deviations as subscripts. A dash marks a baseline that is benchmark-specific and cannot be applied to the other domain; EnvHarness covers every benchmark through the same interface.

ALFWorld and WebArena

SWE-bench Verified, OfficeQA and SpreadsheetBench

SR = success rate, AS = agent steps (lower is better), EM = exact match. The three skill sources correspond to the conditions each reproduce.py prints: nobank (No Skills), orig (Original Envs) and ours (EnvHarness Envs).

Models. Every number in both tables was produced with Gemini. The configs in this repo default to openai/gpt-4.1-mini, so reproducing the tables means pointing them back at Gemini:

MODEL=gemini/gemini-3.5-flash python experiments/swebench/reproduce.py

MODEL reaches every stage, embeddings included -- those runs used Gemini's own gemini-embedding-001, which is what this one switch selects, so nothing else needs setting. Alternatively edit the one model: line in that benchmark's corpus.yaml and reasoning_bank_eval.yaml. Absolute numbers move with the model; what the tables compare is skill sources at a fixed model.

🧱 Adding a New Benchmark

A benchmark joins EnvHarness by implementing one interface, ActionableEnv (reset / step / observe / evaluate / get_env_state / save_state / from_state). Everything downstream — the Environment Designer, the three components, the loop, the evaluation — is benchmark-agnostic and needs no changes.

1. Implement the Bridge

The Bridge is the only layer that may know about a docker container, a browser session, or a simulator. Subclass ActionableEnv, register a stable tag, and implement the seven required methods.

tool_registry declares the action space. Its schemas are what the Policy is given to act with, and what the Environment Designer is shown so a Rule's action hook can match on action.name. Each entry is a Tool whose schema is introspected from its invoke signature — see any envharness/bridges/*/tools.py for the shape.

# envharness/bridges/mybench/bridge.py
from envharness.core.actionable_env import ActionableEnv
from envharness.core.registry import register_env
from envharness.core.types import (
    Action, EnvResetResponse, EnvResponse, EvaluationResult, Observation,
)
from .tools import Search      # declares name="search"

@register_env("mybench")                  # tag written into save files
class MyBenchEnv(ActionableEnv):
    tool_registry = [Search]              # -> tool_schemas() for the Policy

    def reset(self, seed=None, options=None) -> EnvResetResponse:
        # The orchestrator passes the per-task identifier as `seed`; it indexes
        # into the benchmark's task library, it is not a randomness source.
        ...
        return EnvResetResponse(observation=self.observe(), info={})

    def step(self, action: Action) -> EnvResponse:
        # Dispatch on action.name -- the same names the Tools declare.
        if action.name == "search":
            result = self._search(**action.kwargs)
        else:
            result = {"error": f"unknown action {action.name!r}"}
        return EnvResponse(observation=self.observe(), reward=0.0,
                           terminated=self._done, truncated=False,
                           info={"result": result})   # by convention: info["result"]

    def observe(self) -> Observation: ...
    def evaluate(self) -> EvaluationResult: ...
    def get_env_state(self): ...          # data only -- NO runtime handles
    def save_state(self) -> dict: ...
    @classmethod
    def from_state(cls, state: dict): ...

Two contracts matter:

• get_env_state() must carry data, never handles. It is what the designer's generated hooks receive, and it crosses a subprocess boundary. A docker client or a browser page in there breaks both.
• save_state / from_state are yours to define. In-memory benchmarks can snapshot everything; for a container or a browser, store {"reset_seed": ..., "reset_options": {...}} and let from_state re-run reset — valid at episode boundaries, which is where checkpoints are taken. Override close() if there is external state to release; subprocess death does not free it for you.

2. Describe the state to the Environment Designer

env_state_schema() is injected verbatim into the designer's prompt. It is the only thing telling it which fields its generated hooks may read, so be explicit:

    @classmethod
    def env_state_schema(cls) -> str:
        return ("MyBenchState = {\n"
                "  query: str,          # the task's question\n"
                "  hits: list[str],     # results of the last search\n"
                "  submitted: bool,\n"
                "}")

Optional hooks, all with safe defaults: list_tasks() (enables agent-driven task selection), notify_replay_complete() (rewind per-episode counters after a Setup replay), default_reset_args() / reset_after_load() (checkpoint loading), step_reward() (dense per-step signal; non-fatal).

3. Point a corpus config at it

Corpus generation needs no new code — scripts/run_harness.py is bridge-agnostic. Copy the closest existing corpus.yaml and change the import path:

env:
  import_path: envharness.bridges.mybench.bridge:MyBenchEnv
  reset_options: { ... }                 # forwarded to your reset()
policy:
  client_factory: envharness.infra.llm:LiteLLMClient
  client_kwargs: { model: openai/gpt-4.1-mini }
  action_format: function_calling        # or think_action for single-tool text
objective:
  type: difficulty_zone
  target_band: [0.4, 0.6]

Verify it boots before spending a run:

python -c "from envharness.bridges.mybench.bridge import MyBenchEnv; MyBenchEnv(); print('OK')"
python scripts/run_harness.py --config experiments/mybench/corpus.yaml --n-tasks 1

4. Reuse the downstream stages

Skill induction and evaluation are shared. scripts/induce_pair.py works unchanged on per-task rollouts; write a benchmark-local induce.py only when the induction prompt needs domain phrasing. For the evaluation, copy the driver closest to your grading style — they differ only in how they load tasks and score them. Then chain the stages in a reproduce.py mirroring an existing one.

5. Add a preflight target and a test

Add a check_<bench>() to scripts/check_env.py and register it in CHECKS, so a missing dependency or dataset fails loudly before a run rather than silently grading everything as failure. Then copy tests/test_actionable_env_toy24.py — it walks the whole ActionableEnv contract, including a save_state / from_state round-trip, and needs no GPU, docker, or API key:

pytest

🔩 Adding a Harness Layer

The three components ship as three layer classes: Setup (reshape the initial state — replays a fixed action list on every reset), Rules (reshape the interaction — per-step hooks on the inner env's actions / transition / observation), and Link (compose another environment's tasks into the episode). A new layer is how you add a capability those do not cover — reward shaping, budget accounting, anything that belongs between the agent and the environment.

Because an EnvHarness is an ActionableEnv that wraps one, layers stack arbitrarily and nothing above or below needs to know how many there are.

1. Subclass and register

from envharness.core.actionable_env import ActionableEnv
from envharness.core.envharness import EnvHarness
from envharness.core.registry import register_harness
from envharness.core.types import Action, EnvResponse

@register_harness("budget")                 # tag written into save files
class Budget(EnvHarness):
    def __init__(self, inner: ActionableEnv | None = None, max_steps: int = 50):
        super().__init__(inner)
        self.max_steps = max_steps
        self._n = 0

    # Override ONLY the methods this layer affects. reset / step / observe /
    # evaluate / get_env_state / step_reward all delegate to `inner` by
    # default, so a single-axis layer is a single method.
    def step(self, action: Action) -> EnvResponse:
        resp = self.inner.step(action)
        self._n += 1
        if self._n >= self.max_steps:
            resp = EnvResponse(observation=resp.observation, reward=resp.reward,
                               terminated=resp.terminated, truncated=True,
                               info={**resp.info, "budget_exhausted": True})
        return resp

    def save_state(self) -> dict:
        return {"max_steps": self.max_steps}        # THIS layer's fields only

    @classmethod
    def from_state(cls, state: dict, inner: ActionableEnv | None = None):
        return cls(inner=inner, max_steps=state.get("max_steps", 50))

Three contracts:

• save_state returns only this layer's own fields. The persistence walker saves the inner env and every other layer separately, and emits harnesses: [innermost, ..., outermost] — index 0 sits closest to the environment, the last entry is what the agent sees.
• from_state takes an extra inner. This is a deliberate divergence from ActionableEnv.from_state: a layer is meaningless without something to wrap, and the loader supplies it while rebuilding the stack inner-to-outer.
• The tag is your on-disk format. Once a checkpoint is written with it, renaming it invalidates those files.

2. Put it in a stack

Any checkpoint naming your tag now loads, because load resolves tags through the registry rather than through import paths:

{
  "env":       {"type": "mybench", "state": {"reset_seed": 12, "reset_options": {}}},
  "harnesses": [{"type": "setup", "state": {"actions": [...]}},
                {"type": "budget", "state": {"max_steps": 30}}]
}

For a corpus run, note that a Candidate carries exactly two levers — rules_code and in_env_actions — so the episode runner composes Setup and Rules and nothing else. To have the Environment Designer emit your layer directly, extend Candidate, build_env_stack in envharness/orchestration/runner.py, and the designer's propose schema in envharness/agents/harness_agent.py. Until then the layer is usable by hand, from a checkpoint, or wherever you build the stack yourself.

3. Test the composition

tests/test_envharness_composition.py is the template: it stacks layers around a toy env and asserts the pass-through invariants (an un-overridden method must reach the inner env unchanged) plus the save/load round-trip of the whole stack. tests/test_link_envagnostic.py additionally shows how to prove a layer only ever touches the ABC, by composing two dummy envs with deliberately different action and observation shapes.

🙏 Acknowledgements

We adopt the memory design of ReasoningBank in our agent implementation, and we are grateful for their work.

💬 Citation

If our work is useful for you, please consider citing our paper:

@article{huang2026envharness,
      title={EnvHarness: Awakening Static Worlds for Agent Learning}, 
      author={Chengsong Huang and Zifeng Wang and Rujun Han and Jun Yan and Yanfei Chen and Zoey CuiZhu and Ke Jiang and Peng Xia and Han Yu and Yufan Zhuang and Yifei Ming and Jiaqi Pan and Bhavana Dalvi Mishra and Jiaxin Huang and Burak Gokturk and Tomas Pfister and Chen-Yu Lee},
      year={2026},
      eprint={2608.19880},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.19880}, 
}

This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.