Context Language Models (CLMs)

Rulin Shao^1,2, Shannon Zejiang Shen³, Junjie Oscar Yin^1,2, Yuetai Li¹, Minheng Wang¹, Hamish Ivison¹, Radha Poovendran¹, Nathan Lambert⁴, Teng Xiao¹, Mike Lewis², Wen-tau Yih², Luke Zettlemoyer^1,2, Pang Wei Koh¹

¹University of Washington   ²Meta Superintelligence Labs   ³MIT   ⁴Trillium Labs

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files.

• Zero-shot. Building CLMs zero-shot with existing models outperforms SOTA context-management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task.
• In-context learning. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute.
• Reinforcement learning. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs.

Day 1 Support

CLM for Pi:

pi install npm:@lolipopshock/pi-clm

Getting started

Run the minimal CLM agent on any Harbor task:

pip install -e .
clm-harbor run -p <harbor-task> -a clm-minimal -m openai/<model> \
  --agent-kwarg api_base=http://localhost:8000/v1

clm-harbor is the Harbor CLI with CLM available as -a clm-minimal. See clm/clm_harness for configuration and serving.

Repository

Coming soon

• ContextBench

Citation

If you find our work helpful, we would appreciate it if you could cite our paper:

@article{shao2026context,
  title   = {Context Language Models},
  author  = {Shao, Rulin and Shen, Shannon Zejiang and Yin, Junjie Oscar and Li, Yuetai and
             Wang, Minheng and Ivison, Hamish and Poovendran, Radha and Lambert, Nathan and
             Xiao, Teng and Lewis, Mike and Yih, Wen-tau and Zettlemoyer, Luke and Koh, Pang Wei},
  journal = {arXiv preprint arXiv:2609.37725},
  year    = {2026}
}

License

This project is licensed under CC BY-NC 4.0. See also NOTICE.