GPT-Policy: In-Context Robot Learning with VLM Agents

This repository is the public implementation of GPT-Policy, a closed-loop control framework that connects a fixed vision-language model (VLM) to robot tools. At deployment time, the agent can use demonstrations, goal images, interaction history, and execution feedback without gradient updates or task-specific parameter changes.

The implementation currently provides hardware adapters for two robot arm platforms: ARX X5 and I2RT/YAM.

Paper: In-Context Robot Learning with VLM Agents
Authors: Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, and Tong Wu.

Abstract

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.

News

🚀 [2026/09/16] The paper, project page, and code are now publicly available!
🎥 [2026/09/11] The first robot cases and demonstrations are added to the project page!

Demos

Four synchronized views of three GPT-6 Astra robot runs. Click any preview to open its MP4.

_{Plug insertion · top view Plug insertion · right wrist view}

_{Sprite retrieval · search and place the bottle Bottle opening · unscrew and separate the cap}

The full experiment gallery includes the other tasks and context comparisons.

Method overview

GPT-Policy builds one model input from the task, live camera/state observations, task references, and the previous tool result. The VLM emits one structured request; the selected adapter validates and executes it, then returns fresh observations and feedback for the next decision. Adapters support Cartesian targets and waypoint sequences, sequential IK checks, backend-specific timing, gripper control, and append-only run recording.

The available context types are:

Human Video: a visual procedure that can transfer across embodiments.
Robot Video / Video + Action: robot interactions, arm roles, and aligned motion references.
Target Image: the desired object arrangement, position, and spacing.
Self History: earlier observations, actions, results, and discovered subgoals.
Human-Robot Interaction: live intent, pointing, corrections, and turn-taking.

Installation

Python 3.10 or newer is required.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

The base install is hardware-free. Install only the backend you need:

source .venv/bin/activate
python scripts/install_drivers.py arx
python scripts/install_drivers.py yam
python scripts/install_drivers.py realsense
# or: python -m pip install -e '.[arx,realsense]'

Agent CLIs are external dependencies. Install and authenticate the provider you select; credentials are stored outside this repository.

Quick start

For the default ARX profile, edit the placeholders in configs/arx_gpt.json once, then run a task directly:

source .venv/bin/activate
gpt-policy "pick up the red block"

The command resolves configs/arx_gpt.json automatically. To validate the profile without opening hardware or a model session:

gpt-policy --check

For another machine or provider, pass an explicit profile:

cp configs/examples/yam-local.json configs/my-machine.json
# edit interfaces, camera serials, and measured calibration
gpt-policy --config configs/my-machine.json "pick up the red block"

An input package can contain text, images, videos, or reviewed demonstrations:

gpt-policy --input-json task.json

Each motion is planned from fresh feedback, checked with per-sample IK, and recorded as an append-only run directory. Ctrl+C requests software cancellation and cleanup; it does not replace a hardware emergency stop.

Results from the paper

🎯 Target Image / Self History / Human–Robot Interaction: 100% success on each of the six tasks evaluated under these conditions.

👀 Human Video: Success rate improves from 0% to 67% on both towel and notebook pickup.

🤖 Robot Video + Action: Success rate improves from 0% to 100% on bottle opening and from 0% to 67% on plug reinsertion.

Repository layout

src/gpt_policy/       protocol, input preparation, planning, recording, adapters
configs/arx_gpt.json  default sanitized ARX profile for `gpt-policy "..."`
configs/agents/       provider examples
configs/examples/     local-machine templates
scripts/              opt-in driver installation
tests/                offline protocol and configuration tests
docs/assets/          figures used in this README

The public tree excludes deployment hosts, private prompts, real credentials, run recordings, site-specific calibration, and evaluation history. The paper's physical demonstration records and complete evaluation environment are not included by implication.

Development

python -m pytest -q
python -m compileall -q src

TODO

Release the GPT-Policy pipeline for real-world ARX robots, including the complete harness and format adapters for different context types.
Release the YAM pipeline with hardware integration and flexible context support.
Release the RoboDojo simulation pipeline for reproducible evaluation.
Optimize the agent harness for context construction, feedback, and execution efficiency.

Limitations and safety

This is a research control loop. The integrator must verify calibration, workspace limits, collision behavior, camera placement, provider configuration, and emergency-stop procedures before energizing a robot. IK acceptance and a model completion message do not establish collision-free motion or physical task success. The project license is intentionally pending; redistribution and commercial use are not granted by this preview.

See THIRD_PARTY.md for third-party notices and optional SDK sources.

Citation

@misc{cheng2026gptpolicy,
  title        = {In-Context Robot Learning with VLM Agents},
  author       = {Cheng, Dongzhou and Yi, Taoran and Fang, Ye and Zhang, Xingwu and Feng, Fan and Li, Yixuan and Zhuang, Gengxiong and Wang, Rongze and Yang, Shuai and Song, Wei and Xue, Weizhi and Wu, Minyan and Gui, Jie and Wang, Jiaqi and Wu, Tong},
  year         = {2026},
  howpublished = {\url{https://github.com/cheng-haha/GPT-Policy-Eval}}
}

Acknowledgements

GPT-Policy integrates optional ARX, I2RT/YAM, RealSense, and provider CLI interfaces, with reference to RoboCurve's inspect-robots project. Please see THIRD_PARTY.md before redistributing a deployment that includes external SDKs.