Building RL Environments with OpenEnv

A hands-on course for ML engineers, researchers, and hobbyists who want to use and build RL environments for LLM training.

5 modules · ~45-60 min each · Markdown + Jupyter notebooks

Prerequisites

• Basic Python
• Familiarity with the Hugging Face ecosystem
• No RL experience required

How to Use This Course

Each module has two parts:

1. README.md — Concepts, architecture, context. Read this first.
2. notebook.ipynb — Hands-on code. Open in Google Colab and run top-to-bottom.

Modules

Quick Start

# Install OpenEnv core
pip install openenv-core

# Clone the OpenEnv repo to get typed environment clients
git clone https://github.com/meta-pytorch/OpenEnv.git

import sys, os
repo = os.path.abspath('OpenEnv')
sys.path.insert(0, repo)
sys.path.insert(0, os.path.join(repo, 'src'))

# Echo environment — uses MCP tool-calling interface
from envs.echo_env import EchoEnv

with EchoEnv(base_url="https://openenv-echo-env.hf.space").sync() as env:
    env.reset()
    response = env.call_tool("echo_message", message="Hello, OpenEnv!")
    print(response)  # Hello, OpenEnv!

# OpenSpiel environments — use standard reset/step interface
from envs.openspiel_env import OpenSpielEnv
from envs.openspiel_env.models import OpenSpielAction

with OpenSpielEnv(base_url="https://openenv-openspiel-catch.hf.space").sync() as env:
    result = env.reset()
    result = env.step(OpenSpielAction(action_id=1, game_name="catch"))
    print(result.observation.legal_actions)

Every standard OpenEnv environment uses the same 3-method interface: reset(), step(), state().

Links

• OpenEnv GitHub
• Environment Hub Collection
• TRL Documentation

Bonus: Scaling OpenEnv

For production workloads beyond a single container, see the scaling appendix below.

WebSocket vs HTTP

OpenEnv uses WebSocket (/ws) for persistent sessions instead of stateless HTTP. Each step() call is a lightweight frame (~0.1ms overhead) over an existing connection, vs TCP handshake overhead (~10-50ms) with HTTP.

One container handles many isolated sessions — each WebSocket connection gets its own environment instance server-side.

Single Container Scaling

Before adding containers, maximize a single deployment:

With 8 workers, a single container can handle ~2,048 concurrent sessions for simple text environments.

Multi-Container with Load Balancing

When a single container isn't enough, deploy multiple containers behind Envoy:

Benchmark Results

For full scaling experiments and code, see burtenshaw/openenv-scaling.

Recommendations

• Development / moderate load (<2K concurrent): Single Uvicorn or Docker container. Best per-core efficiency (256 sessions/core).
• Demos and published environments: HF Spaces free tier, reliable up to 128 concurrent sessions.
• Large-scale training (>2K concurrent): Multi-node with Envoy load balancer. See tutorial/03-scaling.md.