# DM05 Tutorial

DM0.5 is Dexmal's open-world VLA: a 4B VLM plus a 680M Action Expert that emits continuous actions with Flow Matching. It is larger than DM0, and history, embodied reasoning, action supervision, and data quality are stronger, so the model can track task progress, follow open instructions, and stay stable under camera change and human interference.

Main gains (pretrained):

- Zero-shot: follow language in unseen open scenes (8 primitives, 7 condition types).
- Fine-tune: a stronger base gives cheaper, better specialists.
- Long memory: history up to 60 seconds.
- Robust actions: lighting, camera motion, human perturbation.
- Multi-embodiment: multi-robot pretrain, then transfer.

This repo implements **Gemma3 VLM + Gemma3 Action Expert**.

![DM0.5 model overview](../resources/dm05_model_overview.png)

More narrative and official eval videos: [DM0.5 blog](https://www.dexmal.com/blog/dm0.5).

## Pretrained Model

Download the Hugging Face base weights to `checkpoints/DM05`. Playground training defaults resolve to that path.

| Model | Description | Input Images | Action Dim | Backbone | Link |
| - | - | - | - | - | - |
| DM05 | Base Gemma3 VLM + Flow Matching Action Expert | set by the playground (LIBERO: `agentview` + `wrist`) | padded to 32D (LIBERO valid dim 7) | 4B VLM | [🤗 Hugging Face](https://huggingface.co/Dexmal/DM05) |

```bash
huggingface-cli download Dexmal/DM05 \
  --local-dir checkpoints/DM05 \
  --local-dir-use-symlinks False
```

Entrypoints default to `./checkpoints/DM05` for training. The official LIBERO eval checkpoint is [Dexmal/DM05-libero](https://huggingface.co/Dexmal/DM05-libero); see [Benchmark Results](#benchmark-results).

## Installation and Environment Setup

The DM05 image is `dexmal/dexbotic:dm05`. Clone the repo on the host, mount it into the container, then install the mounted source.

```bash
git clone https://github.com/Dexmal/dexbotic.git

docker run -it --rm --gpus all --network host \
  -v /path/to/dexbotic:/dexbotic \
  dexmal/dexbotic:dm05 \
  bash

cd /dexbotic
conda activate dexbotic
pip install -e .
```

Local conda install if you are not using Docker:

```bash
git clone https://github.com/Dexmal/dexbotic.git
conda create -n dexbotic python=3.10 -y
conda activate dexbotic

pip install torch torchvision \
  --index-url https://download.pytorch.org/whl/cu128

pip install ninja packaging
MAX_JOBS=2 pip install flash-attn --no-build-isolation

cd dexbotic
pip install -e .
pip install transformers==5.3.0
```

The default train recipes below need 8 GPUs (H20 / A100 / H100 for full SFT; 4090 works for LoRA). One GPU is enough for inference. FSDP2 notes: [FSDP2.md](FSDP2.md).

## LIBERO Data

LIBERO is the shipped example. The in-repo recipes train on `libero_pi0_all` from [Dexmal/libero](https://huggingface.co/datasets/Dexmal/libero). After download, that split should sit under `data/libero`. Full SFT vs LoRA differences are in the [Training](#training) table.

```bash
huggingface-cli download Dexmal/libero \
  --repo-type dataset \
  --local-dir data/.hf_downloads/libero \
  --local-dir-use-symlinks False

mkdir -p data/libero
cp -a data/.hf_downloads/libero/libero_pi0_all data/libero/
test -d data/libero/libero_pi0_all/jsonl
test -d data/libero/libero_pi0_all/image
```

If the download is archives instead of a folder, extract into `data/libero` so those two paths exist. Dataset catalog: [Data.md](Data.md#using-provided-data).

## Training

Start from `./checkpoints/DM05` and `data/libero/libero_pi0_all` above. Full SFT and LoRA are two playground files.

### Training a Model with Provided Data

| Item | Full SFT | LoRA |
| --- | --- | --- |
| Entrypoint | [`libero_dm05.py`](../playground/benchmarks/libero/libero_dm05.py) | [`libero_dm05_lora.py`](../playground/benchmarks/libero/libero_dm05_lora.py) |
| `use_lora` | `False` | `True` |
| Backend | FSDP2 | DDP only (`deepspeed=None`) |
| GPUs | 8 × H20 / A100 / H100 | 8 × 4090 or H20; on multi-GPU 4090 set `NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1` |
| Extra install | — | `pip install "peft>=0.13.0"` |
| LLM attention | `flex_attention` | `eager` |
| Vision / action attention | `flash_attention_2` / `sdpa` | `sdpa` / `sdpa` |
| VLM / AE gradient checkpointing | off | on |
| LR / warmup | `2.5e-5` / `1000` | `5e-4` / `500` |
| Per-device batch | `8` | `4` |
| `model_max_length` | `768` | `1024` |
| Steps / save | `50000` / `10000` | `50000` / `10000` |
| LoRA | — | `r=32`, `alpha=16`, `dropout=0`, `all-linear` (no `lm_head`) |
| Output | `./user_checkpoints/dexbotic/libero_dm05/libero-sft` | `./user_checkpoints/dexbotic/libero_dm05_lora/libero-lora-MMDD` |
| Infer default path | `./checkpoints/DM05` unless you pass `--model_name_or_path` | `./user_checkpoints/dexbotic/libero_dm05_lora/libero-lora-MMDD` |

Shared: `model_name_or_path=./checkpoints/DM05`, `dataset_name=libero_pi0_all`, 2 views, `chunk_size=10`, `diffusion_steps=10`, pad-to-square then `448×448`, `bf16`, `save_hf_sidecar=True`.

```bash
# full SFT
torchrun --nproc_per_node=8 playground/benchmarks/libero/libero_dm05.py --task train --train-backend fsdp2

# LoRA
pip install "peft>=0.13.0"
export NCCL_P2P_DISABLE=1   # multi-GPU 4090 only
export NCCL_IB_DISABLE=1
torchrun --nproc_per_node=8 playground/benchmarks/libero/libero_dm05_lora.py --task train --train-backend ddp
```

If `norm_stats.json` is missing at train start, rank 0 computes it and the other ranks wait. Training raises if that file never appears. Inference does not compute it; the file must already sit next to `--model_name_or_path`.

```bash
python playground/benchmarks/libero/libero_dm05.py --task compute_norm_stats
```

LoRA writes `adapter_model.safetensors` + `adapter_config.json` and `checkpoint-{step}-hf`. Infer merges the adapter; continuing from an adapter keeps `PeftModel`. More LoRA recipe notes: [LiberoLora.md](LiberoLora.md#dm05-lora-recipe).

### Training a Model with Your Own Data

Write Dexdata JSONL, register it under `dexbotic/data/data_source`, then copy a LIBERO playground file and edit it for your data. Each frame needs images, `state`, and `prompt`. The example below is three-view:

```python
dexdata_frame = {
    "images_1": {"type": "image", "url": frame["image_path_1"]},
    "images_2": {"type": "image", "url": frame["image_path_2"]},
    "images_3": {"type": "image", "url": frame["image_path_3"]},
    "state": frame["robot_state"],
    "prompt": frame["instruction"],
    "is_robot": True,
}
```

Register under `dexbotic/data/data_source/`:

```python
from dexbotic.data.data_source.register import register_dataset
import math

MY_CUSTOM_DATASET = {
    "my_robot_data": {
        "data_path_prefix": "",
        "annotations": "/path/to/your/custom_dataset/",
        "frequency": 1,
    },
}
meta_data = {
    "non_delta_mask": [6],
    "periodic_mask": [3, 4, 5],
    "periodic_range": 2 * math.pi,
}
register_dataset(MY_CUSTOM_DATASET, meta_data=meta_data, prefix="my_custom")
```

The registered name is `my_custom_my_robot_data`. Copy `libero_dm05.py` or `libero_dm05_lora.py` and change the fields below to match this three-view example (keep two-view defaults if your data is two-view):

```python
# DM05TrainerConfig
output_dir = [checkpoint dir]
num_train_steps = [steps]

# DM05DataConfig
dataset_name = "my_custom_my_robot_data"
num_images = 3
images_keys = ["images_1", "images_2", "images_3"]
valid_action_dim = 7

# DM05ModelConfig / DM05InferenceConfig
model_name_or_path = "./checkpoints/DM05"
action_dim = 7
camera_order = ["agentview", "wrist", "right_wrist"]

# DM05DataCollator (class attr; default is 2 views)
image_prompts = ("Head", "Left wrist", "Right wrist")
```

`num_images` must equal `len(DM05DataCollator.image_prompts)` (default `("Head", "Left wrist")`). A mismatch raises.

```bash
torchrun --nproc_per_node=8 path/to/your_dm05_exp.py --task train
```

More Dexdata fields and multi-dataset `+` mixing: [Data.md](Data.md#using-custom-data).

## Inference Backends

`DM05InferenceConfig.backend` selects one of two inference paths:

| Backend | Vision path | Prefix and action path | History frames |
| --- | --- | --- | --- |
| `default` | PyTorch | Dynamic KV-cache inference | Supported |
| `fast` | TensorRT | DM05 Triton kernels with fixed prefix buckets | Supported; requests containing history bypass CUDA Graph capture |

Both backends expose the same HTTP service on `0.0.0.0:7891` by default,
including `/health`, `/v1/capabilities`, `/v1/infer`, `/v1/reset`, and the
legacy `/process_frame` route. The backend only changes how the model executes;
clients do not need to use a different request format.

Run the commands below from the repository root. Replace
`/path/to/dm05-checkpoint` with a full Hugging Face checkpoint directory or a
`checkpoint-*-hf` directory. The checkpoint directory must contain
`norm_stats.json`.

### Start the LIBERO default backend

The default backend uses PyTorch and does not require the optional fast plugin.
Start it with the LIBERO entrypoint:

```bash
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
  --task inference \
  --backend default \
  --port 7891 \
  --model-name-or-path /path/to/dm05-checkpoint
```

`--backend default` is optional because it is the configured default. The
command blocks while the server is running.

### Start the LIBERO fast backend

The fast backend requires the optional DM05 plugin. Install it in the active
Dexbotic environment:

```bash
bash plugins/dm05-fast/install.sh
```

Start the same LIBERO entrypoint with `--backend fast`:

```bash
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
  --task inference \
  --backend fast \
  --port 7891 \
  --model-name-or-path /path/to/dm05-checkpoint \
  --vision-trt-engine-path checkpoints/trt_engines/dm05-checkpoint.engine
```

The CLI leaves `build_vision_engine_if_missing=True` by default, so the first
startup builds a fixed-shape TensorRT vision engine if a matching engine does
not already exist. Later startups reuse that engine. Use a separate engine path
for each checkpoint because the engine contains checkpoint-specific vision
weights. Alternatively, pass `--force-rebuild-vision-engine` after changing
the checkpoint. The fast backend supports batch size 1.

Other fast options are available through `--prefix-seq-len-buckets`,
`--fast-overflow-policy`, `--fast-prefix-qkv-mode`, and
`--no-build-vision-engine-if-missing`. Run the entrypoint with `--help` for the
complete CLI.

To build the TensorRT engine separately instead, pass the total number of image
slots accepted by the engine. The standard history-disabled setup uses two
current camera views, so `--num-images` is 2:

```bash
CUDA_VISIBLE_DEVICES=0 python -m dexbotic.model.dm05.infer.fast.build_vision_trt \
  --checkpoint /path/to/dm05-checkpoint \
  --onnx-path checkpoints/trt_engines/dm05_vision.onnx \
  --engine-path checkpoints/trt_engines/dm05_vision.engine \
  --num-images 2
```

### Verify and call the service

After either backend finishes loading, verify that the service is ready:

```bash
curl http://localhost:7891/health
curl http://localhost:7891/v1/capabilities
```

Send two images in `camera_order` order (`agentview`, then `wrist`) to the
legacy route:

```bash
curl -X POST \
  -F "text=What action should the robot take to put the bowl on the plate?" \
  -F "image=@/path/to/agentview.png" \
  -F "image=@/path/to/wrist.png" \
  http://localhost:7891/process_frame
```

See [InferenceAPI.md](InferenceAPI.md) for the `/v1/infer` JSON schema and
`DexClient` example.

### Start a RoboTwin2 service

RoboTwin2 uses a dedicated entrypoint because its inference contract differs
from LIBERO: three RGB views (`Head`, `Left wrist`, `Right wrist`), a required
14-dimensional Aloha state, 14-dimensional actions, and a 50-step action
chunk. Do not use `libero_dm05.py` for a RoboTwin2 checkpoint.

Start the default backend:

```bash
CUDA_VISIBLE_DEVICES=0 python \
  playground/benchmarks/robotwin2/robotwin2_dm05.py \
  --backend default \
  --port 7891 \
  --model-name-or-path /path/to/robotwin2-dm05-checkpoint
```

Start the fast backend after installing `plugins/dm05-fast`:

```bash
bash plugins/dm05-fast/install.sh

CUDA_VISIBLE_DEVICES=0 python \
  playground/benchmarks/robotwin2/robotwin2_dm05.py \
  --backend fast \
  --port 7891 \
  --model-name-or-path /path/to/robotwin2-dm05-checkpoint \
  --vision-trt-engine-path \
    checkpoints/trt_engines/robotwin2-dm05-checkpoint.engine
```

The checkpoint directory must contain `norm_stats.json`. If the statistics
live elsewhere, pass `--norm-stats /path/to/norm_stats.json`. The fast command
builds a fixed-shape three-image TensorRT engine on first startup and remains
single-request. Use a checkpoint-specific engine path, as in the example.

Both backends accept the same legacy request and return an action array with
shape `50 x 14`:

```bash
curl -X POST \
  -F "text=Pick up the block" \
  -F "image=@/path/to/head.png" \
  -F "image=@/path/to/left-wrist.png" \
  -F "image=@/path/to/right-wrist.png" \
  -F 'states=[0,0,0,0,0,0,0,0,0,0,0,0,0,0]' \
  http://localhost:7891/process_frame
```

RoboTwin2 supports the same opt-in, explicit history-image contract as LIBERO.
Start the service with `--history-enabled --max-history-images 5`, then repeat
the multipart `history_images` field in oldest-to-newest order on each request.
The service does not retain frames between requests.

### Backend configuration reference

Relevant `DM05InferenceConfig` settings:

| Setting | Default | Meaning |
| --- | --- | --- |
| `backend` | `"default"` | Select `default` or `fast` inference |
| `model_max_length` | `None` | Optional preprocessing prefix-length limit |
| `vision_trt_engine_path` | `checkpoints/trt_engines/dm05_vision.engine` | TensorRT engine used by the fast backend |
| `build_vision_engine_if_missing` | `True` | Ensure that a matching engine exists during startup |
| `force_rebuild_vision_engine` | `False` | Rebuild the engine even if its input shape already matches |
| `prefix_seq_len_buckets` | `[576, 704, 768, 896, 1024]` | Fixed prefix lengths available to the fast runtime |
| `fast_overflow_policy` | `"fallback"` | Use dynamic execution or raise when no bucket fits |
| `fast_prefix_qkv_mode` | `"packed"` | Select packed or separate prefix QKV projection |
| `history_enabled` | `False` | Accept explicit history frames |
| `max_history_images` | `5` | Maximum history frames accepted by the policy |

The fast backend supports batch size 1. Without history frames, the first
request assigned to a prefix bucket lazily captures prefix prefill and
fixed-step action denoising in one CUDA Graph. Later requests using that bucket
stage their inputs into stable buffers and replay the graph. No graph profile is
captured during service startup, although CUDA Graph availability is validated
when the fast runtime is initialized. A prefix longer than the largest
configured bucket either uses the dynamic fallback or raises, according to
`fast_overflow_policy`.

### Inference Performance

| GPU | Default latency | Fast latency | Approx. speedup |
| --- | ---: | ---: | ---: |
| RTX 4090 D | 523.7 ms | 103.4 ms | 5.1x |
| H20 | 610.0 ms | 129.1 ms | 4.7x |

Measured with a real DM05 checkpoint, BF16 loading, batch size 1, chunk size
10, and 10 diffusion steps. Results use warmed-up CUDA Graph replay with two
current images and no history.

These are approximate model-inference speedups, excluding HTTP,
preprocessing, and first-request capture. History and overflow fallback
performance may differ.

### History-frame inference

History is opt-in and explicit: the server does not collect frames between
requests. The caller supplies an ordered `history_images` list with every
request. Frames must be ordered from oldest to newest.

Enable history on either backend by adding these CLI options to its startup
command:

```bash
--history-enabled --max-history-images 5
```

Both backends accept history frames. The default maximum is five images. The
fast backend has a fixed maximum of five and reserves five history slots in its
TensorRT vision engine. Therefore, a history-enabled engine for the standard
two-camera setup needs seven image slots. Give the fast history service a
different engine path from the history-disabled service:

```bash
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
  --task inference \
  --backend fast \
  --model-name-or-path /path/to/dm05-checkpoint \
  --vision-trt-engine-path checkpoints/trt_engines/dm05-checkpoint-history.engine \
  --history-enabled \
  --max-history-images 5
```

For this LIBERO configuration, automatic engine building calculates the
seven-slot shape. To build the same engine separately, use `--num-images 7`:

```bash
CUDA_VISIBLE_DEVICES=0 python -m dexbotic.model.dm05.infer.fast.build_vision_trt \
  --checkpoint /path/to/dm05-checkpoint \
  --onnx-path checkpoints/trt_engines/dm05_vision_history.onnx \
  --engine-path checkpoints/trt_engines/dm05_vision_history.engine \
  --num-images 7
```

The RoboTwin2 entrypoint accepts the same two flags. Its fast backend uses three
current-view slots plus five fixed history slots, so use a separate eight-slot
TensorRT engine path when history is enabled. Automatic engine building selects
the eight-slot shape.

When automatic engine building is enabled, the fast backend calculates this
slot count automatically. A request containing history still uses TensorRT for
vision and the optimized DM05 kernels, but it runs prefix prefill and denoising
without a request-level CUDA Graph. Requests without history can continue to
use the lazily captured graph profiles. Because the TensorRT input shape is
fixed, use different engine files for history-enabled and history-disabled
configurations.

The v1 API accepts history as an array of base64-encoded images. The legacy
`/process_frame` route accepts repeated multipart fields named
`history_images`:

```bash
curl -X POST \
  -F "text=Put the bowl on the plate" \
  -F "image=@/path/to/agentview.png" \
  -F "image=@/path/to/wrist.png" \
  -F "history_images=@/path/to/history-oldest.png" \
  -F "history_images=@/path/to/history-newest.png" \
  http://localhost:7891/process_frame
```

See [InferenceAPI.md](InferenceAPI.md) for the v1 JSON schema and `DexClient`
example.

## Evaluation

Start an inference server and POST to `/process_frame`, or point [dexbotic-benchmark](https://github.com/Dexmal/dexbotic-benchmark.git) at that server.

Pass `--model_name_or_path` at a finished run dir or a `checkpoint-*-hf`. `norm_stats.json` must sit next to that path. Full SFT without the flag falls back to `./checkpoints/DM05` (the pretrained base, not your SFT run). Server port `7891`. Two images: `camera_order=["agentview", "wrist"]`.

```bash
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
  --task inference --model_name_or_path ./user_checkpoints/dexbotic/libero_dm05/libero-sft
# or playground/benchmarks/libero/libero_dm05_lora.py --task inference \
#   --model_name_or_path ./user_checkpoints/dexbotic/libero_dm05_lora/libero-lora-MMDD
```

```bash
curl -X POST \
  -F "text=What action should the robot take to put the bowl on the plate?" \
  -F "image=@/path/to/agentview.png" \
  -F "image=@/path/to/wrist.png" \
  http://localhost:7891/process_frame
```

```bash
cd dexbotic-benchmark
docker run --gpus all --network host -v $(pwd):/workspace \
  dexmal/dexbotic_benchmark \
  bash /workspace/scripts/env_sh/libero.sh /workspace/evaluation/configs/libero/example_dm0_libero.yaml
```

The client must send those two cameras. Docker-free setup: [dexbotic-benchmark README](https://github.com/Dexmal/dexbotic-benchmark). To reproduce the official LIBERO numbers below, serve [Dexmal/DM05-libero](https://huggingface.co/Dexmal/DM05-libero) as in [Benchmark Results](#libero).

## Benchmark Results

Official DM0.5 scores on LIBERO and RoboChallenge Table30 V2.

### LIBERO

Four-suite protocol; last column is the average.

| Method | Spatial | Object | Goal | Long | Average |
| --- | --- | --- | --- | --- | --- |
| π0 | 96.8 | 98.8 | 95.8 | 85.2 | 94.2 |
| π0.5 | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| OpenVLA-OFT | 97.6 | 98.4 | 97.9 | 94.5 | 97.1 |
| GR00T N1.7 | 97.7 | 98.5 | 97.5 | 94.4 | 97.0 |
| StarVLA | 99.0 | 99.8 | 98.5 | 94.1 | 97.9 |
| ABot-M0 | 98.8 | 99.8 | 99.0 | 96.6 | 98.6 |
| Being-H0.5 | 99.2 | 99.6 | 99.4 | 97.4 | 98.9 |
| Cosmos Policy | 98.1 | 100.0 | 98.2 | 97.6 | 98.5 |
| DM0.5 | 99.0 | 99.8 | 99.6 | 97.4 | 99.0 |

The DM0.5 row is the official [Dexmal/DM05-libero](https://huggingface.co/Dexmal/DM05-libero) checkpoint. Download it and serve (`norm_stats.json` is in the repo):

```bash
huggingface-cli download Dexmal/DM05-libero \
  --local-dir checkpoints/DM05-libero \
  --local-dir-use-symlinks False

CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
  --task inference --model_name_or_path ./checkpoints/DM05-libero
```

### RoboChallenge Table30 V2

Real-robot Table30 V2 score and success rate.

| Metric | DM0.5 | π0 | π0.5 | GR00T N1.7 |
| --- | --- | --- | --- | --- |
| Score | 54.42 | - | 31.48 | - |
| SR | 43.0% | - | 14.3% | - |