A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting
_{ECCV 2026}

For a static GS scene without avatars, you only need:

• a 3DGS render asset;
• a Habitat-format .navmesh file for navigation. It defines the walkable area for agents.

IMPORTANT: Habitat-GS recognizes GS stage assets by suffix. This means your scene file MUST end with .gs.ply or .3dgs.ply.

NOTE: In our provided dataset, an outdoor scene may also have:

• a background 3DGS asset (background.ply) for far-field content like sky or distant geometry that was separated from the foreground during reconstruction. To render foreground and background together as a single stage, merge the two .ply files into one with tools_gs/merge_background_gs.py.

For NavMesh generation, use our interactive interactive web tool: it renders the GS scene in your browser and lets you draw/modify the walkable area, then bakes a Habitat .navmesh for simulation.

conda activate habitat-gs

python web_tools/navmesh_editor/server.py --port 8080   # Scene dir by default: data/scene_datasets/gs_scenes (train/ and val/ subfolders).
# then open http://localhost:8080 

python web_tools/navmesh_editor/server.py --port 8080 --scene interior_0007_840137        # Open a specific scene
python web_tools/navmesh_editor/server.py --port 8080 --gs-dir /path/to/gs_scenes/train   # Open scenes in a directory

Every gaussian avatar needs two assets:

• canonical_gs.npz: canonical gaussians exported from either GaussianAvatar or AnimatableGaussians;
• driver.pkl: scene-specific motion driver generated on the NavMesh using GAMMA method.

1.1. Export canonical gaussians from GaussianAvatar

Run this in the GaussianAvatar conda environment after installing the upstream repo:

python tools_gs/export_gaussian_avatar_to_canonical.py \
  --posmap /path/to/query_posemap_.npz \
  --lbs-map /path/to/lbs_map_.npy \
  --joint-mat /path/to/smpl_cano_joint_mat.pth \
  --net-ckpt /path/to/net.pth \
  --out /path/to/canonical_gs.npz \
  --ga-root /path/to/GaussianAvatar

1.2. Export canonical gaussians from AnimatableGaussians

Similarly, run this in the AnimatableGaussians conda environment after installing the upstream repo:

python tools_gs/export_animatable_to_canonical.py \
  --config /path/to/config.yaml \
  --ckpt /path/to/checkpoints \
  --out /path/to/canonical_gs.npz \
  --anim-root /path/to/AnimatableGaussians \
  --smpl-model-path /path/to/AnimatableGaussians/smpl_files/smplx

2. Generate the motion driver on a scene NavMesh

driver.pkl depends on the target scene because it is generated on the scene NavMesh. Run the following command in the GAMMA environment after installing GAMMA and make sure habitat_sim is importable in that environment.

GAMMA env: For setting up the conda environment, please refer to environment.yml (conda env create -f environment.yml) and follow the instruction inside it to apply a necessary patch.

We provide two modes for generating the driver trajectory:

Auto-sample a path by target length:

python tools_gs/generate_trajectory.py \
  --navmesh /path/to/scene.navmesh \
  --output /path/to/driver.pkl \
  --path-length 6.0 \
  --smpl-model-path /path/to/smpl_files/smplx \
  --gamma-root /path/to/GAMMA-release

Specify start/end/several optional via points explicitly:

python tools_gs/generate_trajectory.py \
  --navmesh /path/to/scene.navmesh \
  --output /path/to/driver.pkl \
  --start 0.0 0.0 0.0 \
  --via 1.0 0.0 -1.0 \
  --end 2.0 0.0 -2.0 \
  --smpl-model-path /path/to/smpl_files/smplx \
  --gamma-root /path/to/GAMMA-release

The generated driver.pkl contains precomputed joint_mats for rendering. Avatar rendering uses explicit Gaussians + CUDA LBS without neural forward pass at runtime. The .pkl also contains precomputed proxy_capsules used for NavMesh-level dynamic obstacle handling, guaranteeing agent cannot pass through gaussian avatars.

playroom/
├── playroom.scene_dataset_config.json
├── configs/
│   ├── scenes/
│   │   └── playroom.scene_instance.json
│   └── stages/
│       └── playroom_stage.stage_config.json
├── stages/
│   └── playroom.gs.ply
├── navmeshes/
│   └── playroom.navmesh
└── avatars/
    └── actor01/
        ├── canonical_gs.npz
        ├── driver.pkl
        └── smplx/

playroom.scene_dataset_config.json

{
  "stages": {
    "paths": {
      ".json": ["configs/stages"]
    }
  },
  "scene_instances": {
    "paths": {
      ".json": ["configs/scenes"]
    }
  },
  "navmesh_instances": {
    "playroom_navmesh": "navmeshes/playroom.navmesh"
  }
}

configs/stages/playroom_stage.stage_config.json

{
  "render_asset": "../../stages/playroom.gs.ply"
}

configs/scenes/playroom.scene_instance.json

{
  "stage_instance": {
    "template_name": "playroom_stage"
  },
  "navmesh_instance": "playroom_navmesh",
  "time_max": 20.0,
  "time_loop": true,
  "gaussian_avatars": [
    {
      "name": "actor01",
      "canonical_gaussians": "../../avatars/actor01/canonical_gs.npz",
      "driver": "../../avatars/actor01/driver.pkl",
      "smpl_model_path": "../../avatars/actor01/smplx",
      "smpl_type": "smplx",
      "scale": 1.0,
      "offset_y": 1.0,
      "time_begin": 0.0,
      "time_end": 20.0
    }
  ]
}

Notes:

• If you only want a static GS scene, simply omit gaussian_avatars.

• If your GS exporter already stores normalized quaternions and you observe spikes or blur in rendering results, add "norm_quaternion": false to the stage config.

• Time related fields explanation:

  • time_max: maximum simulation time in seconds. max(time_end) of all avatars by default. 0 if no avatars in scene.
  • time_loop: whether to loop the simulation time. True by default.

  For each avatar, you can also specify:

  • time_begin: the simulation time when the avatar appears. 0 by default.
  • time_end: the simulation time when the avatar disappears. 0.025 * num_frames by default. If time_end > time_begin + 0.025 * num_frames, the avatar will be static at the destination in remaining time.

• --width / --height to change window size.
• --time to start from a specific Gaussian time;
• --time-rate to change playback speed;
• --enable-physics to enable physics simulation if you need physics interaction with mesh objects in a GS scene. Requires building with Bullet.

• W/S: Move forward/backward
• A/D: Move left/right
• Z/X: Move up/down
• Arrow keys: Rotate view
• TAB: switch between RGB and depth
• SPACE: play/pause Gaussian time
• H: print help
• [ / ]: scrub backward/forward
• N: toggle NavMesh visualization
• ESC: exit viewer

data/scene_datasets/gs_scenes/
├── train.scene_dataset_config.json
├── val.scene_dataset_config.json
├── train/                           # 110 training scenes
│   ├── scene01/                     #   self-reconstructed (55 scenes): full assets
│   │   ├── scene01.gs.ply           #     foreground GS render asset
│   │   ├── background.ply           #     background GS asset (sky / distant geometry; optional)
│   │   ├── scene01.mesh.ply         #     collision mesh (will not be used unless physics is enabled)
│   │   └── scene01.navmesh          #     navigation mesh
│   ├── scene02/ ... scene55/
│   ├── interior_0007_840137/        #   InteriorGS (55 scenes): 3DGS and navmesh only
│   │   ├── interior_0007_840137.gs.ply
│   │   └── interior_0007_840137.navmesh
│   └── interior_0022_840117/ ... ×55
├── val/                             # 19 evaluation scenes
│   ├── scene56/ ... scene65/
│   └── interior_0516_840045/ ... ×9
├── configs/                         # Hydra YAML configs (provided)
│   ├── ddppo_pointnav_gs_train.yaml
│   ├── ddppo_pointnav_gs_eval.yaml
│   ├── ddppo_imagenav_gs_train.yaml
│   ├── ddppo_imagenav_gs_eval.yaml
│   ├── ddppo_objectnav_gs_train.yaml
│   └── ddppo_objectnav_gs_eval.yaml
└── episodes/                        # generated by scripts below
    ├── pointnav/{train,val}/
    ├── imagenav/{train,val}/
    └── objectnav/{train,val}/

ObjectNav uses SAM (Segment Anything) + CLIP (zero-shot classification) to automatically detect and classify objects in GS scenes, then generates navigation episodes to those objects.

Required model checkpoints:

Object categories:

All training scripts accept the same options:

--output DIR             Output directory for checkpoints and tensorboard (required)
--num-envs N             Number of parallel environments per GPU (default: 4)
--num-gpus N             Number of GPUs for DDPPO (default: 1)
--total-steps N          Total training steps
--num-ckpts N            Number of checkpoints to save (default: 100)
--pretrained-ckpt PATH   Fine-tune from an existing .pth checkpoint
                         (this sets ddppo.pretrained=True; critic is re-initialised)

Extra arguments are forwarded as Hydra overrides. Example with multi-GPU:

bash scripts_gs/train_objectnav.sh \
    --output output/objectnav \
    --num-envs 8 \
    --num-gpus 4

Fine-tuning from a previously trained checkpoint:

bash scripts_gs/train_pointnav.sh \
    --output output/pointnav_ft \
    --pretrained-ckpt output/pointnav/checkpoints/ckpt.99.pth

By default this loads the whole policy (encoder + RNN + actor head) and re-initialises the critic head — appropriate when continuing on the same task or transferring to a closely related one. To customise the load behaviour, append Hydra overrides, e.g.:

# keep the trained critic
bash scripts_gs/train_pointnav.sh \
    --output output/pointnav_ft \
    --pretrained-ckpt output/pointnav/checkpoints/ckpt.99.pth \
    habitat_baselines.rl.ddppo.reset_critic=False

# load only the visual encoder backbone (e.g. transfer from PointNav to ImageNav)
bash scripts_gs/train_imagenav.sh \
    --output output/imagenav_ft \
    --pretrained-ckpt output/pointnav/checkpoints/ckpt.99.pth \
    habitat_baselines.rl.ddppo.pretrained=False \
    habitat_baselines.rl.ddppo.pretrained_encoder=True \
    habitat_baselines.rl.ddppo.train_encoder=False    # freeze the encoder

Note: optimizer state, step counter and seeds are reset — this is fine-tuning, not resume. To resume an interrupted run, just re-launch with the same --output directory; habitat-baselines auto-detects .resume_state.pth and continues seamlessly.

Output structure:

output/objectnav/
├── checkpoints/    # .pth checkpoint files
├── tb/             # TensorBoard logs
└── train.log       # training log

--ckpt PATH         Path to a .pth checkpoint file or a directory of checkpoints (required)
--num-envs N        Number of parallel environments (default: 1)
--video-dir DIR     Directory to save evaluation rollout videos (optional)

Pass --ckpt a directory to evaluate all checkpoints in it sequentially.

data/scene_datasets/gs_scenes/
├── configs/
│   └── vln_gs_eval.yaml             # habitat config for VLN evaluation (provided)
├── episodes/
│   └── vln/                         # generated by generate_vln_episodes.py
│       ├── train/train.json.gz      # 110 scenes × 200 episodes = 22,000 train
│       └── val/val.json.gz          # 19 scenes × 50 episodes = 950 val
└── trajectory_data/
    └── vln/                         # generated by generate_vln_trajectories.py
        ├── annotations.json         # StreamVLN-format action sequences
        └── images/{scene}_gs_{ep_id}/rgb/*.jpg   # rendered RGB frames

The script applies scripts_gs/streamvln_compat.patch to the StreamVLN clone (4 files, +42/−13 lines) so that:

• streamvln/habitat_extensions/measures.py works with habitat-lab 0.3.3 (which removed try_cv2_import)
• streamvln/streamvln_train.py honors a --vision_tower CLI override for local model paths, fixes low_cpu_mem_usage for quantized loading, removes duplicate quantization kwargs, and adds a tokenizer-loading fallback for merged LoRA checkpoints
• streamvln/streamvln_eval.py explicitly loads the SigLIP vision tower (fixes delay_load issue) and auto-selects flash_attention_2 with eager fallback
• llava/model/multimodal_encoder/siglip_encoder.py passes low_cpu_mem_usage=True for device_map-based loading

The script is idempotent — re-running setup_vln.sh is safe. Available flags:

--skip-env          Skip creating the conda environment
--skip-patch        Skip applying the compat patch
--skip-download     Skip downloading model checkpoints
--skip-deps         Skip installing Python dependencies
--hf-token TOKEN    HuggingFace token for gated models

generate_vln_episodes.py produces 200 episodes per training scene and 50 per evaluation scene by default, in the standard R2RVLN-v1 format consumed by habitat-lab's vln_r2r task. Instructions are generated by querying an OpenAI-compatible VLM endpoint with multi-view renderings along the path. Configure the endpoint via OPENAI_BASE_URL + OPENAI_API_KEY environment variables, or pass --api-config /path/to/config.json.

generate_vln_trajectories.py runs a greedy heading-based path follower (forward_step=0.25m, turn_angle=15°, success_distance=0.25m for the final waypoint) on each episode and records:

• annotations.json — one entry per episode with the instruction, the action sequence (-1=initial, 0=stop, 1=forward, 2=turn-left, 3=turn-right), and per-step poses
• images/{scene}_gs_{ep_id}/rgb/*.jpg — RGB frame at each step, rendered through the GS pipeline

The script supports --resume to skip already-completed scenes, which is useful if generation is interrupted.

--output DIR        Output directory for checkpoints (required)
--stage STAGE       Training stage: stage-one | dagger | stage-two (default: stage-one)
--num-gpus N        Number of GPUs (default: 1)
--ckpt PATH         Base checkpoint (default: local LLaVA-Video-7B-Qwen2 for stage-one)
--epochs N          Number of epochs (default: 1)
--batch-size N      Per-device batch size (default: 2)
--grad-accum N      Gradient accumulation steps (default: 2)
--lr RATE           Learning rate (default: 2e-5)
--num-frames N      Frames per sample (default: 32)
--lora              Enable LoRA mode (see below)

Standard mode (default): full fine-tune of the entire model (~8 GB trainable parameters) with anyres_max_9 image tiling, 32 frames, 32K context, and torch.compile. Multi-GPU training uses DeepSpeed ZeRO-2. Requires ≥80 GB VRAM per GPU (A100 80GB recommended).

LoRA mode (--lora): freezes the LLM backbone, trains only the MM projector + LoRA adapters (r=64, alpha=128, ~17 MB trainable), reduces frames to 4 and context to 2K. Fits on a single 24GB RTX 4090. When chaining stages with --lora (e.g. stage-one → dagger), the script auto-merges the previous LoRA checkpoint before applying new adapters.

--ckpt PATH         Path to a trained StreamVLN checkpoint (required)
--output DIR        Output directory for results (default: results/vln/<ckpt>_<split>)
--num-gpus N        Number of GPUs for parallel rollout (default: 1)
--split SPLIT       Evaluation split: train | val (default: val)
--num-frames N      Frames per sample (default: 32)
--save-video        Save visualization videos

The evaluator uses data/scene_datasets/gs_scenes/configs/vln_gs_eval.yaml (RGB+Depth at 640x480, hfov=79°, forward_step=0.25m, turn_angle=15°, success_distance=3.0m, max_episode_steps=500) and reports the standard VLN metrics: Success, SPL, Oracle Success, Distance-to-Goal, and Oracle Navigation Error.

data/scene_datasets/gs_scenes/
├── configs/
│   └── vln_uninavid_gs_eval.yaml    # habitat config for Uni-NaVid eval (provided)
├── episodes/
│   └── vln/                         # shared with StreamVLN
│       ├── train/train.json.gz      # 110 scenes × 200 episodes = 22,000 train
│       └── val/val.json.gz          # 19 scenes × 50 episodes = 950 val
└── trajectory_data/
    └── uninavid/                    # generated by generate_uninavid_trajectories.py
        ├── nav_gs_train.json        # Uni-NaVid conversation-format annotations
        ├── nav_gs_val.json
        └── nav_videos/              # .mp4 trajectory videos
            ├── scene01_gs_000000.mp4
            ├── interior_0007_840137_gs_000000.mp4
            └── ...

--skip-env          Skip creating the conda environment
--skip-deps         Skip installing Python dependencies
--skip-patch        Skip applying uninavid_compat.patch
--skip-download     Skip downloading model checkpoints
--proxy URL         HTTP proxy for downloads

The greedy controller uses forward_step=0.25m, turn_angle=30°, HFOV=120°. Each trajectory video is encoded at 10 fps. The output JSON uses Uni-NaVid's conversation format with NAV_ID prefix and NAVIGATION_IDENTIFIER string to trigger navigation-specific token processing during training.

--output DIR         Output directory for checkpoints (required)
--stage STAGE        Training stage: stage-1|stage-2 (default: stage-2)
--num-gpus N         Number of GPUs (default: 1)
--ckpt PATH          Base checkpoint path (auto-selected per stage)
--epochs N           Number of epochs (default: 1)
--batch-size N       Per-device batch size (default: 8)
--grad-accum N       Gradient accumulation steps (default: 2)
--lr RATE            Learning rate (default: 1e-5)

--ckpt PATH         Path to trained Uni-NaVid checkpoint (required)
--output DIR        Output directory for results (default: results/uninavid/<ckpt>_<split>)
--num-gpus N        Number of GPUs for parallel evaluation (default: 1)
--split SPLIT       Evaluation split: train|val (default: val)
--save-video        Save evaluation rollout videos

The evaluator reports: Success Rate (SR), SPL, Oracle Success (OSR), and Distance-to-Goal (DTG).

data/scene_datasets/gs_scenes/
├── configs/
│   ├── ddppo_dynamic_track_gs_{train,eval}.yaml            # tracking
│   ├── ddppo_dynamic_avoid_gs_{train,eval}.yaml            # dynamic PointNav
│   ├── ddppo_dynamic_avoid_imagenav_gs_{train,eval}.yaml   # dynamic ImageNav
│   └── ddppo_dynamic_avoid_objectnav_gs_{train,eval}.yaml  # dynamic ObjectNav
└── dynamic_nav/                         # 10 scenes (scene01–scene10), one walking avatar each
    ├── dynamic_nav.scene_dataset_config.json
    ├── scenes/<scene>.scene_instance.json      # stage + navmesh + gaussian_avatars wiring
    ├── stages/<scene>.stage_config.json
    ├── trajectories/<scene>.driver.pkl         # GAMMA walk (joint_mats + proxy_capsules)
    ├── episodes/{train,val}/                   # PointNav format: 1,000 train + 100 val
    │                                           #   (shared by avoid / avoid_imagenav / track)
    └── episodes_objectnav/{train,val}/         # ObjectNav format: 1,000 train + 100 val