# v2d_hoi_object_reconstruction

End-to-end textured 3D mesh reconstruction from hand-object interaction video.

Two reconstruction modes:
- **BundleSDF** (default) — two-stage scan (stationary → rotated → stationary) → full textured NeRF mesh
- **SAM3D** — default two-stage or explicit stationary-object capture →
  representative single-image meshes → silhouette-based scale estimation

For downstream rigid USD generation and an Isaac Sim drop test, see
[`mesh_to_usd/README.md`](mesh_to_usd/README.md).

---

## Agentic Workflow

Repository-local agent support is split into three focused skills, matching the
setup/run/doctor pattern used by other modules:

| Skill | Use it for |
|---|---|
| [`hoi-object-reconstruction-setup`](../../../.claude/skills/hoi-object-reconstruction-setup/SKILL.md) | Host packages, Docker/GPU checks, images, weights, and input preflight |
| [`hoi-object-reconstruction-run`](../../../.claude/skills/hoi-object-reconstruction-run/SKILL.md) | Launching, monitoring, resuming, and verifying BundleSDF or SAM3D |
| [`hoi-object-reconstruction-doctor`](../../../.claude/skills/hoi-object-reconstruction-doctor/SKILL.md) | Diagnosing the first bad setup, input, pipeline, or output stage |

The run skill is action-oriented. Unless a command-only response is requested,
it launches the pipeline rather than stopping after a questionnaire. Missing
values use these safe defaults: BundleSDF, GPU 0, a fresh output directory, and
the included basketball capture with prompt `basketball`. Custom paths, modes,
prompts, GPUs, and output directories always take precedence.

Start the agent from the repository checkout. Examples:

```text
/hoi-object-reconstruction-setup Prepare BundleSDF on GPU 0.

/hoi-object-reconstruction-run Run the included basketball example with
BundleSDF on GPU 0, monitor it, and verify the mesh and alignment video.

/hoi-object-reconstruction-run Run SAM3D on
/absolute/path/to/mapping_data with prompt "toy airplane" and write a fresh job
under /absolute/path/to/outputs.

/hoi-object-reconstruction-run Run SAM3D on this arbitrary-path scan; the
object remains stationary throughout.

/hoi-object-reconstruction-doctor Diagnose the failed job at
/absolute/path/to/job and give me the narrowest resume command.
```

Before a long GPU job, setup and run use a deterministic input check. It can
also be run manually from `reconstruction/`:

```bash
python ../.claude/skills/hoi-object-reconstruction-setup/scripts/preflight_input.py \
  /absolute/path/to/mapping_data
```

A completed agentic run must still report the source revision, exact command,
input and output paths, selected mode and GPU, final artifact paths, and visual
QA result. A zero exit code or a generated GLB alone is not sufficient.

These skills end at a reviewed reconstruction GLB. Mesh-to-USD conversion and
physics validation are separate downstream workflows and must have their own
output directory and status. Their dedicated setup/run/doctor skills are
documented in the [mesh-to-USD agentic workflow](mesh_to_usd/README.md#agentic-workflow).

---

## Data Collection

### Capture hardware

The reference data was collected with a backpack-based capture rig whose main
components are a **Hawk stereo camera** and an **NVIDIA Orin** compute and
recording unit. This hardware is not a pipeline requirement. Input may originate
from stereo videos, image sequences, or another capture format as long as it is
converted to synchronized left/right JPEG keyframes and calibration metadata in
the HOI reconstruction layout described in
[Input Data Format](#input-data-format).

### Preparing the input dataset

The base image and metadata format is defined by public
[PyCuSFM](https://github.com/nvidia-isaac/pyCuSFM). This module consumes a
stricter, single-stereo-pair profile of PyCuSFM's `frames_meta.json` contract.
See the public [PyCuSFM input
tutorial](https://github.com/nvidia-isaac/pyCuSFM/blob/main/docs/tutorial.md#raw-data-requirements)
for the underlying `KeyframesMetadataCollection` format.

For source videos or another capture format:

1. Extract synchronized left and right frames as JPEG images under
   `front_stereo_camera_left/` and `front_stereo_camera_right/`. Preserve
   nanosecond capture timestamps in the filenames when available.
2. Create a PyCuSFM generator YAML containing the real image dimensions,
   rectified pinhole intrinsics, camera-to-rig transforms, sensor names, and
   stereo baseline. Do not infer or copy calibration from the example dataset.
3. Use PyCuSFM's public [keyframe metadata
   generator](https://github.com/nvidia-isaac/pyCuSFM/blob/main/pycusfm/generate_frame_meta.py)
   in stereo-images mode:

```bash
python pycusfm/generate_frame_meta.py \
  --images /path/to/mapping_data_dir \
  --config /path/to/camera_config.yaml \
  --output /path/to/mapping_data_dir/frames_meta.json
```

Add `--use-pseudo-timestamps` when synchronized image sequences have no capture
timestamps. The generator assigns synchronization IDs and emits the CuSFM
metadata structure. For rectified pinhole input, empty or zero distortion
coefficients cause it to emit the 3×4 `projection_matrix` required by this HOI
pipeline.

Finally, validate the generated JSON against
[`schemas/frames_meta.schema.json`](schemas/frames_meta.schema.json), check the
cross-field invariants below, and compare the directory structure with
[`assets/basketball_example/`](assets/basketball_example/). A custom converter
is also compatible if it produces the same contract with correct
synchronization and calibration.

Before a scan, verify that both stereo feeds are clear and synchronized, the
recording frame rate is stable (the reference collection uses 30 FPS), and the
capture system has enough free storage.

### Object scanning guidelines

Choose a rigid, opaque object with matte surfaces and visible texture whenever
possible. Transparent or reflective materials, nearly textureless surfaces,
very dark objects, and objects that deform while handled are substantially more
difficult to reconstruct. Use diffuse, stable lighting and a stationary,
feature-rich background; clean the object and make sure it can rest securely in
both scan orientations.

BundleSDF and the default SAM3D capture mode use one continuous two-round
recording to obtain six-face coverage:

1. Place the object near the center of the capture area with approximately
   20–30 cm of clearance. Keep it fully in frame, typically from a distance of
   50–100 cm.
2. Walk slowly through a full 360-degree pass at roughly the object's
   mid-height. Keep the camera aimed at the object, maintain a consistent
   distance, and target 50–70% overlap between consecutive views. A typical
   pass takes 20–40 seconds.
3. Without stopping the recording, rotate the object about 90 degrees to expose
   its previously hidden bottom face. Brief hand occlusion is expected; touch
   edges or the base where possible and keep most of the object and background
   visible.
4. Complete a second 360-degree pass around the new orientation, concentrating
   on the newly exposed face and its adjacent surfaces. Confirm that all six
   faces and any concave or easily missed areas have coverage, then stop and
   verify the capture was saved.

Move smoothly to limit motion blur and avoid abrupt changes in distance or
lighting.

SAM3D also supports a stationary-object capture. Keep the object fixed for the
entire recording and move the stereo camera smoothly through diverse azimuth
and elevation viewpoints. The path does not need to be planar, circular, or
ordered into loops, but the object must remain visible and the views must have
enough overlap for CuSFM. Run this contract explicitly with
`--mode sam3d --sam3d_capture_mode stationary`. It does not silently replace a
failed two-stage scan, and it cannot observe a bottom face that remains hidden.

---

## Input Data Format

`--mapping_data_dir` must point to a synchronized stereo dataset in this layout:

```text
mapping_data_dir/
├── frames_meta.json                         # required
├── frame_metadata.jsonl                     # optional compatibility metadata; not consumed
├── front_stereo_camera_left/
│   └── <timestamp_ns>.jpeg
└── front_stereo_camera_right/
    └── <timestamp_ns>.jpeg
```

The bundled [`assets/basketball_example/`](assets/basketball_example/) is the
reference dataset (203 synchronized stereo pairs at 960×600). JPEG is the
known-good input format. Metadata image paths are relative to
`mapping_data_dir`.

The reference directory also contains a `stereo.edex` source artifact, but it
is not part of this module's input contract and the current reconstruction path
does not read it. `frames_meta.json` is the calibration source of truth: its
camera entries provide the rectified projection matrices and image dimensions,
and its first `stereo_pair` entry provides the baseline.

This is a **CuSFM-compatible HOI stereo profile**. CuSFM determines the core
`frames_meta.json` representation, relative image paths, timestamp units,
camera parameter map, and synchronization IDs. The HOI wrapper additionally
requires one usable stereo pair, the sensor names shown below, rectified 3×4
projection matrices, and a positive baseline. JPEG is the currently tested
image format for the complete HOI pipeline even though CuSFM itself accepts
additional image formats.

### `frames_meta.json` (required)

This file is the pipeline's input contract:

| Field | Required | Description |
|-------|----------|-------------|
| `keyframes_metadata` | yes | One entry per camera image: camera ID, synchronized sample ID, relative image path, and timestamp in microseconds. |
| `camera_params_id_to_camera_params` | yes | Camera definitions keyed by ID, including sensor name, image dimensions, and calibration matrices. |
| `stereo_pair` | yes | Left/right camera IDs and physical stereo baseline in meters. The pipeline uses the first pair. |
| `initial_pose_type` | yes | CuSFM pose interpretation, normally `"EGO_MOTION"` for sequential stereo input. |
| `camera_params_id_to_session_name` | no | Producer/session metadata; not interpreted here. |

Minimal structural example (projection values abbreviated):

```json
{
  "initial_pose_type": "EGO_MOTION",
  "keyframes_metadata": [
    {
      "id": "0",
      "camera_params_id": "0",
      "synced_sample_id": "5",
      "image_name": "front_stereo_camera_left/1771445398921412053.jpeg",
      "timestamp_microseconds": "1771445398921412"
    },
    {
      "id": "1",
      "camera_params_id": "1",
      "synced_sample_id": "5",
      "image_name": "front_stereo_camera_right/1771445398921412053.jpeg",
      "timestamp_microseconds": "1771445398921412"
    }
  ],
  "camera_params_id_to_camera_params": {
    "0": {
      "sensor_meta_data": {"sensor_name": "front_stereo_camera_left"},
      "calibration_parameters": {
        "image_width": 960,
        "image_height": 600,
        "projection_matrix": {
          "row_count": 3,
          "column_count": 4,
          "data": [426.2, 0, 473.2, 0, 0, 426.2, 278.4, 0, 0, 0, 1, 0]
        }
      }
    },
    "1": {
      "sensor_meta_data": {"sensor_name": "front_stereo_camera_right"},
      "calibration_parameters": {
        "image_width": 960,
        "image_height": 600,
        "projection_matrix": {
          "row_count": 3,
          "column_count": 4,
          "data": [426.2, 0, 473.2, 0, 0, 426.2, 278.4, 0, 0, 0, 1, 0]
        }
      }
    }
  },
  "stereo_pair": [{
    "left_camera_param_id": "0",
    "right_camera_param_id": "1",
    "baseline_meters": 0.14956
  }]
}
```

Important invariants:

- Left and right entries pair by equal `synced_sample_id`; incomplete pairs are
  dropped.
- Camera IDs in frame and stereo-pair entries must reference keys in
  `camera_params_id_to_camera_params`.
- Sensor names must be `front_stereo_camera_left` and
  `front_stereo_camera_right`; current frame indexing relies on these names.
- `timestamp_microseconds` is a decimal string. Timestamp-based filenames use
  nanoseconds, but `image_name` is the authoritative path.
- `projection_matrix.data` is a row-major 3×4 rectified projection matrix. The
  pipeline reads `fx`, `fy`, `cx`, and `cy` from indices 0, 5, 2, and 6.
- `baseline_meters` is in meters and must be positive.

See the machine-readable
[`schemas/frames_meta.schema.json`](schemas/frames_meta.schema.json). JSON
Schema cannot verify cross-field references, stereo pairing, image existence,
or calibration accuracy.

### `frame_metadata.jsonl` (optional compatibility metadata)

The current `run_reconstruction.py` path does not read this file, but some
dataset producers and older tools emit it. Each non-empty line is one
synchronized-frame JSON record:

```json
{"frame_id":0,"cams":[{"id":0,"filename":"front_stereo_camera_left/1771445398921412053.jpeg","timestamp":1771445398921412000},{"id":1,"filename":"front_stereo_camera_right/1771445398921412053.jpeg","timestamp":1771445398921412000}]}
```

`timestamp` is in nanoseconds. Validate each line independently against
[`schemas/frame_metadata_record.schema.json`](schemas/frame_metadata_record.schema.json).
A JSONL file as a whole is not a single JSON value.

---

## Environment Setup

### GPU compatibility

The v0.2 containers require NVIDIA Container Toolkit and an NVIDIA GPU with at
least 24 GB of VRAM. Development and local validation used RTX A6000 and L40S
GPUs. VRAM capacity alone does not indicate compatibility.

Blackwell GPUs with compute capability 12.0 (`sm_120`), including the NVIDIA
RTX PRO 6000 Blackwell, are not supported by the TensorRT and cuVSLAM versions
in `v2d_cusfm` or by all current native CUDA extensions. Without the host
preflight, CuSFM can fail while building a TensorRT engine or another stage may
report `no kernel image is available for execution on the device`.

`scripts/build_containers.sh` checks the host architecture before starting the
container builds, and `run_reconstruction.py` checks it again before launching
the pipeline. Use a validated GPU for v0.2. The build-only
`--skip-gpu-preflight` escape hatch is intended for cross-host image builds;
the runtime check remains mandatory. Blackwell support requires upgrading and
validating the `v2d_cusfm` TensorRT/cuVSLAM stack and rebuilding native CUDA
extensions with `sm_120` targets.

### 1. Install host packages (from `reconstruction/`)

Activate the Python 3.10+ environment used for this source checkout, then run
the repository-wide package installer:

```bash
./scripts/install_packages.sh
```

If you do not already have an isolated environment, create one first with
`python3 -m venv .venv` and `source .venv/bin/activate`. Do not install
Matplotlib or SciPy manually to make this workflow pass. BundleSDF and SAM3D
numerical work—including scan checks, frame selection, SRT optimization, mask
processing, and media encoding—runs in containers.

### 2. Build all required containers

```bash
./scripts/build_containers.sh
```

The `v2d_sam3d` image built from this tree includes the EGL/GLVND loader and
Pyrender required for headless overlay-video rendering. Rebuild that image
after changing `v2d_sam3d` code or Docker dependencies; model weights remain a
separate runtime download.

The release-image prerequisite check is:

```bash
docker run --rm --gpus all v2d_sam3d:latest \
  python -c 'import pyrender; r=pyrender.OffscreenRenderer(64,64); \
print("egl_renderer=pass"); r.delete()'
```

It must print `egl_renderer=pass`.

### 3. Download model weights

```bash
# Shared (both modes)
python -m v2d.sam2.docker.run_download_weights --output_dir data/weights/sam2
python -m v2d.grounding_dino.docker.run_download_weights --output_dir data/weights/grounding_dino

# BundleSDF mode
python modules/v2d_foundation_stereo/docker/run_download_weights.py \
  --output_dir data/weights/foundationstereo
python modules/v2d_foundation_pose/docker/run_download_weights.py \
  --output_dir data/weights/foundationpose
python modules/v2d_bundlesdf/docker/run_download_weights.py \
  --output_dir data/weights

# SAM3D mode
python modules/v2d_sam3d/docker/run_download_weights.py \
  --output_dir data/weights/sam3d
# (optional: FoundationStereo for depth-assisted scale estimation)
python modules/v2d_foundation_stereo/docker/run_download_weights.py \
  --output_dir data/weights/foundationstereo
```

---

## BundleSDF Pipeline

Two-stage scan: object stationary → rotated 360° → stationary again.

### Quick Start

```bash
python modules/v2d_hoi_object_reconstruction/docker/run_reconstruction.py \
  --mapping_data_dir data/hoi_obj_recon/raw_data/<job> \
  --job_dir          data/outputs/hoi_recon/<job> \
  --prompt           "basketball"
```

### Pipeline Steps

```mermaid
flowchart TD
    A["Calibrated stereo input<br/>images + frames_meta.json"] --> B["Prepare job<br/>left/right frames, calibration, video"]
    B --> C["CuSFM + scan QA<br/>camera poses and stage boundary"]
    C --> D["Grounding DINO + SAM2<br/>object boxes and masks"]
    C --> E["FoundationStereo<br/>stereo depth"]
    D --> F["Stage-1 BundleSDF<br/>initial textured mesh"]
    E --> F
    F --> G["FoundationPose<br/>track the initial mesh"]
    G --> H["World-pose alignment<br/>merge stationary and rotated stages"]
    H --> I["Final BundleSDF<br/>merged_recon/output.glb"]
    I --> J["Final overlay video<br/>fp_render_final/render.mp4"]
```

### Results

- `merged_recon/textured_mesh.obj` — final textured mesh (+ `.mtl`, `_0.png`)
- `merged_recon/output.glb` — self-contained GLB exported from the final textured mesh
- `stage1_recon/textured_mesh.obj` — Stage-1 mesh (bottom missing)
- `stage1_recon/output.glb` — self-contained GLB exported from the Stage-1 mesh

### Key Options

| Flag | Default | Description |
|------|---------|-------------|
| `--stage1_end_frame N` | auto | Override Stage-1 end (sequential frame index) |
| `--stage1_end_timestamp NS` | auto | Override Stage-1 end (nanosecond timestamp) |
| `--stage1_buffer_deg DEG` | 10.0 | Angle buffer before detected transition |
| `--config PATH` | bundlesdf default | NeRF/SDF config YAML |
| `--pipeline_config PATH` | `docker/data/configs/hoi_pipeline.yaml` | Pipeline config YAML |
| `--reference_frame N` | 0 | SAM2/FP reference frame |
| `--num_depth_workers N` | 2 | Parallel FoundationStereo workers |
| `--fp_weights_dir PATH` | `data/weights/foundationpose` | FoundationPose weights |

### Skip Flags (BundleSDF)

Resume from any checkpoint by skipping completed steps:

```
--skip_prepare  --skip_sfm  --skip_sfm_quality_check  --skip_stage1_detect  --skip_dino  --skip_depth  --skip_mask
--skip_stage1_setup  --skip_stage1_nerf  --skip_center_mesh
--skip_fp_tracking  --skip_fp_render  --skip_world_poses  --skip_merged_setup  --skip_full_nerf
--skip_glb_export  --skip_final_fp_tracking  --skip_final_fp_render
```

### Configuration

**Pipeline config** — `docker/data/configs/hoi_pipeline.yaml` (override with `--pipeline_config`):

| Section | Parameter | Default | Description |
|---------|-----------|---------|-------------|
| `stage1_detect` | `buffer_deg` | 10.0 | Angle buffer (°) before detected Stage-1 transition |
| `sfm` | `config_set` | `backpack` | CuSFM preset (`backpack` \| `av` \| `isaac` \| `rgbd`) |
| `sfm_scan_quality` | `enabled` | `true` | Validate CuSFM pose count and continuity; also validate the two-loop contract when applicable |
| `sfm_scan_quality` | `min_angle_span_deg` | 600.0 | Minimum projected orbit span in `two_stage` mode; not applied to stationary mode |
| `sfm_scan_quality` | `max_backtracking_fraction` | 0.25 | Maximum reverse angular motion fraction in `two_stage` mode; not applied to stationary mode |
| `sfm_scan_quality` | `max_translation_step_m` | 2.0 | Maximum allowed consecutive CuSFM translation jump |
| `depth` | `num_workers` | 1 | FoundationStereo depth workers; increase explicitly for multi-GPU hosts |
| `foundationpose` | `reference_frame` | 0 | FP registration reference frame |
| `foundationpose` | `weights_dir` | `null` | FP weights path; `null` = `data/weights/foundationpose` |

**NeRF/SDF config** — `modules/v2d_bundlesdf/lib/data/configs/theseus_optimizer_hawk.yaml` (override with `--config`):

| Section | Parameter | Default | Description |
|---------|-----------|---------|-------------|
| `camera_config` | `step` | 4 | CuSFM keyframe subsampling for SDF training |
| `nerf` | `n_step` | 3000 | SDF training steps |
| `nerf` | `far` | `auto` | SDF training depth range, resolved from masked object depth |
| `nerf` | `trunc` | 0.01 | TSDF truncation distance (normalized). Larger = fewer holes |
| `nerf` | `mesh_resolution` | 0.005 | Voxel size for mesh extraction. Must be ≤ `trunc` |
| `texture_bake` | `texture_res` | 2048 | Output texture atlas resolution |
| `texture_bake` | `zfar` | `auto` | Texture-renderer clipping plane, resolved from camera distance |
| `texture_bake` | `downscale` | 1.0 | Image downscale for texture baking |
| `texture_bake` | `min_keyframe_translation` | 0.0 | Min camera translation (m) between texture keyframes |
| `texture_bake` | `min_keyframe_rotation_deg` | 5.0 | Min camera rotation (°) between texture keyframes |
| `texture_bake` | `min_keyframes` | 30 | Minimum keyframes after subsampling |

---

## SAM3D Pipeline

Single-image 3D reconstruction per representative frame. The existing
two-stage/two-loop capture remains the default. A separate stationary mode
supports an arbitrary camera path when the object does not move.

### Quick Start

```bash
python modules/v2d_hoi_object_reconstruction/docker/run_reconstruction.py \
  --mapping_data_dir data/hoi_obj_recon/raw_data/<job> \
  --job_dir          data/outputs/hoi_recon/<job> \
  --prompt           "basketball" \
  --mode sam3d
```

With depth-assisted scale estimation:

```bash
python modules/v2d_hoi_object_reconstruction/docker/run_reconstruction.py ... \
  --mode sam3d --sam3d_use_depth
```

For a stationary object and a non-loop or otherwise arbitrary camera path:

```bash
python modules/v2d_hoi_object_reconstruction/docker/run_reconstruction.py \
  --mapping_data_dir data/hoi_obj_recon/raw_data/<job> \
  --job_dir          data/outputs/hoi_recon/<job> \
  --prompt           "basketball" \
  --mode sam3d \
  --sam3d_capture_mode stationary
```

Choose the mode explicitly from the capture procedure. `two_stage` requires a
detected or supplied Stage-1 boundary. `stationary` skips transition detection,
selects source frames by camera viewing-direction diversity, and makes the full
sequence eligible for SRT alignment. It never automatically falls back from a
failed two-stage capture. Object stationarity is a capture-procedure contract;
the pipeline records the assumption but does not independently prove that the
object never moved.

### Pipeline Steps

```mermaid
flowchart TD
    A["Calibrated stereo input<br/>images + frames_meta.json"] --> B["Prepare job<br/>left/right frames, calibration, video"]
    B --> C["CuSFM + capture-aware scan QA<br/>camera poses"]
    C --> C2["Two-stage: detect boundary<br/>Stationary: use full sequence"]
    C2 --> D["Grounding DINO + SAM2<br/>object boxes and masks"]
    D --> E["Select representative views<br/>sam3d/selected_frames.json"]
    E --> F["SAM3D per selected frame<br/>mesh.glb + camera transform"]
    F --> G["SRT scale and pose<br/>output_scaled.glb"]
    G --> H["Diagnostic overlays<br/>render_debug.jpg + render_video.mp4"]
    H --> I["Rank candidates<br/>best/best_frame.json"]
    I --> J["Suggested final mesh<br/>best/output_scaled.glb"]
    C -.-> K["Optional FoundationStereo<br/>depth-assisted SRT"]
    K -.-> G
```

### Results

- `sam3d/<frame_id>/mesh.glb` — raw SAM3D mesh (SAM3D camera space)
- `sam3d/<frame_id>/srt/output_scaled.glb` — scale-corrected mesh (world space)
- `sam3d/<frame_id>/srt/srt_result.json` — estimated scale, rotation, translation
- `sam3d/<frame_id>/render_debug.jpg` — SAM3D mesh overlaid on source image (single frame)
- `sam3d/<frame_id>/render_video.mp4` — mesh overlaid on Stage-1 (`two_stage`) or full-sequence (`stationary`) keyframes
- `sam3d/capture_contract.json` — requested capture mode, object-motion assumption, and stage boundary
- `sam3d/selection_report.json` — frame-selection method and selected-frame provenance
- `sam3d/best/best_frame.json` — ranked candidate frames and selection score
- `sam3d/best/output_scaled.glb` — copied suggested best aligned mesh

### Key Options

| Flag | Default | Description |
|------|---------|-------------|
| `--sam3d_capture_mode {two_stage,stationary}` | `two_stage` | Capture contract; stationary mode requires the object to remain fixed throughout |
| `--sam3d_use_depth` | off | Use FoundationStereo depth as extra loss in SRT scale estimation |
| `--sam3d_bin_deg DEG` | 60.0 | Two-stage mode: cumulative orbit-angle bin size for frame selection |
| `--sam3d_stationary_views N` | 6 | Stationary mode: number of pose-diverse SAM3D source views |
| `--sam3d_seed N` | 42 | Random seed for SAM3D inference |
| `--sam3d_srt_max_views N` | 25 | Maximum silhouette views per SRT candidate |
| `--sam3d_srt_maxiter N` | 60 | Powell iterations per SRT optimisation |
| `--sam3d_srt_top_k N` | 1 | Axis orientations fully optimised when no SAM3D orientation prior is available |
| `--sam3d_srt_parallel N` | 8 | Candidate-level SRT workers; each worker caps native numerical threads to one |
| `--sam3d_force_srt` | off | Recompute candidates that already have a valid result and scaled mesh |

The SRT defaults match the OSMO fast path. Candidates are independent and run
in parallel. A resumed run automatically reuses a candidate when both
`srt_result.json` and `output_scaled.glb` are valid; use `--sam3d_force_srt` to
deliberately recompute them. The underlying library retains its higher-cost
100-view, 120-iteration, top-5 defaults for direct callers that need the full
accuracy reference.

### Skip Flags (SAM3D)

```
--skip_prepare  --skip_sfm  --skip_sfm_quality_check  --skip_stage1_detect  --skip_dino  --skip_depth  --skip_mask
--skip_select_frames  --skip_sam3d  --skip_srt_scale  --skip_render_debug  --skip_render_video
--skip_select_best_mesh
```

---

## Quality Envelope and Limitations

### Reconstruction Trade-offs

#### Reference Input and Final Outputs

The reference input shows representative left-camera views from a calibrated
stereo scan. The airplane is stationary during the first scan stage and is then
rotated so the workflows can observe additional surfaces.

![Toy-airplane reference input views](assets/expected_outputs/evt04_airplane/input_reference.gif)

The turntables below were rendered from the final GLB produced by each
workflow. They illustrate the expected method trade-offs; visual QA should use
the acceptance criteria below rather than require an exact match to these
examples.

| BundleSDF | SAM3D |
|:---------:|:-----:|
| ![BundleSDF final toy-airplane mesh turntable](assets/expected_outputs/evt04_airplane/bundlesdf.gif) | ![SAM3D final toy-airplane mesh turntable](assets/expected_outputs/evt04_airplane/sam3d.gif) |
| `merged_recon/output.glb` | `sam3d/best/output_scaled.glb` |

BundleSDF and SAM3D optimize for different outcomes; their final meshes should
not be reviewed against the same visual standard.

| Mode | Final output | Strength | Expected limitations |
|------|--------------|----------|----------------------|
| BundleSDF | `merged_recon/output.glb` | Reconstructs observed geometry with metric scale | Thin or partially observed parts can be missing or thickened. Small holes or fragments and texture bleeding between nearby parts can occur. |
| SAM3D | `sam3d/best/output_scaled.glb` | Usually produces a more visually complete shape and texture from a representative frame | Unseen geometry is generated rather than reconstructed, and shape dimensions and estimated metric scale can be less reliable. |

#### Reference Performance

These measurements are a reproducibility reference, not a performance SLA.
They exclude container builds and model-weight downloads. Runtime varies with
the GPU, driver, input size, selected views, and whether completed stages can be
resumed.

| Workflow | Input / configuration | End-to-end runtime | Result |
|----------|-----------------------|--------------------|--------|
| BundleSDF | 2,433 stereo pairs, default two-stage workflow | 5,986 s (~100 min) | `merged_recon/output.glb` and final overlay produced |
| SAM3D | Same scan, 13 selected frames, default fast SRT settings | 4,130 s (~69 min) | 13 candidate meshes and `sam3d/best/output_scaled.glb` produced |

Reference host: two NVIDIA RTX A6000 GPUs (48 GB each), driver 580.159.03,
measured on 2026-07-15. Within the SAM3D total, the 13 sequential neural mesh
inferences took 841 s (~14 min); parallel SRT scale/alignment was the largest
stage at approximately 37 minutes.

For BundleSDF visual QA, accept the mesh when the object is recognizable, the
main body is coherent, and no defect materially changes its main geometry,
overall dimensions, or metric scale. Missing thin details, small reconstruction
fragments, minor holes, and local texture bleeding are expected limitations.
Use `fp_render_final/render.mp4` to confirm that the final mesh remains aligned
with the object.

For SAM3D visual QA, inspect `render_debug.jpg` and `render_video.mp4` for the
selected candidate. A visually complete mesh is not by itself evidence that
unseen geometry, proportions, or metric scale are accurate.

The current generated-mesh inventory is a practical guide, not a formal
benchmark. In general, the pipeline works best on rigid, opaque household
objects with enough visible surface area and texture for segmentation, stereo
depth, and camera tracking: boxes, bottles, cans, balls, mugs, toy tools, and
similar tabletop or carryable objects.

Use extra visual QA for these cases:

- **Thin or high-aspect-ratio geometry** — hoops, rackets, swords, canes, chair
  legs, handles, and tool tips are easy to miss, thicken, fuse to nearby
  surfaces, or reconstruct with holes.
- **Large support-like objects** — desks, tables, platforms, crates, and large
  boxes need broad view coverage. Partial views often leave missing backs,
  undersides, or weak texture alignment.
- **Reflective, dark, transparent, or textureless surfaces** — depth and SfM can
  become unstable, and masks may leak onto background or hands.
- **Occluded hand-object interaction frames** — hands can hide important object
  surfaces; BundleSDF quality depends on clean masks and coherent stereo depth
  across the selected scan frames.
- **Two-stage alignment failures** — BundleSDF final meshes depend on
  FoundationPose staying locked between the stationary and rotated stages. Drift
  usually shows up in `fp_render/render.mp4` or `poses_world_debug.png`.
- **SAM3D scale and back-side geometry** — scale is estimated after inference,
  and unseen geometry can be less reliable even when the mesh looks complete.

For production assets, prefer an available scanner mesh as the geometric
reference. Treat HOI-generated meshes as requiring inspection with the overlay,
spin, point-cloud, and chamfer tools before accepting them.

---

## Troubleshooting

Start from the final output and work backwards.

### BundleSDF: Bad final mesh

**Step 1 — Check Stage-1 mesh** (`stage1_recon/textured_mesh.obj`).

If Stage-1 mesh is bad, check inputs in order:

1. **Masks** — Spot-check `masks/0/`. Object should be cleanly segmented in every frame.
   - Wrong box: try a more specific `--prompt` (e.g. `"red spray bottle"` not `"bottle"`)
   - Mask drifts: adjust `foundationpose.reference_frame` in the pipeline config

2. **Camera poses and point cloud** — Run:
   ```bash
   python modules/v2d_bundlesdf/tools/visualize_reconstruction_standalone.py <job_dir>/stage1_recon/
   ```
   Cameras should orbit the object cleanly and the point cloud should form a coherent shape.

3. **Surface holes** — If cameras/point cloud are good but mesh has holes:
   - Increase `nerf.trunc` (default `0.01`, try `0.02`)
   - Decrease `nerf.mesh_resolution`, must stay ≤ `trunc` (default `0.005`, try `0.003`)
   - Increase `texture_bake.texture_res` for sharper texture (default `2048`, try `4096`)

4. **No zero surface** — If BundleSDF logs
   `Surface level must be within volume data range`, inspect
   `stage1_recon/resolved_config.yaml`.
   - `nerf.far` must cover the masked object depth range during SDF training.
   - The default config resolves `nerf.far: auto` from masked depth; if using a custom config, use `nerf.far: auto` or set an explicit larger value.
   - This is separate from `texture_bake.zfar`, which only affects texture rendering after a mesh exists.

If Stage-1 is good but final mesh is bad, the problem is in two-stage alignment:

1. **FP tracking** — Watch `fp_render/render.mp4`. The mesh overlay should stay locked to the object. Drifting indicates poor FoundationPose tracking.

2. **World poses** — Inspect `poses_world_debug.png`:
   - Object position (subplot 3): near-flat during Stage 1 and Stage 2
   - Angular velocity (subplot 5): spike only during stage transition
   - Vertical lines show detected stage boundaries — verify they match the video

### BundleSDF: FoundationPose weights not found

```
[error] FoundationPose weights not found at: ...
```

Download them:
```bash
python modules/v2d_foundation_pose/docker/run_download_weights.py \
  --output_dir data/weights/foundationpose
```

### BundleSDF: zfar clipping artifacts

If the mesh appears sliced at a fixed distance, the zfar clipping plane is too close. This is patched in the `v2d_bundlesdf` Dockerfile — rebuild the container:
```bash
docker build -t v2d_bundlesdf modules/v2d_bundlesdf/docker/
```

### SAM3D: `libEGL.so.1` missing during overlay-video rendering

If SRT completes but `render_textured_video` exits with code 139 and reports
`Could not find library libEGL.so.1`, the local `v2d_sam3d` image predates the
EGL/Pyrender renderer dependencies. Rebuild only that image:

```bash
python modules/v2d_sam3d/docker/build.py
```

Then rerun the same job with the documented skip flags. Complete SRT
candidates are reused automatically, so rebuilding the renderer does not
require regenerating meshes or scale results.

### Internal: cross-container symlinks

The pipeline uses **relative symlinks** to map sparse keyframe indices (from SfM) to contiguous `left{N:06d}` indices expected by BundleSDF, without copying large depth and mask files.

All symlink targets must live under `job_dir`. The `v2d_hoi_object_reconstruction` container and the `v2d_bundlesdf` container each mount `job_dir` at a different internal path, so a relative symlink like `../../depth/000001.png` resolves correctly in both.

**Constraints:**
- `depth/` must be a direct subdirectory of `job_dir` ✓
- `masks/` must also be a direct subdirectory of `job_dir` ✓

If you supply external depth or mask directories outside `job_dir`, symlinks inside BundleSDF will be broken. Keep all intermediate data within `job_dir`.

### FoundationPose Tracking Only

To run FoundationPose tracking with an existing mesh (skips reconstruction):

```bash
python modules/v2d_hoi_object_reconstruction/docker/run_reconstruction.py \
  --mapping_data_dir data/hoi_obj_recon/raw_data/<job> \
  --job_dir          data/hoi_obj_recon/jobs/<job> \
  --prompt           "basketball" \
  --skip_sfm --skip_stage1_detect --skip_stage1_setup \
  --skip_stage1_nerf --skip_center_mesh --skip_world_poses \
  --skip_merged_setup --skip_full_nerf
```

Place the mesh at `<job_dir>/mesh_input.obj` before running.

---

## Tools

| Tool | Location | Description |
|------|----------|-------------|
| `detect_stage1_end.py` | `v2d_hoi_object_reconstruction/lib/` | Manually inspect Stage-1 end detection from CuSFM trajectory |
| `visualize_reconstruction_standalone.py` | `v2d_bundlesdf/tools/` | Visualize camera trajectory and point cloud for reconstruction-quality checks |
| `plot_tum_file.py` | `v2d_cusfm/tools/` | Plot TUM-format trajectory file |
| `spin_mesh_video.py` | `tools/` | Render an appearance-faithful spinning video, including SAM3D vertex colors |
| `fuse_depth_to_pointcloud.py` | `v2d_bundlesdf/tools/` | Fuse depth maps into a point cloud |
| `view_glb.py` | `tools/` | View a `.glb` mesh file |
| `evaluate_mesh_quality.py` | `tools/` | Compare reconstructed geometry with a reference mesh using deterministic surface sampling, rigid registration, Chamfer distance, and threshold coverage |

### Appearance-faithful mesh turntable

[`tools/spin_mesh_video.py`](tools/spin_mesh_video.py) renders GLB, OBJ, and
other trimesh-supported inputs as H.264 turntable videos. It preserves GLB scene
node transforms. In the default `--shading auto` mode, conventional materials
and textures use lit PBR rendering, while SAM3D-style `COLOR_0` vertex colors
use flat, opaque shading so bright lights do not wash their appearance toward
white.

```bash
python modules/v2d_hoi_object_reconstruction/tools/spin_mesh_video.py \
  /path/to/output.glb /tmp/output_spin.mp4 \
  --frames 120 --fps 24 --width 1280 --height 720
```

For watertight vertex-colored inputs with negative signed volume, the renderer
also repairs clearly inward-facing winding in memory before rendering. It does
not rewrite the source mesh, and it does not guess the outside of an open
surface. Use `--shading lit` to force material-lit rendering or
`--shading flat` to force flat rendering.

### Offline mesh-geometry evaluation

[`tools/evaluate_mesh_quality.py`](tools/evaluate_mesh_quality.py) provides the
reproducible geometry-only comparison used for offline reconstruction analysis.
It is not part of either reconstruction pipeline and does not assign an EVT
pass/fail result.

For one reference/candidate pair:

```bash
python modules/v2d_hoi_object_reconstruction/tools/evaluate_mesh_quality.py pair \
  --reference /path/to/einstar/output.glb \
  --candidate /path/to/sam3d/output.glb \
  --object-id cyan_water_bottle \
  --method sam3d \
  --output /tmp/cyan_water_bottle_metrics.json
```

For a batch, create a JSON manifest. Relative mesh paths are resolved from the
manifest's directory:

```json
{
  "cases": [
    {
      "object_id": "cyan_water_bottle",
      "reference": "meshes/cyan_water_bottle/einstar/output.glb",
      "candidates": {
        "bundlesdf": "meshes/cyan_water_bottle/bundlesdf/output.glb",
        "sam3d": "meshes/cyan_water_bottle/sam3d/output.glb"
      }
    }
  ]
}
```

```bash
python modules/v2d_hoi_object_reconstruction/tools/evaluate_mesh_quality.py manifest \
  --manifest /path/to/mesh_pairs.json \
  --output-dir /tmp/mesh_quality \
  --keep-going
```

The evaluator samples 10,000 points from each surface, aligns the candidate to
the reference with rotation and translation only, and preserves the delivered
scale. `chamfer_mean_pct_diag` is the unsquared symmetric mean nearest-neighbor
distance, divided by the reference axis-aligned bounding-box diagonal and
reported as a percentage. Precision is the percentage of candidate points near
the reference; recall is the percentage of reference points covered by the
candidate. The default near-surface thresholds are 1%, 2%, and 5% of the same
reference diagonal.

`as_delivered` reports geometry with the candidate's original scale.
`shape_scale_normalized` uniformly corrects the candidate's RMS surface radius
before repeating registration, which helps separate scale error from shape
error. Both results ignore texture, color, materials, and semantic correctness.
Sampling and registration are deterministic for a fixed mesh and seed, but
symmetric shapes and large missing regions can still make registration
ambiguous. Always review the meshes and reconstruction overlays alongside the
numbers.