Surround Vision for ROS 2

A surround-view visualisation stack for ROS 2 Jazzy. The package fuses four fisheye cameras, renders a calibrated top-down view, and overlays a textured 3D vehicle model. It ships with a fast JPEG decoder based on libturbojpeg, a standalone OBJ inspector, and a demo bag so you can evaluate the pipeline in minutes.

https://github.com/user-attachments/assets/86f12873-8ef3-4b9e-bac3-4688c6c7f0d1

Highlights

• No NVIDIA GPU is required; any OpenGL-supported GPU will do just great, including AMD or even Intel dedicated GPUs.
• Real-time stitched bird’s-eye view with vehicle overlay.
• Works with either sensor_msgs/msg/Image (raw) or sensor_msgs/msg/CompressedImage (JPEG) inputs per camera.
• TurboJPEG-accelerated decoding path with automatic OpenCV fallback when streams are corrupted.
• Interactive OpenGL viewer (orbit / pan / zoom) and an OBJ-only node to inspect vehicle meshes.
• Configuration-first: all topics, decoding modes, window settings, and file paths live in YAML.

ROS 2 Nodes

Sample Bag

A short demo bag is included to validate the pkg:

ros2 bag info sample_bag/

Files:             sample_bag_0.mcap
Duration:          2.69 s
Messages:          176
Topics:            /frontCamera/v4l2/compressed
                   /leftCamera/v4l2/compressed
                   /rightCamera/v4l2/compressed
                   /rearCamera/v4l2/compressed
                   /insideCamera/v4l2/compressed
                   /groundCamera/v4l2/compressed

Quick Start

# Clone into your workspace
cd ~/ros2_ws/src
git clone https://github.com/JeyP4/SurroundVisionROS2.git

# Install runtime dependencies
sudo apt update
sudo apt install -y \
    libopencv-dev \
    libyaml-cpp-dev \
    libglew-dev \
    libglm-dev \
    libassimp-dev \
    libturbojpeg0-dev \
    libglfw3-dev \
    libgl1-mesa-dev \
    libglu1-mesa-dev \
    ros-jazzy-cv-bridge \
    ros-jazzy-sensor-msgs

# Build
cd ~/ros2_ws
colcon build --packages-select surround_vision --cmake-args -DCMAKE_BUILD_TYPE=Release
source install/setup.bash

Demo Run (compressed sample bag)

ros2 bag play -l sample_bag

# Another terminal
ros2 launch surround_vision surround_vision_rover.launch.py

The launch file loads config/surround_vision_compressed.yaml by default, which subscribes to the /v4l2/compressed topics and enables the TurboJPEG decoding path.

Configuration

Two YAML configuration presets are provided:

• config/surround_vision_compressed.yaml — expects JPEG (sensor_msgs/msg/CompressedImage) topics.
• config/surround_vision_raw.yaml — uses raw sensor_msgs/msg/Image topics instead.

Each camera block exposes fields:

front_camera:
  topic: "/frontCamera/v4l2/compressed"   # or "/frontCamera/image"
  transport: "compressed"   # or "raw"

Switching between raw and compressed streams is as simple as editing the YAML or supplying an alternative file:

ros2 launch surround_vision surround_vision_rover.launch.py \
  config:=/path/to/surround_vision_raw.yaml

Launch Arguments

Camera Calibration

Surround-view accuracy is tied to intrinsic and extrinsic calibration quality. This package includes a unified marker-free calibration system that simplifies extrinsic parameter estimation for multi-camera setups.

Extrinsic Calibration System

The package includes a complete calibration workflow in camera_calibration/ that:

• No checkerboard required: Use simple 3D point measurements from overlapping camera FOVs
• Flexible and scalable: Minimum 3 points per set, add more for better accuracy
• Visual validation: Debug overlays and Rviz visualization show calibration quality
• Automated workflow: Single launch command performs calibration and saves results

📖 Detailed Calibration Guide →

The calibration system uses four point sets (FL, FR, RL, RR) corresponding to overlapping fields of view between adjacent cameras. Simply measure 3D coordinates of identifiable points and their corresponding image pixel coordinates, then run the calibration script.

For intrinsic calibration, use standard OpenCV checkerboard methods or your preferred calibration pipeline. Replace config/camerasParam.yaml with your intrinsic parameters before running extrinsic calibration.

Controls (Surround Viewer)

• Left drag: orbit around the vehicle.
• Right drag: pan.
• Scroll: zoom.
• R (OBJ viewer): reset camera.
• F (surround viewer): fullscreen toggle.

Dependencies at a Glance

• ROS 2 Jazzy (rclcpp, sensor_msgs, cv_bridge)
• OpenGL 3.3+, GLEW, GLFW, GLM
• OpenCV
• yaml-cpp
• assimp
• libturbojpeg (optional but strongly recommended)
• bundled headers: tiny_obj_loader, stb_image

Consult CMakeLists.txt if you are packaging for another platform or need additional compiler flags.

Repository Layout

config/                 # YAML presets (topics, window defaults, calibration points)
camera_calibration/     # Extrinsic calibration system (see README.md inside)
launch/                 # Minimal launch files
models/smartCar/        # Default vehicle OBJ + textures
sample_bag/             # Demo bag
src/
  surround_vision_node.cpp
  fast_image_republisher_optimized.cpp
  obj_viewer_node.cpp

Tips & Troubleshooting

• Non-JPEG data on compressed topics → The decoder logs and skips the frame, falling back to OpenCV only when TurboJPEG fails. Verify your camera driver publishes valid JPEG payloads.
• Raw topics → Switch the YAML transport to raw; the node automatically skips JPEG checks and consumes sensor_msgs/msg/Image.
• Rendering issues → Ensure you have an OpenGL 3.3 compatible driver and run with a local display (ssh -X for remote sessions).
• Model alignment → Adjust scale/orientation in your OBJ or update the parameter file if you replace smartCar.

Author & Maintainer

• Dedicated LinkedIn post
• Jai Prakash
• Feel free to open an issue or reach out via LinkedIn for collaborations or questions regarding the pipeline.

Recognition

Project born from the need to visualize multi-camera surround-view in real time during the vehicle teleoperation research at Politecnico di Milano. Its use can even be extended to other robotics fields like legged robots, etc.

Thanks to Open Robotics for featuring in ROS News! Cheers to ROS community.👍

License: Apache-2.0