[feat] Improve Multisensor mode documentation (#65)

* [feat] Complete Multisensor mode integration

Align the depth-or-overlap contract across APIs and tools, document experimental tracking limitations, and add focused configuration and tracking coverage.

* [clean] Trim Multisensor release documentation

Remove unnecessary standalone artifacts and keep wheel guidance version-independent.

* [clean] Extract Multisensor tools integration

Keep the API and documentation change focused while tool support receives separate validation.

* [test] Use distinct Multisensor stereo buffers

Match the tracker input contract by giving each camera its own image allocation.

* [fix] Address Multisensor review feedback

Clarify runtime behavior and make tests and YAML validation enforce the documented input contracts.

* [fix] Align timestamp docs with implementation

Document non-decreasing cross-sensor ordering while retaining strict camera-frame ordering.

Author: vikuznetsov-nvidia

README.md

@@ -29,7 +29,7 @@ cuVSLAM is the library by NVIDIA, providing various Visual Tracking Camera modes
 # Tracking modes
 
 cuVSLAM's tracker supports several odometry modes selected via `Odometry::Config::odometry_mode`
-(C++) / `cuvslam.Odometry.OdometryMode` (Python). Modes differ in which sensors they require, which
+(C++) / `cuvslam.Tracker.OdometryMode` (Python). Modes differ in which sensors they require, which
 ones they can additionally fuse, and how they handle scale. Pick the mode that matches the most
 informative sensor set on your rig:
 
@@ -39,18 +39,23 @@ informative sensor set on your rig:
 | `RGBD` | 1 RGB-D camera (aligned RGB + depth) | — | `RGBDSettings` | Single depth-aligned camera (e.g. RealSense, TUM RGB-D). |
 | `Multicamera` | ≥2 cameras with at least one overlapping pair (a stereo pair) | up to 32 cameras total | — | Stereo or multi-stereo rigs. Most accurate purely-visual mode. |
 | `Inertial` | 1 stereo pair + 1 IMU | — | — | Stereo VIO. Adds robustness to brief visual failures. |
-| `Multisensor` | At least one camera pair with overlapping frustums; cuNLS-enabled build | RGB-D cameras (any subset), 0 or 1 IMU | `MultisensorSettings` | Any-mix RGB / RGB-D rigs with optional IMU. Use when the legacy modes don't fit: ≥3 cameras of mixed RGB and RGB-D types; a multi-camera rig where only a subset provides depth; or RGB-D + IMU fusion (note that `Inertial` mode is stereo-only). |
+| `Multisensor` | ≥1 RGB-D camera **or** ≥1 overlapping camera pair; cuNLS-enabled build | Additional RGB/RGB-D cameras, 0 or 1 IMU | `MultisensorSettings` | Any-mix RGB / RGB-D rigs with optional IMU, including one RGB-D camera with IMU. |
 
 Notes:
 - `Multisensor` is the only mode that requires a cuNLS-enabled build — see
-  [Optional: cuNLS](#optional-cunls). All other modes work with the default build.
+  [cuNLS](#cunls). All other modes work with the default build.
 - IMU fusion is available in `Inertial` (always on) and `Multisensor` (auto-enabled when
   `Rig::imus` is non-empty).
 - For a runnable Multisensor walkthrough on a multi-RGB-D + IMU rig, see
-  [examples/multisensor/](examples/multisensor/README.md). For the full per-field API reference,
-  see the [C++](https://nvidia-isaac.github.io/cuVSLAM/cpp/) or
+  [examples/multisensor/](examples/multisensor/README.md). For the full per-field API reference, see the
+  [C++](https://nvidia-isaac.github.io/cuVSLAM/cpp/) or
   [Python](https://nvidia-isaac.github.io/cuVSLAM/python/) docs.
 
+## Multisensor mode
+
+Multisensor tracking is experimental: tracking may be inaccurate or fail for some sensor configurations and scenes.
+The current implementation supports pinhole cameras only.
+
 # Using cuVSLAM
 
 The quickest way to get started is to [install PyCuVSLAM from a pre-built wheel](#install-from-wheels)
@@ -109,6 +114,8 @@ for the following configurations:
 
 **Prerequisite**: [CUDA Toolkit 12 or 13](https://developer.nvidia.com/cuda/toolkit) must be installed separately (not included in the wheels).
 
+Official wheels include cuNLS support for `Multisensor` mode; no separate cuNLS installation is required.
+
 To install (virtual environment is recommended):
 
 1. Go to the [releases page](https://github.com/nvidia-isaac/cuVSLAM/releases).
@@ -200,6 +207,10 @@ required. The source is pinned in `cmake/ext/cunls.cmake`; cuNLS downloads its o
 network connection and the CUDA Toolkit are the only prerequisites. The resulting static archive
 is bundled into `libcuvslam`.
 
+cuNLS source and license information are published in the
+[nvidia-isaac/cuNLS repository](https://github.com/nvidia-isaac/cuNLS). Release wheels and C++ packages built with the
+default configuration include cuNLS support and do not require a separate runtime package.
+
 `USE_CUDA` must be `ON`. To build without cuNLS (and disable the multisensor odometry mode that
 depends on it), configure with `-DUSE_CUNLS=OFF`.
 

TROUBLESHOOTING.md

@@ -448,13 +448,16 @@ See [Isaac ROS cuVSLAM parameters][].
 
 Approximate accuracy ranking, least to most accurate:
 
-Mono → RGBD → Stereo Inertial → Stereo → Multicamera ≈ Multisensor
+Mono → RGBD → Stereo Inertial → Stereo → Multicamera; Multisensor depends on its sensor set
 
 Multisensor (see [examples/multisensor/](examples/multisensor/README.md)) does not slot at a fixed
-position — its accuracy scales with the sensor set it is given. A single RGB-D + IMU rig in
-Multisensor is roughly RGBD-class, whereas a multi-stereo + RGB-D + IMU rig in Multisensor matches
-or exceeds Multicamera. Use Multisensor when you have a mixed-camera-type rig or want IMU fusion on
-a non-stereo configuration; otherwise prefer the mode that exactly matches your rig.
+position — its accuracy scales with the sensor set it is given. It requires at least one RGB-D
+camera or one overlapping camera pair and currently supports pinhole cameras only. A single
+RGB-D + IMU rig is roughly RGBD-class. A multi-stereo + RGB-D + IMU rig may approach or exceed
+Multicamera under favorable conditions, but this is an empirical expectation rather than a
+guarantee; benchmark the actual rig and environment. Use Multisensor when you have a
+mixed-camera-type rig or want IMU fusion on a non-stereo configuration; otherwise prefer the mode
+that exactly matches your rig.
 
 ### Adjust motion prediction
 

cuvslam-skills/README.md

@@ -10,7 +10,7 @@ Build, install, and run cuVSLAM / PyCuVSLAM. Covers:
 
 - Environment setup and build (source + wheel + Docker)
 - Dataset preparation (KITTI, EuRoC, TUM RGB-D, multi-camera)
-- Running all tracking modes (stereo, mono, mono-depth, stereo-inertial, multi-camera)
+- Running all tracking modes (stereo, mono, mono-depth, stereo-inertial, multi-camera, Multisensor)
 - SLAM workflow (mapping, localization, loop closure)
 - Live camera setup (RealSense, ZED, OAK-D, Orbbec)
 - C++ tools and ROS 2 integration

cuvslam-skills/cuvslam-onboard/SKILL.md

@@ -3,8 +3,8 @@ name: cuvslam-onboard
 description: >
   Build, install, and run NVIDIA cuVSLAM and PyCuVSLAM from source or wheels.
   Covers environment setup, dataset preparation, and running examples for all
-  tracking modes (stereo, mono, mono-depth, stereo-inertial, multi-camera) and
-  SLAM (mapping, localization, loop closure). Use when asked to: build cuVSLAM,
+  tracking modes (stereo, mono, mono-depth, stereo-inertial, multi-camera, and
+  Multisensor) and SLAM (mapping, localization, loop closure). Use when asked to: build cuVSLAM,
   install PyCuVSLAM, set up cuVSLAM environment, run cuVSLAM examples, prepare
   KITTI/EuRoC/TUM datasets, run visual odometry, set up live camera tracking
   (RealSense/ZED/OAK-D/Orbbec), run cuVSLAM in Docker, or use cuVSLAM C++ tools.
@@ -83,6 +83,7 @@ CMake options:
 - `-DUSE_RERUN=ON` — enable Rerun visualization for C++ tools
 - `-DCUVSLAM_BUILD_SHARED_LIB=TRUE` — build shared library (default)
 - `-DUSE_CUDA=ON` — use CUDA (default)
+- `-DUSE_CUNLS=ON` — enable Multisensor mode (default; requires CUDA)
 
 ### Install PyCuVSLAM from source
 
@@ -126,7 +127,7 @@ cuVSLAM supports these visual tracking modes:
 | Stereo-Inertial | `OdometryMode.Inertial` (1) | Stereo + IMU for robustness |
 | Mono-Depth (RGB-D) | `OdometryMode.RGBD` (2) | Monocular + depth image |
 | Monocular | `OdometryMode.Mono` (3) | Single camera (no scale) |
-| Multisensor | `OdometryMode.Multisensor` (4) | Any-mix RGB / RGB-D cameras with optional IMU. Requires cuNLS-enabled build. Configure via `MultisensorSettings`. See `examples/multisensor/`. |
+| Multisensor | `OdometryMode.Multisensor` (4) | At least one RGB-D camera or one overlapping camera pair, with optional IMU. Requires cuNLS and currently supports pinhole cameras only. Configure via `MultisensorSettings`. See `examples/multisensor/`. |
 
 ## 5. Run Examples — Public Datasets
 
@@ -196,16 +197,30 @@ python3 download_tartan.py
 python3 track_multicamera_tartan.py
 ```
 
+### Multisensor (Tartan Ground, RGB-D + optional IMU)
+
+Multisensor tracking is experimental and may be inaccurate or fail for some sensor configurations and scenes.
+
+```bash
+cd examples/multisensor
+pip install tartanair     # x86_64 only
+python3 download_tartan.py
+python3 track_multisensor_tartan.py
+python3 track_multisensor_tartan.py --no-imu
+```
+
 ## 6. Run Examples — Live Cameras
 
 See `references/live-cameras.md` for detailed setup per camera.
-
-| Camera | Stereo | VIO | RGB-D | Multi-cam |
-|--------|--------|-----|-------|-----------|
-| RealSense | `run_stereo.py` | `run_vio.py` | `run_rgbd.py` | `run_multicamera.py` |
-| ZED | `run_stereo.py` | — | `run_rgbd.py` | — |
-| OAK-D | `run_stereo.py` | — | — | — |
-| Orbbec | `run_stereo.py` | — | `run_rgbd.py` | — |
+This table lists ready-made Python examples shipped in the repository. An em dash means no
+ready-made example exists; it does not mean the hardware integration is unsupported.
+
+| Camera | Stereo | VIO | RGB-D | Multi-cam | Multisensor |
+|--------|--------|-----|-------|-----------|--------------------------|
+| RealSense | `run_stereo.py` | `run_vio.py` | `run_rgbd.py` | `run_multicamera.py` | `run_multisensor.py` |
+| ZED | `run_stereo.py` | — | `run_rgbd.py` | — | — |
+| OAK-D | `run_stereo.py` | — | — | — | — |
+| Orbbec | `run_stereo.py` | — | `run_rgbd.py` | — | — |
 
 ## 7. C++ API
 

cuvslam-skills/cuvslam-onboard/references/dataset-guides.md

@@ -7,6 +7,7 @@ Detailed setup for each public dataset supported by cuVSLAM examples.
 - [EuRoC MAV](#euroc-mav)
 - [TUM RGB-D](#tum-rgb-d)
 - [Multi-Camera Datasets](#multi-camera-datasets)
+- [Multisensor](#multisensor)
 - [Distortion Models](#distortion-models)
 - [EDEX File Format](#edex-file-format)
 - [Converting ROS Bags](#converting-ros-bags)
@@ -155,6 +156,23 @@ ros2 run isaac_ros_rosbag_utils extract_edex \
 
 ---
 
+## Multisensor
+
+The TartanGround example exercises multiple RGB-D cameras with an optional IMU:
+
+```bash
+cd examples/multisensor
+pip install tartanair    # x86_64 only; on Jetson, download on x86 and transfer
+python3 download_tartan.py
+python3 track_multisensor_tartan.py
+python3 track_multisensor_tartan.py --no-imu
+```
+
+Multisensor requires `USE_CUNLS=ON`, pinhole cameras, and at least one RGB-D camera or one overlapping camera pair.
+Multisensor tracking is experimental and may be inaccurate or fail for some sensor configurations and scenes.
+
+---
+
 ## Distortion Models
 
 | Model | Coefficients | Enum |

cuvslam-skills/cuvslam-onboard/references/live-cameras.md

@@ -12,7 +12,7 @@ Detailed setup for each camera platform supported by PyCuVSLAM.
 
 ## RealSense
 
-**Supported modes:** Stereo, Stereo-Inertial (VIO), Mono-Depth (RGB-D), Multi-Camera
+**Supported modes:** Stereo, Stereo-Inertial (VIO), Mono-Depth (RGB-D), Multi-Camera, Multisensor
 
 ### Install librealsense
 
@@ -42,6 +42,7 @@ python3 run_stereo.py        # Stereo odometry
 python3 run_vio.py           # Stereo-inertial (requires IMU calibration)
 python3 run_rgbd.py          # Mono-depth (enable IR emitter, use RGB not IR for tracking)
 python3 run_multicamera.py   # Multi-camera (needs frame_nano_rig.yaml with your extrinsics)
+python3 run_multisensor.py   # Multisensor RGB-D + IMU (requires cuNLS; pinhole only)
 ```
 
 ### IMU calibration for VIO

cuvslam-skills/cuvslam-troubleshoot/SKILL.md

@@ -80,7 +80,11 @@ If the user has not provided a config, say:
 > "To root cause this accurately I need the cuVSLAM configuration you were using. Please share the parameters file (for Isaac ROS) or the relevant `OdometryConfig` / `CUVSLAM_Configuration` settings you had set when the issue occurred."
 
 Key parameters to note once you have the config:
-- `tracking_mode` (Isaac ROS) / `OdometryMode` (Python) — determines whether IMU is used (`0`/`Multicamera`=VO only, `1`/`Inertial`=VIO). Knowing this up front directs the diagnosis toward vision or IMU root causes.
+- `tracking_mode` (Isaac ROS) / `OdometryMode` (Python) — `0`/`Multicamera` is VO-only, while
+  `1`/`Inertial` requires and uses an IMU. `Multisensor` is different: it automatically enables IMU fusion only when
+  `Rig.imus` is non-empty, so the mode value alone does not determine IMU use. Verify the rig's IMU list alongside
+  cuNLS availability, `depth_camera_ids`, and pinhole calibration.
+  Knowing this up front directs the diagnosis toward vision, depth, or IMU root causes.
 - `rectified_images` (Isaac ROS) / `rectified_stereo_camera` (C++ API) — wrong value flips the entire stereo pipeline
 - `image_jitter_threshold_ms` — tolerated timestamp jitter between left/right images; too tight a value causes frames to be dropped silently
 - `async_sba` / `sync_mode` — affects reproducibility

doc/index.md

@@ -9,8 +9,8 @@ Tracking modes
 --------------
 
 The tracker supports several odometry modes — see `cuvslam::Odometry::OdometryMode` for the enum
-and per-mode requirements. The unified `Multisensor` mode (any mix of RGB / RGB-D cameras with an
-optional IMU; requires a cuNLS-enabled build) is configured through
+and per-mode requirements. `Multisensor` accepts at least one RGB-D camera or one overlapping camera
+pair, with an optional IMU, and requires a cuNLS-enabled build. It is configured through
 `cuvslam::Odometry::MultisensorSettings`. A high-level mode chooser table lives in the top-level
 README's "Tracking modes" section.
 

examples/euroc/README.md

@@ -14,7 +14,7 @@ PyCuVSLAM supports multiple visual tracking modes. You can specify the desired t
 
 * **Monocular**: Visual tracking using a monocular camera. This mode provides accurate camera rotation estimation but does not estimate scale. (set as `OdometryMode(3)` or `OdometryMode.Mono`)
 
-* **Multisensor**: Unified mode that accepts any mix of plain RGB and RGB-D cameras with an optional single IMU (set as `OdometryMode(4)` or `OdometryMode.Multisensor`). Requires a cuNLS-enabled build and is configured via `MultisensorSettings`; see [examples/multisensor/](../multisensor/README.md) for a runnable walkthrough.
+* **Multisensor**: Unified mode that accepts at least one RGB-D camera or one overlapping camera pair, with optional additional RGB/RGB-D cameras and one IMU (set as `OdometryMode(4)` or `OdometryMode.Multisensor`). Requires a cuNLS-enabled build, currently supports pinhole cameras only, and is configured via `MultisensorSettings`. See [examples/multisensor/](../multisensor/README.md) for a runnable walkthrough.
 
 PyCuVSLAM exercises three modes on the EuRoC MAV dataset — Stereo (`Multicamera`), Stereo-Inertial (`Inertial`), and Monocular (`Mono`) — because *Mono-Depth* has no aligned depth stream (see the [TUM-RGBD dataset example](../tum/README.md)) and *Multisensor* has its own [example](../multisensor/README.md). You can try different tracking modes by modifying the following line in `track_euroc.py`:
 

examples/multisensor/README.md

@@ -1,11 +1,28 @@
-# Tutorial: Running PyCuVSLAM Multisensor Odometry (multi RGB-D + IMU)
+# Tutorial: PyCuVSLAM Multisensor Odometry (multi RGB-D + IMU)
+
+> **Experimental:** Tracking may be inaccurate or fail for some sensor configurations and scenes.
 
 This tutorial demonstrates how to run PyCuVSLAM in **Multisensor** odometry
 mode, which solves a single tightly-coupled cuNLS step over any mix of plain
 RGB cameras, RGB-D cameras, and one optional IMU. The example uses two
 RGB-D cameras (`lcam_front`, `lcam_back`) plus a synthetic IMU from the
 [TartanGround dataset](https://tartanair.org/tartanground/).
 
+## Requirements
+
+- Use an official wheel or build with `USE_CUNLS=ON`.
+- Configure at least one RGB-D camera in `depth_camera_ids`, or provide at least one camera pair with overlapping
+  frustums. A single RGB-D camera is valid, with or without an IMU.
+- Use pinhole cameras. Other camera models are not supported by the current solver.
+- Align each depth image pixel-for-pixel with the RGB image at the same camera index. PyCuVSLAM accepts 2D `uint16`
+  depth; the C++ API accepts `UINT16` or `FLOAT32`.
+- Configure no more than one IMU. Serialize image and IMU calls in non-decreasing timestamp order; camera frame
+  timestamps must be strictly increasing.
+
+Construction raises `ValueError` when the rig or settings are invalid. A valid tracker may return a
+`PoseEstimate` with `world_from_rig=None` while initializing or after tracking loss. Configured depth streams may be
+omitted from an individual frame after a sensor drop, but every supplied depth must match a configured camera index.
+
 ## Set Up the PyCuVSLAM Environment
 
 Refer to the [Installation Guide](../README.md#prerequisites) for detailed