Feature/unified depth node

[CenekAlbl]

Purpose

This PR introduces the unified Depth node, that automatically selects the appropriate depth algorithm for a currently used device. This version uses hardcoded choices of specific devices.

Specification

Depth node is a source node, which does not need any inputs.
It creates the required depth nodes as its subnodes:

• StereoDepth
• NeuralDepth
• NeuralAssistedStereo
• GPUStereo
• ToF
  Each of these nodes has the ability to output 2 unified output:
• Depth
• Confidence

For this PR, the logic of choosing the backend Depth algorithm is the following:

flowchart TD
    Start["User requests depth"] --> Q1{"User pinned<br/>algorithm + config?"}

    Q1 -- yes --> Pin{"Supported by device<br/>& valid config?"}
    Pin -- no --> Err["Error"]
    Pin -- yes --> Use["Use requested<br/>algorithm + config"]

    Q1 -- no --> Q2{"User requested<br/>a concrete algorithm?"}

    Q2 -- yes --> Sup{"Algorithm<br/>supported?"}
    Sup -- no --> Err
    Sup -- yes --> Cfg["Use requested algorithm,<br/>auto-pick best config<br/>for FPS + resolution"]

    Q2 -- "no (AUTO)" --> Plat{"Platform?"}

    Plat -- "RVC4" --> Auto["Pick best backend by priority<br/>(NEURAL → NAS → STEREO → GPU_STEREO)<br/>matching FPS + resolution;<br/>fall back to best-fit / StereoDepth"]
    Plat -- "other" --> RVC2{"ToF sensor<br/>connected?"}
    RVC2 -- yes --> ToF["Use ToF"]
    RVC2 -- no --> Stereo["Use StereoDepth"]

Loading

Then for RVC4 it chooses the appropriate backend using the following logic based on Quality first in priority order Neural>NeuralAssisted>StereoDepth>EVA.

flowchart TD
    Start["RVC4 AUTO<br/>(target FPS + optional resolution known)"] --> Scan["Priority scan over backend profiles<br/>in fixed order"]

    Scan --> Order["NEURAL models (quality-first)<br/>→ NEURAL_ASSISTED_STEREO<br/>→ STEREO presets<br/>→ GPU_STEREO profiles"]

    Order --> Match{"First profile that:<br/>• algorithm supported<br/>• meets target FPS (× 0.9)<br/>• passes model filter<br/>• fits resolution?"}

    Match -- found --> UseIt["Use that algorithm + config"]

    Match -- "none" --> Pinned{"User pinned<br/>both FPS + resolution?"}
    Pinned -- yes --> Err["Error: nothing can serve<br/>requested FPS + resolution"]

    Pinned -- no --> HasRes{"Resolution known?"}
    HasRes -- yes --> FitRes{"Any supported profile<br/>fits the resolution<br/>(ignoring FPS)?"}
    FitRes -- yes --> BestFps["Use profile that fits resolution<br/>with the highest available FPS"]
    FitRes -- no --> LastResort
    HasRes -- no --> LastResort

    LastResort["Last resort: StereoDepth"] --> StPreset{"A StereoDepth preset<br/>fits resolution<br/>(no FPS gate)?"}
    StPreset -- yes --> UsePreset["Use STEREO + that preset"]
    StPreset -- no --> UseFast["Use STEREO + FAST_ACCURACY"]

Loading

For Stereo based methods, the required Camera nodes are created if not existing in the pipeline yet, if they exist, they are reused.

Now the FPS and Resolution that is used in resolving the RVC4 algorithm to be used is determined by the following logic

flowchart TD
    Start["RVC4 resolution/FPS decision"] --> Pair["Find device stereo pair"]
    Pair --> FindCams["Find existing pipeline Camera nodes<br/>matching left/right sockets"]

    FindCams --> FpsOverride{"Depth build(... fps)<br/>provided?"}
    FpsOverride -- yes --> UseBuildFps["Use build() FPS"]
    FpsOverride -- no --> BothCamsFps{"Both left + right<br/>Camera nodes exist?"}
    BothCamsFps -- yes --> UseCamFps["Infer target FPS from cameras:<br/>max(left.getMaxRequestedFps,<br/>right.getMaxRequestedFps)"]
    BothCamsFps -- no --> DefaultFps["Use default target FPS: 30"]

    UseBuildFps --> ResOverride
    UseCamFps --> ResOverride
    DefaultFps --> ResOverride

    ResOverride{"Depth build(... stereoSize)<br/>provided?"}
    ResOverride -- yes --> UseBuildRes["Use build() stereoSize"]
    ResOverride -- no --> BothCamsRes{"Both left + right<br/>Camera nodes exist?"}
    BothCamsRes -- yes --> MatchingReq{"Do both cameras have<br/>matching requested output size<br/>with width/height > 0?"}
    MatchingReq -- yes --> UseCamRes["Infer resolution from cameras:<br/>left requested width/height"]
    MatchingReq -- no --> DeviceFeatures["Infer resolution from device<br/>camera features for stereo pair"]
    BothCamsRes -- no --> DeviceFeatures

    UseBuildRes --> Select["Call selectBackend(resolution, targetFps, ...)"]
    UseCamRes --> Select
    DeviceFeatures --> Select

Loading

Dependencies & Potential Impact

Pipeline interaction:

• The Depth node uses lazy graph bulding, where node creation and initialization is defered to when the actual pipeline is finalized, using buildInternal method.
• There is always interaction if a user has explicitly initalized the Camera nodes in the pipeline. In this case they are reused, but their setting is not altered and the user setting is reused. The requestOutput method is used to get the ISP outputs of the cameras, which supports "lazy" FPS setting which happens only after all requests are collected and highest requested FPS is chosen with throttling down the other ISP connections.
• There are some 'unhappy' paths expected, where user e.g. initializes the individual subnodes themselves outside the depth node, but that would be an edge case as the user has no reason to do that.

CI / HIL: depth_node_test drops the ci label; tests/run_tests.py uses ^(ci|ondevice)$ so hardware jobs still select it; anyone relying on ctest -L ci alone to run every device test should note depth_node_test is excluded unless they add an explicit filter or use run_tests.py.
Breaking changes: None called out for public API beyond additive Depth surface and pipeline adoption behavior for DeviceNodeGroup subtrees.

Deployment Plan

This branch has been merged with develop branch commit 9849076. Further testing on more device types will be performed to complete the algorithm vs device matrix. HIL tests should be set up. Needs to be tested against various pipelines that use cameras with different settings. ToF and GPUStereo needs to be tested. If testing goes well this can be scheduled for 3.7.0 release.

Testing & Validation

Automated: tests/src/ondevice_tests/pipeline/node/depth_node_test.cpp — algorithm selection (AUTO / explicit NEURAL, STEREO, NAS, TOF, GPU_STEREO where applicable), negative guards (e.g. NAS off RVC4, TOF without sensor), stereo camera order (cameras before vs after Depth), extra nodes (SystemLogger + optional third camera), plus host-only pipeline rejection.
Manual / on-device: Python unified_depth.py / related Depth examples for smoke runs on hardware (exact matrix depends on connected device). Tested on RVC4 and RVC2 Stereo devices so far.
Note: Full depth_node_test run needs an exclusive device; label/ctest usage should use anchored -R if needed to avoid pulling unrelated long-running tests.

Known issues:

• When running unified_depth.py example, there is an error thrown at the beginning and end of the pipeline, which does not interrupt the full functionality, to be investigated:
• [ThreadedNode] [error] Node threw exception, stopping the node. Exception message: Communication exception - possible device error/misconfiguration. Original message 'Invalid Member Count'

AI Usage

Assisted-by: Cursor 2.3 and 3.2, model Auto chosen

Submitted code was reviewed by a human: YES

The author is taking the responsibility for the contribution: YES

Summary by CodeRabbit

• New Features

  • GPUStereo node with full configuration, debug views and a PyQt GUI for live tuning
  • Unified Depth composite node exposing unified depth/confidence outputs
  • Depth-to-point-cloud shader for efficient 3D point generation

• Bug Fixes

  • Python Pipeline.create now forwards constructor args/kwargs to node factories

• Documentation

  • Detailed Depth node docs with usage examples and platform guidance

• Examples

  • Multiple new Python examples and runnable GPUStereo scripts

• Tests

  • New on-device tests validating Depth behavior and backend selection

[MaticTonin]

.cpp counterpart....

[aljazkonec1]

Ready to be merged 😃 thanks for the effort everyone!

[MaticTonin]

LGTM!!

afk