Skip to content

Add flat mesh slots for heterogeneous multi-mesh raycasts#5531

Open
ooctipus wants to merge 13 commits into
isaac-sim:developfrom
ooctipus:octi/raycaster-flat-mesh-slots
Open

Add flat mesh slots for heterogeneous multi-mesh raycasts#5531
ooctipus wants to merge 13 commits into
isaac-sim:developfrom
ooctipus:octi/raycaster-flat-mesh-slots

Conversation

@ooctipus
Copy link
Copy Markdown
Collaborator

@ooctipus ooctipus commented May 7, 2026

Motivation

PR #5510 adds ClonePlan-based multi-mesh target discovery as part of the RayCaster backend split. To keep that PR conservative, heterogeneous layouts still use the existing rectangular dynamic-mesh kernel with dummy padding for environments that have fewer meshes.

This PR removes that padding cost. Instead of launching raycasts over num_envs * max_meshes_per_env, it launches over the actual flat list of real mesh targets.

What Changed

  • Added raycast_dynamic_mesh_slots_kernel, launched as (num_slots, num_rays).
  • Switched MultiMeshRayCaster and MultiMeshRayCasterCamera from rectangular padded tables to flat mesh slots.
  • Added slot_env_ids so each mesh slot maps back to its owning environment.
  • Updated PhysX and Newton dynamic target pose refresh to write into flat slot buffers.
  • Updated ClonePlan integration coverage to assert heterogeneous slot counts [1, 2, 1] without USD object clones.

Why

The rectangular kernel is correct, but for heterogeneous ClonePlan layouts it does work for dummy meshes. Flat slots match the actual scene layout: one slot per real mesh, no fake mesh ids, no far-away padding poses, and less wasted kernel work.

Dependency

Depends on #5510. Please review and land #5510 first, then review this PR as the follow-up flat-slot optimization. After #5510 lands, this PR should be rebased onto develop so the diff only shows the flat-slot change.

Test Plan

  • ./isaaclab.sh -p -m pytest source/isaaclab/test/sensors/test_ray_caster_integration.py::test_multi_mesh_consumes_clone_plan_without_usd_object_clones -q
  • ./isaaclab.sh -p -m pytest source/isaaclab/test/sensors/test_multi_mesh_ray_caster_camera.py -q
  • ./isaaclab.sh -p -m pytest source/isaaclab/test/sensors/test_ray_caster_integration.py::test_update_mesh_transforms_non_identity_offset -q
  • ./isaaclab.sh -f

ooctipus and others added 10 commits May 6, 2026 23:01
Build clone plans from scene configuration so representative sources are spawned directly in environments before USD and physics replication run.

Co-authored-by: Cursor <cursoragent@cursor.com>
Restore the immutable existing fragment and add fresh package fragments for the branch changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the skip fragment consistent with existing empty marker files.

Co-authored-by: Cursor <cursoragent@cursor.com>
Restore the default clone-plan builder and keep planning from mutating prim paths before entity construction.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep preset resolution fully upstream of scene construction instead of adding a scene-level fallback or error branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid constructing a default plan before replacing it with the cfg-derived scene plan.

Co-authored-by: Cursor <cursoragent@cursor.com>
Allow rigid object collections to fall back to prim_path when they are constructed outside InteractiveScene, while still honoring clone-planned spawn_path values.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels May 7, 2026
@ooctipus ooctipus changed the title Add flat mesh slots for multi-mesh raycasts Add flat mesh slots for heterogeneous multi-mesh raycasts May 7, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 7, 2026

Greptile Summary

This PR replaces the rectangular padded-table layout for multi-mesh raycasts with a compact flat-slot Warp kernel (raycast_dynamic_mesh_slots_kernel), enabling heterogeneous per-environment mesh counts without padding. It also restructures ClonePlan from a per-group object (with dest_template/prototype_paths) into a single flat plan (sources/destinations/clone_mask), moves heterogeneous clone-plan construction into InteractiveScene, and extracts shared ray-caster logic into backend-agnostic base classes.

  • New raycast_dynamic_mesh_slots_kernel dispatches over a flat (num_slots, num_rays) grid; each slot carries its owning environment id, eliminating per-env padding.
  • ClonePlan is refactored to carry flat parallel sources/destinations lists; clone_from_template is removed and make_clone_plan now returns a ClonePlan directly.
  • BaseMultiMeshRayCaster and BaseRayCaster are extracted as backend-neutral base classes; PhysX and Newton subclasses are thin mixins.

Confidence Score: 3/5

Not safe to merge as-is: two files carry stale references to removed ClonePlan fields that will raise AttributeError at runtime, and the ray_mesh_id semantic change in the new flat-slot kernel may silently corrupt per-env mesh-id lookups in downstream consumers.

The _target_tracker_utils.py module references plan.dest_template — a field that was deleted in this PR — meaning any caller of walk_target_prototypes crashes immediately. The check_multi_mesh_ray_caster.py test helper similarly constructs a ClonePlan with the old field names and calls the renamed set_clone_plans method, so the script fails before exercising any sensor logic. Additionally, the new raycast_dynamic_mesh_slots_kernel records the global slot_id in ray_mesh_id rather than the per-env mesh index used by the existing rectangular kernel, which can silently produce wrong indices for any code that treats the output as a dense per-env mesh selector.

_target_tracker_utils.py (stale dest_template access), check_multi_mesh_ray_caster.py (old ClonePlan constructor and set_clone_plans call), and kernels.py (undocumented ray_mesh_id semantic change in the flat-slot kernel).

Important Files Changed

Filename Overview
source/isaaclab/isaaclab/utils/warp/kernels.py Adds raycast_dynamic_mesh_slots_kernel — flat slot layout for heterogeneous scenes; ray_mesh_id stores global slot index (not per-env mesh index), diverging from the existing rectangular-kernel convention without documentation.
source/isaaclab/isaaclab/sensors/ray_caster/_target_tracker_utils.py New shared target-tracker utility; _find_covering_plan accesses plan.dest_template — a field that no longer exists on the refactored ClonePlan, causing AttributeError at runtime for any caller.
source/isaaclab/isaaclab/sensors/ray_caster/base_multi_mesh_ray_caster.py New base class for multi-mesh ray-casting using compact flat slots; ClonePlan integration and mesh slot initialization look structurally correct.
source/isaaclab/isaaclab/cloner/clone_plan.py Replaces dest_template/prototype_paths fields with flat sources/destinations lists; adds eq=False to keep hash identity-based. Several consumers were not fully updated.
source/isaaclab/isaaclab/scene/interactive_scene.py Migrates heterogeneous clone-plan construction into the scene itself; correctly uses the new flat ClonePlan API and publishes via set_clone_plan (singular).
source/isaaclab/test/sensors/check_multi_mesh_ray_caster.py Check script uses old set_clone_plans (plural) and constructs ClonePlan with removed fields dest_template/prototype_paths; will fail immediately on execution.
source/isaaclab/isaaclab/cloner/cloner_utils.py Removes clone_from_template; make_clone_plan now returns a ClonePlan directly instead of a tuple — clean simplification.
source/isaaclab_physx/isaaclab_physx/scene_data_providers/physx_scene_data_provider.py Correctly migrated to flat ClonePlan; reads plan.sources/plan.destinations directly instead of reconstructing paths from dest_template.

Sequence Diagram

sequenceDiagram
    participant Scene as InteractiveScene
    participant Cloner as cloner_utils
    participant SimCtx as SimulationContext
    participant Sensor as BaseMultiMeshRayCaster
    participant Kernel as raycast_dynamic_mesh_slots_kernel

    Scene->>Cloner: make_clone_plan(sources, dests, num_envs, strategy)
    Cloner-->>Scene: ClonePlan(sources[], destinations[], clone_mask)
    Scene->>SimCtx: set_clone_plan(plan)
    Scene->>Scene: clone_environments() → usd_replicate / physics_clone_fn

    Sensor->>SimCtx: get_clone_plan()
    SimCtx-->>Sensor: ClonePlan
    Sensor->>Sensor: _initialize_warp_meshes_from_clone_plan(plan)
    Note over Sensor: Builds flat slot arrays

    loop each simulation step
        Sensor->>Sensor: _update_mesh_transforms()
        Sensor->>Kernel: "launch(dim=(num_slots, num_rays))"
        Note over Kernel: slot_id → tid_env via slot_env_ids
        Kernel-->>Sensor: ray_hits, ray_distance, ray_mesh_id
    end
Loading

Reviews (1): Last reviewed commit: "Add flat mesh slots for multi-mesh rayca..." | Re-trigger Greptile

best: ClonePlan | None = None
best_len = -1
for plan in plans:
prefix = plan.dest_template.split("{}")[0]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale dest_template reference on refactored ClonePlan

_find_covering_plan accesses plan.dest_template, but the ClonePlan dataclass was restructured in this PR to replace the single dest_template: str field with destinations: list[str] (one entry per source row). Any caller of walk_target_prototypes will hit an AttributeError at runtime. The function needs to iterate over plan.destinations rows and use each row's template string to compute the prefix, rather than calling plan.dest_template.

Comment on lines +108 to 116
sim.set_clone_plans(
{
env_fmt: lab_cloner.ClonePlan(
dest_template=env_fmt,
prototype_paths=[env_fmt.format(0)],
clone_mask=torch.ones((1, num_envs), dtype=torch.bool),
)
}
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Test helper uses old ClonePlan API and old set_clone_plans method

This block calls sim.set_clone_plans(...) (plural), which was renamed to set_clone_plan(...) (singular) in this PR, and constructs ClonePlan(dest_template=env_fmt, prototype_paths=[...], ...) with field names that no longer exist (replaced by sources and destinations). Running this check script would immediately fail with TypeError at ClonePlan construction and AttributeError on set_clone_plans. It should use sim.set_clone_plan(ClonePlan(sources=[...], destinations=[...], clone_mask=...)).

if return_face_id == 1:
ray_face_id[tid_env, tid_ray] = mesh_query_ray_t.face
if return_mesh_id == 1:
ray_mesh_id[tid_env, tid_ray] = wp.int16(slot_id)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 ray_mesh_id stores global slot index, not per-env mesh index

In raycast_dynamic_meshes_kernel (the rectangular-table variant), ray_mesh_id is set to wp.int16(tid_mesh_id) — the per-env mesh index. Here the flat-slot kernel stores wp.int16(slot_id) — the global slot index, which is different for environments that start at a non-zero slot offset. Any consumer that treats ray_mesh_ids as a dense per-env mesh index (e.g. to index into a per-env property table) will receive incorrect results for environments whose slots do not start at 0. The docstring for raycast_dynamic_mesh_slots_kernel should call out this semantic difference, and callers building on ray_mesh_ids should be verified against the new convention.

@ooctipus ooctipus force-pushed the octi/raycaster-flat-mesh-slots branch from 7e9b31e to bafae7c Compare May 7, 2026 08:55
ooctipus and others added 3 commits May 7, 2026 01:55
Move raycaster logic behind backend-specific PhysX and Newton implementations so sensors can track poses without FrameView. Preserve ClonePlan-based multi-mesh target discovery while keeping the existing rectangular raycast kernel for this PR.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use compact per-environment mesh slots so ClonePlan-backed heterogeneous layouts avoid dummy rectangular padding while preserving closest-hit behavior across raycaster variants.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ooctipus ooctipus force-pushed the octi/raycaster-flat-mesh-slots branch from bafae7c to 86d73c5 Compare May 7, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant