Add flat mesh slots for heterogeneous multi-mesh raycasts#5531
Add flat mesh slots for heterogeneous multi-mesh raycasts#5531ooctipus wants to merge 13 commits into
Conversation
Build clone plans from scene configuration so representative sources are spawned directly in environments before USD and physics replication run. Co-authored-by: Cursor <cursoragent@cursor.com>
Restore the immutable existing fragment and add fresh package fragments for the branch changes. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the skip fragment consistent with existing empty marker files. Co-authored-by: Cursor <cursoragent@cursor.com>
Restore the default clone-plan builder and keep planning from mutating prim paths before entity construction. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep preset resolution fully upstream of scene construction instead of adding a scene-level fallback or error branch. Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid constructing a default plan before replacing it with the cfg-derived scene plan. Co-authored-by: Cursor <cursoragent@cursor.com>
Allow rigid object collections to fall back to prim_path when they are constructed outside InteractiveScene, while still honoring clone-planned spawn_path values. Co-authored-by: Cursor <cursoragent@cursor.com>
Greptile SummaryThis PR replaces the rectangular padded-table layout for multi-mesh raycasts with a compact flat-slot Warp kernel (
Confidence Score: 3/5Not safe to merge as-is: two files carry stale references to removed The
Important Files Changed
Sequence DiagramsequenceDiagram
participant Scene as InteractiveScene
participant Cloner as cloner_utils
participant SimCtx as SimulationContext
participant Sensor as BaseMultiMeshRayCaster
participant Kernel as raycast_dynamic_mesh_slots_kernel
Scene->>Cloner: make_clone_plan(sources, dests, num_envs, strategy)
Cloner-->>Scene: ClonePlan(sources[], destinations[], clone_mask)
Scene->>SimCtx: set_clone_plan(plan)
Scene->>Scene: clone_environments() → usd_replicate / physics_clone_fn
Sensor->>SimCtx: get_clone_plan()
SimCtx-->>Sensor: ClonePlan
Sensor->>Sensor: _initialize_warp_meshes_from_clone_plan(plan)
Note over Sensor: Builds flat slot arrays
loop each simulation step
Sensor->>Sensor: _update_mesh_transforms()
Sensor->>Kernel: "launch(dim=(num_slots, num_rays))"
Note over Kernel: slot_id → tid_env via slot_env_ids
Kernel-->>Sensor: ray_hits, ray_distance, ray_mesh_id
end
Reviews (1): Last reviewed commit: "Add flat mesh slots for multi-mesh rayca..." | Re-trigger Greptile |
| best: ClonePlan | None = None | ||
| best_len = -1 | ||
| for plan in plans: | ||
| prefix = plan.dest_template.split("{}")[0] |
There was a problem hiding this comment.
Stale
dest_template reference on refactored ClonePlan
_find_covering_plan accesses plan.dest_template, but the ClonePlan dataclass was restructured in this PR to replace the single dest_template: str field with destinations: list[str] (one entry per source row). Any caller of walk_target_prototypes will hit an AttributeError at runtime. The function needs to iterate over plan.destinations rows and use each row's template string to compute the prefix, rather than calling plan.dest_template.
| sim.set_clone_plans( | ||
| { | ||
| env_fmt: lab_cloner.ClonePlan( | ||
| dest_template=env_fmt, | ||
| prototype_paths=[env_fmt.format(0)], | ||
| clone_mask=torch.ones((1, num_envs), dtype=torch.bool), | ||
| ) | ||
| } | ||
| ) |
There was a problem hiding this comment.
Test helper uses old
ClonePlan API and old set_clone_plans method
This block calls sim.set_clone_plans(...) (plural), which was renamed to set_clone_plan(...) (singular) in this PR, and constructs ClonePlan(dest_template=env_fmt, prototype_paths=[...], ...) with field names that no longer exist (replaced by sources and destinations). Running this check script would immediately fail with TypeError at ClonePlan construction and AttributeError on set_clone_plans. It should use sim.set_clone_plan(ClonePlan(sources=[...], destinations=[...], clone_mask=...)).
| if return_face_id == 1: | ||
| ray_face_id[tid_env, tid_ray] = mesh_query_ray_t.face | ||
| if return_mesh_id == 1: | ||
| ray_mesh_id[tid_env, tid_ray] = wp.int16(slot_id) |
There was a problem hiding this comment.
ray_mesh_id stores global slot index, not per-env mesh index
In raycast_dynamic_meshes_kernel (the rectangular-table variant), ray_mesh_id is set to wp.int16(tid_mesh_id) — the per-env mesh index. Here the flat-slot kernel stores wp.int16(slot_id) — the global slot index, which is different for environments that start at a non-zero slot offset. Any consumer that treats ray_mesh_ids as a dense per-env mesh index (e.g. to index into a per-env property table) will receive incorrect results for environments whose slots do not start at 0. The docstring for raycast_dynamic_mesh_slots_kernel should call out this semantic difference, and callers building on ray_mesh_ids should be verified against the new convention.
7e9b31e to
bafae7c
Compare
Move raycaster logic behind backend-specific PhysX and Newton implementations so sensors can track poses without FrameView. Preserve ClonePlan-based multi-mesh target discovery while keeping the existing rectangular raycast kernel for this PR. Co-authored-by: Cursor <cursoragent@cursor.com>
Use compact per-environment mesh slots so ClonePlan-backed heterogeneous layouts avoid dummy rectangular padding while preserving closest-hit behavior across raycaster variants. Co-authored-by: Cursor <cursoragent@cursor.com>
bafae7c to
86d73c5
Compare
Motivation
PR #5510 adds ClonePlan-based multi-mesh target discovery as part of the RayCaster backend split. To keep that PR conservative, heterogeneous layouts still use the existing rectangular dynamic-mesh kernel with dummy padding for environments that have fewer meshes.
This PR removes that padding cost. Instead of launching raycasts over
num_envs * max_meshes_per_env, it launches over the actual flat list of real mesh targets.What Changed
raycast_dynamic_mesh_slots_kernel, launched as(num_slots, num_rays).MultiMeshRayCasterandMultiMeshRayCasterCamerafrom rectangular padded tables to flat mesh slots.slot_env_idsso each mesh slot maps back to its owning environment.[1, 2, 1]without USD object clones.Why
The rectangular kernel is correct, but for heterogeneous ClonePlan layouts it does work for dummy meshes. Flat slots match the actual scene layout: one slot per real mesh, no fake mesh ids, no far-away padding poses, and less wasted kernel work.
Dependency
Depends on #5510. Please review and land #5510 first, then review this PR as the follow-up flat-slot optimization. After #5510 lands, this PR should be rebased onto
developso the diff only shows the flat-slot change.Test Plan
./isaaclab.sh -p -m pytest source/isaaclab/test/sensors/test_ray_caster_integration.py::test_multi_mesh_consumes_clone_plan_without_usd_object_clones -q./isaaclab.sh -p -m pytest source/isaaclab/test/sensors/test_multi_mesh_ray_caster_camera.py -q./isaaclab.sh -p -m pytest source/isaaclab/test/sensors/test_ray_caster_integration.py::test_update_mesh_transforms_non_identity_offset -q./isaaclab.sh -f