fix(sim/mujoco): metric depth passthrough + capture fd-2 depth warning + silence export_xml attach noise#891
Conversation
…g + silence export_xml attach noise render_depth treated MuJoCo's depth output as a normalized [0,1] OpenGL buffer and re-linearized it with znear*zfar/(zfar - d*(zfar-znear)). But MuJoCo >= 3.0's Renderer.enable_depth_rendering() returns METRIC depth in meters directly, so that re-projection collapsed near-field depth onto the near-clip plane: a scene whose nearest surface sits at 0.41 m reported a minimum of 0.01 m (znear) regardless of content. Consume the metric array as-is; only sanitize NaN/+inf -> far-clip, -inf/negative -> 0, clamp to [0, zfar]. The ARB_clip_control depth warning is a C-level write to fd 2, which contextlib.redirect_stderr cannot see (it only swaps sys.stderr), so the warning both leaked to the console and never reached the response text it was meant to populate. Add a reusable capture_stderr_fd() context manager (os.dup2-based fd-2 capture, mirroring filter_mujoco_attach_noise, with the same no-op fallback when fd 2 can't be duped) and use it in render_depth: surface the notice in the response text, drop the duplicate console line (log at DEBUG), and pass any other stderr lines through unchanged. export_xml called spec.to_xml() raw while every other call site wraps it in filter_mujoco_attach_noise(), so it spammed benign 'Attach conflict ... keeping parent value' chatter to the console for every attached-robot scene. Wrap it too. Tests: rewrite test_render_depth_linearizes_normalized_buffer_to_meters (which enshrined the bug) as test_render_depth_passes_through_metric_depth, add test_render_depth_sanitizes_nan_and_inf. Both fail on pre-fix code.
yinsong1986
left a comment
There was a problem hiding this comment.
Summary
Three well-scoped fixes to the MuJoCo rendering path. (1) render_depth now consumes MuJoCo >= 3.0's metric depth buffer as-is instead of re-applying the OpenGL znear*zfar/(...) re-linearization that was collapsing the near field onto znear — a real correctness bug, verified empirically against raw mujoco.Renderer output (0.41 m near surface now reported correctly). Sanitation (nan_to_num -> far-clip, negatives -> 0, clamp to [0, zfar]) is sound and the uniform-depth (span == 0) PNG path is handled. (2) A new capture_stderr_fd() helper does real os.dup2-based fd-2 capture, faithfully mirroring the proven filter_mujoco_attach_noise structure (lock-guarded swap, restore in finally, no-op fallback when fd 2 can't be duped) so the one-time ARB_clip_control notice reaches the response text instead of leaking to the console. (3) export_xml now wraps spec.to_xml() in filter_mujoco_attach_noise(), matching every other call site.
No blocking concerns found: no user input flows into subprocess/XML/path construction, no public API or wire-format/response-shape changes (the depth_min/depth_max json block and __all__-free internal helper are unchanged surfaces), no new locking regression (the PR doesn't alter the lock acquisition around render_depth; the module-level _noise_lock correctly serializes the fd swap), and the depth math cannot crash on the validated non-empty array. The two new regression tests fail on pre-fix code and pass after.
What's good
- Correctness fix is empirically grounded (raw-renderer comparison table) and pinned with a regression test that fails on the old code.
- New
capture_stderr_fdreuses the established fd-2 capture pattern rather than reinventing it;export_xmlnoise wrapping closes a real call-site inconsistency. - Narrow
except (ValueError, OSError)clauses on the stderr best-effort paths, consistent with AGENTS.md > Review Learnings (#86) > 'Exception Clauses Must Be Narrow'. - No emojis in user-facing strings; tight scope discipline.
Verification suggestions
- Headless EGL: run
tests/simulation/mujoco/test_rendering.pyand confirmtest_render_depth_passes_through_metric_depth/test_render_depth_sanitizes_nan_and_infpass and fail on pre-fix code. - Smoke-test
export_xml()on a two-arm attached scene and confirm noAttach conflictchatter reaches the console while the returned MJCF is non-empty.
…uses (addresses CodeQL threads backend.py:387,400)
yinsong1986
left a comment
There was a problem hiding this comment.
Summary
Three well-scoped fixes to the MuJoCo rendering path. (1) RenderingMixin.render_depth now consumes MuJoCo >= 3.0's metric depth buffer as-is instead of re-applying the OpenGL znear*zfar / (zfar - d*(zfar-znear)) re-linearization that was collapsing the near field onto znear -- a real correctness bug, empirically verified against raw mujoco.Renderer output (0.41 m near surface now reported correctly). The sanitation path (nan_to_num mapping NaN/+inf -> far-clip and -inf -> 0, then clamp to [0, zfar]) is sound, and the uniform-depth span == 0 PNG branch avoids a divide-by-zero. (2) A new capture_stderr_fd() helper does real os.dup2-based fd-2 capture, faithfully mirroring the proven filter_mujoco_attach_noise structure (lock-guarded swap under _noise_lock, restore in finally, no-op fallback when fd 2 can't be duped), so the one-time ARB_clip_control notice reaches the response text instead of leaking to the console. (3) PhysicsMixin.export_xml now wraps spec.to_xml() in filter_mujoco_attach_noise(), matching every other call site.
No blocking concerns found: no user input flows into subprocess/XML/path construction; no public API, wire-format, or response-shape changes (the depth_min/depth_max json block is unchanged and the new helper is an internal, non-__all__ backend function); the fd-swap reuses the existing _noise_lock serialization with no reentrant deadlock on the depth path; and the depth math cannot crash on the validated non-empty array. The two new regression tests fail on pre-fix code and pass after, and the CodeQL empty-except findings from prior rounds were already addressed with explanatory comments at the flagged lines.
What's good
- Correctness fix is empirically grounded (raw-renderer comparison table) and pinned with a regression test that fails on the old code.
- New
capture_stderr_fdreuses the established fd-2 capture pattern rather than reinventing it; theexport_xmlwrap closes a genuine call-site inconsistency. - Narrow
except (ValueError, OSError)clauses on the best-effort stderr paths, consistent with AGENTS.md > Review Learnings (#86) > 'Exception Clauses Must Be Narrow'. - No emojis in user-facing strings; tight scope discipline.
Verification suggestions
- Headless EGL: run
tests/simulation/mujoco/test_rendering.pyand confirmtest_render_depth_passes_through_metric_depthandtest_render_depth_sanitizes_nan_and_infpass on this branch and fail on pre-fix code. - Smoke-test
export_xml()on a two-arm attached scene and confirm noAttach conflictchatter reaches the console while the returned MJCF is non-empty.
Summary
Three fixes to the MuJoCo simulation rendering path. The depth one is a correctness bug; the other two are log-hygiene/observability fixes that share a root cause (C-level fd-2 writes that Python-level stderr redirection cannot see).
1.
render_depthcollapses near-field depth to the near-clip plane (correctness)RenderingMixin.render_depthtreated MuJoCo's depth output as a normalized[0, 1]OpenGL buffer and re-linearized it withz = znear*zfar / (zfar - d*(zfar - znear)). But MuJoCo >= 3.0'sRenderer.enable_depth_rendering()returns metric depth in meters directly (distance along each ray). Applying the OpenGL re-linearization to already-metric values collapses the near field ontoznear, so the reported minimum is wrong regardless of scene content.Verified in headless MuJoCo sim (EGL) on a two-arm SO-101 scene whose nearest surface sits at 0.41 m. Raw
mujoco.Rendererreturnsmin=0.4129m. The engine reported:Left: pre-fix (re-linearized) depth map - near geometry crushed to a flat near-clip value. Right: post-fix metric passthrough - correct depth gradient across both arms and the floor. Headless MuJoCo render (EGL), 640x480.
Fix: consume the metric array as-is; only sanitize - replace
NaN/+infwith the far-clip distance (zfar = model.vis.map.zfar * model.stat.extent), replace-inf/negatives with 0, and clamp to[0, zfar]. The now-stale "Linearized Min/Max" wording becomes "Metric Min/Max".2.
ARB_clip_controldepth warning leaked to the console and never reached the responserender_depthtried to capture MuJoCo's one-timeARB_clip_controlnotice withcontextlib.redirect_stderr(StringIO()). That only swaps Python'ssys.stderrobject and is blind to the C-level write to file descriptor 2 from MuJoCo's OpenGL backend, so the warning (a) leaked to the console anyway and (b) was never captured into the response text it was meant to populate.Fix: add a reusable
capture_stderr_fd()context manager inbackend.pythat does realos.dup2-based fd-2 capture (mirroring the proven approach infilter_mujoco_attach_noise, including the same no-op fallback when fd 2 can't be duped - pytest capsys / Jupyter). Use it inrender_depth: surface the notice in the response text, drop the duplicate console line (log at DEBUG instead), and pass any other stderr lines through unchanged so genuine errors never vanish.3.
export_xmlspammed benign attach noise to the consolePhysicsMixin.export_xmlcalledspec.to_xml()raw, while every other call site (scene_ops.py,simulation.py) wraps it infilter_mujoco_attach_noise(). Result: 30+ lines of benignAttach conflict ... keeping parent valuechatter (aUserWarning+ raw fd-2 writes) on every attached-robot scene export. Wrapped it too.Verified in sim:
export_xml()on a two-arm scene now returns cleanly (2028-char MJCF) with no attach-noise spam on the console.Tests
test_render_depth_linearizes_normalized_buffer_to_meters(which enshrined the bug - it fed a[0,1]buffer and expected znear/zfar linearization) astest_render_depth_passes_through_metric_depth(asserts metric meters pass through unchanged).test_render_depth_sanitizes_nan_and_inf(NaN/+inf -> far-clip, negatives -> 0).assert 0.0099... == 0.75,assert nan == 0.0) and pass after._FakeDepthRendererdocstring + module comment to the metric-depth contract.Gate (headless EGL):
ruff check/ruff format --check/mypyclean;tests/simulation/mujoco/897 passed, 18 skipped, 0 failed.Files
strands_robots/simulation/mujoco/backend.py(+capture_stderr_fd)strands_robots/simulation/mujoco/physics.py(wrapexport_xmlto_xml)strands_robots/simulation/mujoco/rendering.py(fd capture + metric depth passthrough)tests/simulation/mujoco/test_rendering.py(rewrite + regression test)§13 Review Round Changelog
rendering.py:7638425ccfbackend.py:387,400bb6ab9c