Skip to content

fix(sim/mujoco): metric depth passthrough + capture fd-2 depth warning + silence export_xml attach noise#891

Merged
cagataycali merged 3 commits into
strands-labs:mainfrom
cagataycali:fix/mujoco-depth-and-noise
Jun 30, 2026
Merged

fix(sim/mujoco): metric depth passthrough + capture fd-2 depth warning + silence export_xml attach noise#891
cagataycali merged 3 commits into
strands-labs:mainfrom
cagataycali:fix/mujoco-depth-and-noise

Conversation

@cagataycali

@cagataycali cagataycali commented Jun 30, 2026

Copy link
Copy Markdown
Member

Summary

Three fixes to the MuJoCo simulation rendering path. The depth one is a correctness bug; the other two are log-hygiene/observability fixes that share a root cause (C-level fd-2 writes that Python-level stderr redirection cannot see).

1. render_depth collapses near-field depth to the near-clip plane (correctness)

RenderingMixin.render_depth treated MuJoCo's depth output as a normalized [0, 1] OpenGL buffer and re-linearized it with z = znear*zfar / (zfar - d*(zfar - znear)). But MuJoCo >= 3.0's Renderer.enable_depth_rendering() returns metric depth in meters directly (distance along each ray). Applying the OpenGL re-linearization to already-metric values collapses the near field onto znear, so the reported minimum is wrong regardless of scene content.

Verified in headless MuJoCo sim (EGL) on a two-arm SO-101 scene whose nearest surface sits at 0.41 m. Raw mujoco.Renderer returns min=0.4129m. The engine reported:

depth_min depth_max
pre-fix 0.0102 m (= znear, wrong) 49.13 m
post-fix 0.4129 m (matches raw metric) 50.77 m

pre vs post depth

Left: pre-fix (re-linearized) depth map - near geometry crushed to a flat near-clip value. Right: post-fix metric passthrough - correct depth gradient across both arms and the floor. Headless MuJoCo render (EGL), 640x480.

Fix: consume the metric array as-is; only sanitize - replace NaN/+inf with the far-clip distance (zfar = model.vis.map.zfar * model.stat.extent), replace -inf/negatives with 0, and clamp to [0, zfar]. The now-stale "Linearized Min/Max" wording becomes "Metric Min/Max".

2. ARB_clip_control depth warning leaked to the console and never reached the response

render_depth tried to capture MuJoCo's one-time ARB_clip_control notice with contextlib.redirect_stderr(StringIO()). That only swaps Python's sys.stderr object and is blind to the C-level write to file descriptor 2 from MuJoCo's OpenGL backend, so the warning (a) leaked to the console anyway and (b) was never captured into the response text it was meant to populate.

Fix: add a reusable capture_stderr_fd() context manager in backend.py that does real os.dup2-based fd-2 capture (mirroring the proven approach in filter_mujoco_attach_noise, including the same no-op fallback when fd 2 can't be duped - pytest capsys / Jupyter). Use it in render_depth: surface the notice in the response text, drop the duplicate console line (log at DEBUG instead), and pass any other stderr lines through unchanged so genuine errors never vanish.

3. export_xml spammed benign attach noise to the console

PhysicsMixin.export_xml called spec.to_xml() raw, while every other call site (scene_ops.py, simulation.py) wraps it in filter_mujoco_attach_noise(). Result: 30+ lines of benign Attach conflict ... keeping parent value chatter (a UserWarning + raw fd-2 writes) on every attached-robot scene export. Wrapped it too.

Verified in sim: export_xml() on a two-arm scene now returns cleanly (2028-char MJCF) with no attach-noise spam on the console.

Tests

  • Rewrote test_render_depth_linearizes_normalized_buffer_to_meters (which enshrined the bug - it fed a [0,1] buffer and expected znear/zfar linearization) as test_render_depth_passes_through_metric_depth (asserts metric meters pass through unchanged).
  • Added test_render_depth_sanitizes_nan_and_inf (NaN/+inf -> far-clip, negatives -> 0).
  • Both new tests fail on pre-fix code (assert 0.0099... == 0.75, assert nan == 0.0) and pass after.
  • Updated _FakeDepthRenderer docstring + module comment to the metric-depth contract.

Gate (headless EGL): ruff check / ruff format --check / mypy clean; tests/simulation/mujoco/ 897 passed, 18 skipped, 0 failed.

Files

  • strands_robots/simulation/mujoco/backend.py (+capture_stderr_fd)
  • strands_robots/simulation/mujoco/physics.py (wrap export_xml to_xml)
  • strands_robots/simulation/mujoco/rendering.py (fd capture + metric depth passthrough)
  • tests/simulation/mujoco/test_rendering.py (rewrite + regression test)

§13 Review Round Changelog

Round Concern Fix Commit Pin Test
R2 CodeQL empty-except comment on rendering.py:763 8425ccf (comment-only, no new test)
R3 CodeQL empty-except comments on backend.py:387,400 bb6ab9c (comment-only, no new test)

…g + silence export_xml attach noise

render_depth treated MuJoCo's depth output as a normalized [0,1] OpenGL
buffer and re-linearized it with znear*zfar/(zfar - d*(zfar-znear)). But
MuJoCo >= 3.0's Renderer.enable_depth_rendering() returns METRIC depth in
meters directly, so that re-projection collapsed near-field depth onto the
near-clip plane: a scene whose nearest surface sits at 0.41 m reported a
minimum of 0.01 m (znear) regardless of content. Consume the metric array
as-is; only sanitize NaN/+inf -> far-clip, -inf/negative -> 0, clamp to
[0, zfar].

The ARB_clip_control depth warning is a C-level write to fd 2, which
contextlib.redirect_stderr cannot see (it only swaps sys.stderr), so the
warning both leaked to the console and never reached the response text it
was meant to populate. Add a reusable capture_stderr_fd() context manager
(os.dup2-based fd-2 capture, mirroring filter_mujoco_attach_noise, with the
same no-op fallback when fd 2 can't be duped) and use it in render_depth:
surface the notice in the response text, drop the duplicate console line
(log at DEBUG), and pass any other stderr lines through unchanged.

export_xml called spec.to_xml() raw while every other call site wraps it in
filter_mujoco_attach_noise(), so it spammed benign 'Attach conflict ...
keeping parent value' chatter to the console for every attached-robot scene.
Wrap it too.

Tests: rewrite test_render_depth_linearizes_normalized_buffer_to_meters
(which enshrined the bug) as test_render_depth_passes_through_metric_depth,
add test_render_depth_sanitizes_nan_and_inf. Both fail on pre-fix code.
Comment thread strands_robots/simulation/mujoco/backend.py Fixed
Comment thread strands_robots/simulation/mujoco/backend.py Fixed
Comment thread strands_robots/simulation/mujoco/rendering.py Fixed
yinsong1986
yinsong1986 previously approved these changes Jun 30, 2026

@yinsong1986 yinsong1986 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Three well-scoped fixes to the MuJoCo rendering path. (1) render_depth now consumes MuJoCo >= 3.0's metric depth buffer as-is instead of re-applying the OpenGL znear*zfar/(...) re-linearization that was collapsing the near field onto znear — a real correctness bug, verified empirically against raw mujoco.Renderer output (0.41 m near surface now reported correctly). Sanitation (nan_to_num -> far-clip, negatives -> 0, clamp to [0, zfar]) is sound and the uniform-depth (span == 0) PNG path is handled. (2) A new capture_stderr_fd() helper does real os.dup2-based fd-2 capture, faithfully mirroring the proven filter_mujoco_attach_noise structure (lock-guarded swap, restore in finally, no-op fallback when fd 2 can't be duped) so the one-time ARB_clip_control notice reaches the response text instead of leaking to the console. (3) export_xml now wraps spec.to_xml() in filter_mujoco_attach_noise(), matching every other call site.

No blocking concerns found: no user input flows into subprocess/XML/path construction, no public API or wire-format/response-shape changes (the depth_min/depth_max json block and __all__-free internal helper are unchanged surfaces), no new locking regression (the PR doesn't alter the lock acquisition around render_depth; the module-level _noise_lock correctly serializes the fd swap), and the depth math cannot crash on the validated non-empty array. The two new regression tests fail on pre-fix code and pass after.

What's good

  • Correctness fix is empirically grounded (raw-renderer comparison table) and pinned with a regression test that fails on the old code.
  • New capture_stderr_fd reuses the established fd-2 capture pattern rather than reinventing it; export_xml noise wrapping closes a real call-site inconsistency.
  • Narrow except (ValueError, OSError) clauses on the stderr best-effort paths, consistent with AGENTS.md > Review Learnings (#86) > 'Exception Clauses Must Be Narrow'.
  • No emojis in user-facing strings; tight scope discipline.

Verification suggestions

  • Headless EGL: run tests/simulation/mujoco/test_rendering.py and confirm test_render_depth_passes_through_metric_depth / test_render_depth_sanitizes_nan_and_inf pass and fail on pre-fix code.
  • Smoke-test export_xml() on a two-arm attached scene and confirm no Attach conflict chatter reaches the console while the returned MJCF is non-empty.

…uses (addresses CodeQL threads backend.py:387,400)

@yinsong1986 yinsong1986 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Three well-scoped fixes to the MuJoCo rendering path. (1) RenderingMixin.render_depth now consumes MuJoCo >= 3.0's metric depth buffer as-is instead of re-applying the OpenGL znear*zfar / (zfar - d*(zfar-znear)) re-linearization that was collapsing the near field onto znear -- a real correctness bug, empirically verified against raw mujoco.Renderer output (0.41 m near surface now reported correctly). The sanitation path (nan_to_num mapping NaN/+inf -> far-clip and -inf -> 0, then clamp to [0, zfar]) is sound, and the uniform-depth span == 0 PNG branch avoids a divide-by-zero. (2) A new capture_stderr_fd() helper does real os.dup2-based fd-2 capture, faithfully mirroring the proven filter_mujoco_attach_noise structure (lock-guarded swap under _noise_lock, restore in finally, no-op fallback when fd 2 can't be duped), so the one-time ARB_clip_control notice reaches the response text instead of leaking to the console. (3) PhysicsMixin.export_xml now wraps spec.to_xml() in filter_mujoco_attach_noise(), matching every other call site.

No blocking concerns found: no user input flows into subprocess/XML/path construction; no public API, wire-format, or response-shape changes (the depth_min/depth_max json block is unchanged and the new helper is an internal, non-__all__ backend function); the fd-swap reuses the existing _noise_lock serialization with no reentrant deadlock on the depth path; and the depth math cannot crash on the validated non-empty array. The two new regression tests fail on pre-fix code and pass after, and the CodeQL empty-except findings from prior rounds were already addressed with explanatory comments at the flagged lines.

What's good

  • Correctness fix is empirically grounded (raw-renderer comparison table) and pinned with a regression test that fails on the old code.
  • New capture_stderr_fd reuses the established fd-2 capture pattern rather than reinventing it; the export_xml wrap closes a genuine call-site inconsistency.
  • Narrow except (ValueError, OSError) clauses on the best-effort stderr paths, consistent with AGENTS.md > Review Learnings (#86) > 'Exception Clauses Must Be Narrow'.
  • No emojis in user-facing strings; tight scope discipline.

Verification suggestions

  • Headless EGL: run tests/simulation/mujoco/test_rendering.py and confirm test_render_depth_passes_through_metric_depth and test_render_depth_sanitizes_nan_and_inf pass on this branch and fail on pre-fix code.
  • Smoke-test export_xml() on a two-arm attached scene and confirm no Attach conflict chatter reaches the console while the returned MJCF is non-empty.

@cagataycali
cagataycali merged commit de7ee2a into strands-labs:main Jun 30, 2026
7 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in Strands Labs - Robots Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

4 participants