Skip to content

[CI unblock] Skip viewergl-fully-black test + fix warp intersphinx 404#5538

Merged
AntoineRichard merged 2 commits intoisaac-sim:developfrom
hujc7:jichuanh/disable-viewergl-flaky
May 8, 2026
Merged

[CI unblock] Skip viewergl-fully-black test + fix warp intersphinx 404#5538
AntoineRichard merged 2 commits intoisaac-sim:developfrom
hujc7:jichuanh/disable-viewergl-flaky

Conversation

@hujc7
Copy link
Copy Markdown
Collaborator

@hujc7 hujc7 commented May 8, 2026

Summary

Two unrelated CI breakages on develop, bundled here so develop turns green in one PR.

1. Skip the failing viewergl test

test_cartpole_newton_visualizer_viewergl_rgb_motion[physx,newton] started returning all-black frames on develop after nvcr.io/nvidian/isaac-sim:latest-develop flipped to a Kit 110.1.1 + USD 25.11 base. The failure has been deterministic across multiple PRs (#5523, #5495, #5408, …).

Investigation so far has ruled out:

Diagnostic output (PR #5540 v2):

[VIZDIAG] fbo=c_uint(8)  pbo=None  size=600x600
[VIZDIAG] glGetError before: GL_NO_ERROR
[VIZDIAG] CPU-readback: nonzero=0/1080000  max=0  err=GL_NO_ERROR
[VIZDIAG] PBO-result: nonzero=0/1080000  max=0

The FBO itself is empty — Newton's pyglet/EGL renderer is not depositing pixels under Kit 110.1.1, even though tiled_camera_rgb_non_black (Kit RTX path) on the same env passes. Underlying root cause still being chased; this PR ships the skip to unblock develop.

2. Fix warp intersphinx 404 in docs build

https://nvidia.github.io/warp/objects.inv started returning 404 — Warp's objects.inv only lives at /stable/ and /latest/ now. With Sphinx's warnings_treated_as_errors, the broken intersphinx fetch fails the docs build on every PR. Pinning to /stable/ (matches the existing PyTorch /docs/2.11/ workaround pattern in the same file).

Verified https://nvidia.github.io/warp/stable/objects.inv returns 200.

Test plan

  • CI isaaclab_visualizers on this branch — was passing earlier with the skip; will re-verify with the bundled docs fix
  • CI Build Latest Docs on this branch — must turn green (was failing on every recent PR before this fix)

Re-enable plan

Once the underlying viewergl bug is identified and fixed, drop the @pytest.mark.skip decorator and remove the jichuanh-disable-viewergl-flaky.skip fragment.

ViewerGL.get_frame returns a fully-black 600x600x3 buffer in CI on the
current Isaac Sim image + Newton 1.2.0rc2 + warp-lang 1.13 cohort. The
failure is deterministic across two consecutive reruns of the same SHA
and reproduces on every recent PR that touches the rendering / camera /
sensor / USD stack:

  isaac-sim#5523 (Newton 1.2.0rc2 bump)
  isaac-sim#5534 (Camera/Renderer warp rewrite)
  isaac-sim#5024 (heterogeneous dexsuite)
  isaac-sim#5495 (usd-core 25.11)
  isaac-sim#5492 (OVRTX rendering tests)

9 visualizer-job failures across these 5 PRs in the last 100 build.yaml
runs, zero failures on PRs outside that scope.

Investigation ruled out:
  * rc1 -> rc2 viewer code diff -- only a 7-line image_logger.clear
    addition.
  * wp.RegisteredGLBuffer API -- byte-identical between warp-lang 1.12
    and 1.13.
  * pure CI flakiness -- deterministic across reruns.
  * the bump cohort alone -- branches on warp-lang 1.12 + Newton @a27277e
    both pass and fail this test depending on PR content.

Strongest remaining hypothesis: a CUDA-OpenGL interop init-order
fragility in the PBO + glReadPixels + RegisteredGLBuffer.map readback
path that gets tipped to deterministic-fail by any source change
perturbing GL/CUDA bring-up. Pinning it down requires py-spy or
glReadPixels return-code inspection inside the CI container, which is
out of scope for this change.

Skip both ['physx', 'newton'] parametrizations until the root cause
is identified. Re-enable in a follow-up PR.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label May 8, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 8, 2026

Greptile Summary

This PR skips test_cartpole_newton_visualizer_viewergl_rgb_motion (both physx and newton parametrizations) while a root cause for the deterministic all-black ViewerGL.get_frame output in CI is investigated. An empty .skip changelog fragment is also added to avoid a version bump for the test-only change.

  • @pytest.mark.skip is placed above @pytest.mark.parametrize, which correctly propagates the skip to all parametrized variants; no test logic is changed.
  • Investigation evidence in the PR description rules out several false leads (Newton rc1→rc2 diff, warp API change, pure flakiness) and points to a CUDA-OpenGL interop init-order fragility as the likely cause.

Confidence Score: 5/5

Safe to merge — a single skip decorator is added to one test function; no production code is touched.

The change is limited to adding @pytest.mark.skip above @pytest.mark.parametrize, which is the correct placement to skip all parametrized variants. The empty changelog fragment follows established project convention. No logic, APIs, or production paths are affected.

No files require special attention.

Important Files Changed

Filename Overview
source/isaaclab_visualizers/test/test_visualizer_cartpole_integration.py Adds @pytest.mark.skip above @pytest.mark.parametrize on test_cartpole_newton_visualizer_viewergl_rgb_motion; both [physx,newton] and [newton,newton] variants will be skipped correctly.
source/isaaclab_visualizers/changelog.d/jichuanh-disable-viewergl-flaky.skip Empty changelog fragment with .skip extension; prevents a version bump for a test-only change, consistent with the precedent set in PR #5508.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["test_cartpole_newton_visualizer_viewergl_rgb_motion"] -->|"@pytest.mark.skip"| B["SKIPPED — both parametrizations"]
    B --> C["[physx, newton]"]
    B --> D["[newton, newton]"]
    C -.->|"pending root-cause fix"| E["Re-enable in follow-up PR"]
    D -.->|"pending root-cause fix"| E
Loading

Reviews (1): Last reviewed commit: "Skip test_cartpole_newton_visualizer_vie..." | Re-trigger Greptile

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 8, 2026
@hujc7 hujc7 changed the title [test] Skip viewergl cartpole motion test pending root-cause [CI unblock] Skip viewergl-fully-black test + fix warp intersphinx 404 May 8, 2026
https://nvidia.github.io/warp/objects.inv now returns 404; the inventory
moved under /stable/ (and /latest/). With warnings treated as errors, the
broken intersphinx fetch fails the docs build on every PR.

Repro: any recent CI run on develop, e.g.
https://github.com/isaac-sim/IsaacLab/actions/runs/25533637651/job/74955918336

  WARNING: failed to reach any of the inventories with the following issues:
  intersphinx inventory 'https://nvidia.github.io/warp/objects.inv' not
    fetchable due to 404 Client Error: Not Found
  build finished with problems, 1 warning (with warnings treated as errors).
@hujc7 hujc7 force-pushed the jichuanh/disable-viewergl-flaky branch from be1553e to 0de218c Compare May 8, 2026 05:36
@AntoineRichard AntoineRichard merged commit a784ed9 into isaac-sim:develop May 8, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants