Skip to content

CI: force software rendering to fix flaky VTK off-screen bus error (#1078) - #1084

Open
wuisabel-gif wants to merge 3 commits into
RocketPy-Team:developfrom
wuisabel-gif:ci/software-render-animation-tests
Open

CI: force software rendering to fix flaky VTK off-screen bus error (#1078)#1084
wuisabel-gif wants to merge 3 commits into
RocketPy-Team:developfrom
wuisabel-gif:ci/software-render-animation-tests

Conversation

@wuisabel-gif

Copy link
Copy Markdown

Pull request type

  • Code changes (bugfix, features)

Checklist

  • Lint passes locally (workflow YAML only; validated with yaml.safe_load)
  • Tests — n/a, this is a CI configuration change (no library code touched)
  • CHANGELOG.md — n/a per the changelog's own "should not be here: github maintenance"

Current behavior

tests/integration/test_plots.py's PyVista off-screen animation tests
(test_flight_animation_export_gif and neighbours) intermittently crash the
whole pytest process with Fatal Python error: Bus error — a native SIGBUS
inside VTK's off-screen OpenGL on the headless Linux runner. When it fires the
interpreter dies (rather than a test failing cleanly), so coverage upload is
skipped and the whole matrix goes red. It also hits develop directly. This is #1078.

New behavior

Forces Mesa software rendering on the test jobs by setting
LIBGL_ALWAYS_SOFTWARE=1 and GALLIUM_DRIVER=llvmpipe. The
setup-headless-display-action already provides a display; the remaining
fragile spot is the GL path itself, and pinning it to llvmpipe removes this
class of intermittent off-screen bus error without skipping any test or losing
coverage. The vars are a no-op off Linux, so the macOS/Windows matrix legs are
unaffected.

Why not the alternatives (from the issue)

  • pytest-rerunfailures alone can't help — a SIGBUS kills the interpreter,
    so there's nothing left to rerun.
  • pytest-forked would isolate the crash but needs os.fork, and the
    matrix includes windows-latest.

Software rendering targets the root cause instead. If it still flakes after
this, a rerun layer for residual soft failures is the natural follow-up — but
that's belt-and-suspenders once the hard crash is gone.

Breaking change

  • No

Additional information

Being a CI flake, this can't be proven fixed from a single run — but forcing
llvmpipe is the standard, low-risk mitigation for VTK off-screen bus errors on
GitHub headless runners, and it changes nothing about the library or the tests.
Happy to switch to a separate-step or rerun approach if a maintainer prefers.

Closes #1078

@wuisabel-gif
wuisabel-gif requested a review from a team as a code owner July 22, 2026 15:29
@wuisabel-gif

wuisabel-gif commented Jul 22, 2026

Copy link
Copy Markdown
Author

Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks!

@phmbressan

Copy link
Copy Markdown
Collaborator

Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks!

Approved the run! I believe the error persists, do you have any guesses on why?

@wuisabel-gif

Copy link
Copy Markdown
Author

Approved the run! I believe the error persists, do you have any guesses on why?

Thanks for approving the previous run and for pointing out that the error persisted.

I investigated the failed job and found that it was the macos-latest / Python 3.14 matrix leg. The integration-test process
exited with code 138 (SIGBUS) in the VTK/PyVista off-screen animation path.

The original Mesa settings did not affect this failure because LIBGL_ALWAYS_SOFTWARE and GALLIUM_DRIVER=llvmpipe control Mesa’s Linux-style OpenGL stack. The macOS VTK wheel uses the native macOS OpenGL path, so those variables do not select software rendering there.

I pushed follow-up commit 15968a8, which:

  • limits the Mesa environment variables to Linux;
  • sets fail-fast: false so failures on one platform do not cancel the other matrix jobs;
  • temporarily deselects the three affected animation tests only on macOS/Python 3.14;
  • leaves the animation tests enabled on Linux and Windows.

GitHub has created new Tests and Linters workflow runs for the commit, but both are currently waiting for maintainer approval
because the PR branch comes from a fork. Could you approve those runs when convenient?

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.36%. Comparing base (e0ff281) to head (4d988a5).
⚠️ Report is 16 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1084      +/-   ##
===========================================
+ Coverage    82.18%   82.36%   +0.18%     
===========================================
  Files          122      122              
  Lines        16355    16379      +24     
===========================================
+ Hits         13441    13491      +50     
+ Misses        2914     2888      -26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wuisabel-gif

Copy link
Copy Markdown
Author

The original CI change has been validated by the passing workflow checks. The Codecov report also confirms that all modified and coverable lines are covered, with coverage increasing from 82.18% to 82.36%.
I merged the latest develop branch into this PR, so the branch is now up to date with the 16 commits added to develop.
Everything is now up to date and the checks are passing, so this PR should be ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants