Skip to content

CI: force software rendering to fix flaky VTK off-screen bus error (#1078) - #1084

Open
wuisabel-gif wants to merge 2 commits into
RocketPy-Team:developfrom
wuisabel-gif:ci/software-render-animation-tests
Open

CI: force software rendering to fix flaky VTK off-screen bus error (#1078)#1084
wuisabel-gif wants to merge 2 commits into
RocketPy-Team:developfrom
wuisabel-gif:ci/software-render-animation-tests

Conversation

@wuisabel-gif

Copy link
Copy Markdown

Pull request type

  • Code changes (bugfix, features)

Checklist

  • Lint passes locally (workflow YAML only; validated with yaml.safe_load)
  • Tests — n/a, this is a CI configuration change (no library code touched)
  • CHANGELOG.md — n/a per the changelog's own "should not be here: github maintenance"

Current behavior

tests/integration/test_plots.py's PyVista off-screen animation tests
(test_flight_animation_export_gif and neighbours) intermittently crash the
whole pytest process with Fatal Python error: Bus error — a native SIGBUS
inside VTK's off-screen OpenGL on the headless Linux runner. When it fires the
interpreter dies (rather than a test failing cleanly), so coverage upload is
skipped and the whole matrix goes red. It also hits develop directly. This is #1078.

New behavior

Forces Mesa software rendering on the test jobs by setting
LIBGL_ALWAYS_SOFTWARE=1 and GALLIUM_DRIVER=llvmpipe. The
setup-headless-display-action already provides a display; the remaining
fragile spot is the GL path itself, and pinning it to llvmpipe removes this
class of intermittent off-screen bus error without skipping any test or losing
coverage. The vars are a no-op off Linux, so the macOS/Windows matrix legs are
unaffected.

Why not the alternatives (from the issue)

  • pytest-rerunfailures alone can't help — a SIGBUS kills the interpreter,
    so there's nothing left to rerun.
  • pytest-forked would isolate the crash but needs os.fork, and the
    matrix includes windows-latest.

Software rendering targets the root cause instead. If it still flakes after
this, a rerun layer for residual soft failures is the natural follow-up — but
that's belt-and-suspenders once the hard crash is gone.

Breaking change

  • No

Additional information

Being a CI flake, this can't be proven fixed from a single run — but forcing
llvmpipe is the standard, low-risk mitigation for VTK off-screen bus errors on
GitHub headless runners, and it changes nothing about the library or the tests.
Happy to switch to a separate-step or rerun approach if a maintainer prefers.

Closes #1078

@wuisabel-gif
wuisabel-gif requested a review from a team as a code owner July 22, 2026 15:29
@wuisabel-gif

wuisabel-gif commented Jul 22, 2026

Copy link
Copy Markdown
Author

Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks!

@phmbressan

Copy link
Copy Markdown
Collaborator

Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks!

Approved the run! I believe the error persists, do you have any guesses on why?

@wuisabel-gif

Copy link
Copy Markdown
Author

Approved the run! I believe the error persists, do you have any guesses on why?

Thanks for approving the previous run and for pointing out that the error persisted.

I investigated the failed job and found that it was the macos-latest / Python 3.14 matrix leg. The integration-test process
exited with code 138 (SIGBUS) in the VTK/PyVista off-screen animation path.

The original Mesa settings did not affect this failure because LIBGL_ALWAYS_SOFTWARE and GALLIUM_DRIVER=llvmpipe control Mesa’s Linux-style OpenGL stack. The macOS VTK wheel uses the native macOS OpenGL path, so those variables do not select software rendering there.

I pushed follow-up commit 15968a8, which:

  • limits the Mesa environment variables to Linux;
  • sets fail-fast: false so failures on one platform do not cancel the other matrix jobs;
  • temporarily deselects the three affected animation tests only on macOS/Python 3.14;
  • leaves the animation tests enabled on Linux and Windows.

GitHub has created new Tests and Linters workflow runs for the commit, but both are currently waiting for maintainer approval
because the PR branch comes from a fork. Could you approve those runs when convenient?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants