Transformations: Isolated CI JIT test#696
Draft
mlange05 wants to merge 4 commits into
Draft
Conversation
Provide a reusable run_isolated utility for JIT tests that need native extension work to happen in a short-lived child process, preventing loaded f90wrap state from leaking into the pytest worker. Cover result propagation, Python exceptions, and non-zero child exits.
Run the compile-and-execute phase of the parametrise transformation tests in a short-lived child process. This prevents f90wrap modules and native Fortran symbols loaded by one parametrised test case from leaking into later cases scheduled on the same xdist worker.
Document the expected area-prefixed subject format and wrapped body style so future local commits follow the repository convention consistently.
Allow isolated JIT helpers to use a clean spawn context and to exit immediately after reporting a successful result. This avoids f90wrap or Fortran finalizers crashing during child interpreter shutdown while still surfacing Python exceptions and non-zero exits. Use the hardened isolation in parametrise transformation tests and keep generated module names unique so wider xdist runs cannot alias native symbols across concurrently compiled cases.
|
Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/696/index.html |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #696 +/- ##
==========================================
- Coverage 94.20% 94.16% -0.05%
==========================================
Files 288 288
Lines 47625 47686 +61
==========================================
+ Hits 44867 44903 +36
- Misses 2758 2783 +25
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR aims to address a long-standing issue with racy failure in the transformation test base. In particular,
test_parametrise.pycreates multiple variations of the same module procedures with varying signatures and no renaming for the various test variations, leading to a racy situation where the assignment of two subsequent variants in the same thread can cause symbol aliasing and thus trigger mem corruption and sigsevs. To prevent this, simple de-aliasing techniques are not sufficient, as the compiled fortran symbols are still loaded dynamically into the Python runtime within the same process.To fix this, this PR adds a small
run_isolatedutility that executes a helper method in a short-lived child process, allowing thecompile-and-checkutility in the problematic test to be forced onto a separate process and thus avoid dynamic symbol aliasing.Contributor Declaration
By opening this pull request, I affirm the following: