Examples of CI hanging or being flaky/weird

[jbusecke]

I have had multiple occasions where the VZ tests seem very flaky, which makes the dev experience harder.

This is not a full reproducer just a collection of where things went wrong:

• Test python 3.12, 3.13 #932 (comment) (Tests pass after one or multiple reruns)
• "Cannot import VirtualiZarrDatasetAccessor" (e.g. fix: add shutdown methods to executors #925 (comment)). I have noticed this locally and it was definitely flaky (went away when rerunning)

[TomNicholas]

are both of these in lithops tests do we know?

[TomNicholas]

(Maybe useful, Claude obviously)

Summary of the intermittent VirtualiZarrDatasetAccessor import failure in lithops tests

The tests test_lithops_executor_with_partial and test_lithops_executor_with_multiple_args intermittently fail with:
ImportError: cannot import name 'VirtualiZarrDatasetAccessor' from 'virtualizarr.accessor'
(/tmp/lithops-runner/modules/12f5e9-1-M000/virtualizarr/accessor.py)

The class definitely exists in the source code. The error path /tmp/lithops-runner/modules/... tells us the failure is happening in lithops' shipped copy of
the module, not the installed copy.

How lithops localhost mode works:

1. When a callable is submitted, lithops inspects it for module dependencies (serialize.py:_module_inspect)
2. It discovers virtualizarr as a dependency and ships all **/*.py files from the package (serialize.py:create_module_data)
3. It spawns a fresh subprocess via subprocess.Popen([runtime_name, runner_file, 'run_job', ...]) where runtime_name defaults to "python3"
   (localhost.py:324-325)
4. The subprocess unpickles the function and writes the shipped modules to /tmp/lithops-runner/modules/<job_key>/, adding that path to sys.path
   (worker/utils.py:67-91)
5. The subprocess then imports and runs the function

Theory: The subprocess resolves python3 to a different Python than the one running the tests (e.g. system Python vs pixi-managed Python). Since the subprocess
relies on the shipped module files rather than the installed package, any issue with the module shipping (incomplete glob, file write race under pytest-xdist
-n auto, or pixi's editable install layout confusing importlib.util.find_spec) results in an incomplete copy of virtualizarr, causing the import to fail.

The intermittent nature likely comes from: (a) the module dependency analysis (_module_inspect) using bytecode inspection which can produce different results
depending on import state, or (b) parallel test execution (pytest -n auto) causing concurrent lithops subprocesses to interfere with each other's module
directories.

We verified locally that create_module_data does correctly include accessor.py with the VirtualiZarrDatasetAccessor class, so the issue is specific to the CI
environment (pixi on Linux).