WIP Migrate to pixi

[crusaderky]

Follow-up to dask/dask#12389

• drastically simplify deployment for developers, particularly when juggling multiple environments
• drastically improve reproducibility between CI and desktop
• prevent CI from failing overnight - with the exception of the upstream workflows, which retain the old behaviour by design

Functional changes

• The Tests workflow is still triggered on schedule in order to detect test flakiness, but is no longer affected by upstream changes.

• ⚠️ The Tests workflow no longer automatically tracks the dask/dask git tip. ⚠️ If a PR in dask/dask breaks something in dask/distributed CI, you will still notice it in the Upstream workflow.

• Added Upstream workflow, which features two runs:

  • nightly (nightly wheels and git tip of key dependencies)
  • python 3.14 (all latest releases of all dependencies).

  Upstream / py314 differs from Tests / py314 as Tests uses the lockfile, meaning dependency versions don't change on their own, whereas Upstream throws out the lockfile, getting all latest and greatest packages as soon as they are released. In other words, Upstream / py314 replicates the behaviour of the old Tests workflow.

• Repaired the Conda build workflow, which has been broken for some time. Now Conda build runs on all PRs (takes under 5 minutes).

• Run tests on Linux ARM

• Tweaked the triggers of some CI jobs for sanity

Follow-ups

• Migrate test report to pixi (depends on dask/distributed)
• Add pixi.toml to dask/partd and dask/zict (short of a full migration to pixi) and reintroduce their git tips to the nightly environment. This is low priority as these packages barely see any movement nowadays.
• Run a new pixi task in CI to build and validate pypi wheels
• Use pixi to manage linters
• Use pixi to automate docs building locally
• Use pixi to run with cupy (no CI; only local)

Known issues

• pixi currently has poor support for SCM versioning. There are a few workarounds in this PR.
• github does not offer a way to hide line counts in lockfiles. This annoyingly causes bugous line counts (e.g. 51k lines in this PR!) whenever pixi.lock changes. For clarity, this is an issue shared by all package managers for all languages which use a lock file.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    2 files   -     28      2 suites   - 28   0s ⏱️ - 10h 19m 50s
2 720 tests  -  1 360  2 584 ✅  -  1 383  134 💤 +   22  2 ❌ +1 
5 422 runs   - 51 177  5 085 ✅  - 49 033  335 💤  - 2 145  2 ❌ +1 

For more details on these failures, see this check.

Results for commit dac43ed. ± Comparison against base commit d3c7b24.

This pull request removes 1447 and adds 87 tests. Note that renamed tests count towards both.

distributed.cli.tests.test_dask_scheduler ‑ test_signal_handling[Signals.SIGINT]
distributed.cli.tests.test_dask_scheduler ‑ test_signal_handling[Signals.SIGTERM]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_scheduler[Signals.SIGINT]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_scheduler[Signals.SIGTERM]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGINT-Nanny]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGINT-Worker]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGTERM-Nanny]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGTERM-Worker]
distributed.cli.tests.test_dask_ssh
distributed.cli.tests.test_dask_worker ‑ test_listen_address_ipv6[tcp://[::1]:---nanny]
…

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape2]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape3]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape4]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape2]
…

This pull request removes 59 skipped tests and adds 85 skipped tests. Note that renamed tests count towards both.

distributed.cli.tests.test_dask_ssh
distributed.cli.tests.test_dask_worker ‑ test_listen_address_ipv6[tcp://[::1]:---nanny]
distributed.cli.tests.test_dask_worker ‑ test_listen_address_ipv6[tcp://[::1]:---no-nanny]
distributed.dashboard.tests.test_components
distributed.dashboard.tests.test_scheduler_bokeh
distributed.dashboard.tests.test_worker_bokeh
distributed.deploy.tests.test_old_ssh
distributed.deploy.tests.test_ssh
distributed.diagnostics.tests.test_cudf_diagnostics
distributed.diagnostics.tests.test_memray
…

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape2]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape3]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape4]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape2]
…

This pull request skips 2 and un-skips 6 tests.

distributed.deploy.tests.test_ssh ‑ test_defer_to_old
distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2

distributed.cli.tests.test_dask_scheduler ‑ test_uvloop
distributed.cli.tests.test_dask_spec ‑ test_uvloop
distributed.cli.tests.test_dask_worker ‑ test_uvloop[--nanny]
distributed.cli.tests.test_dask_worker ‑ test_uvloop[--no-nanny]
distributed.tests.test_config ‑ test_uvloop
distributed.tests.test_dask_collections ‑ test_sparse_arrays

♻️ This comment has been updated with latest results.