ci(polars): parallelize the polars lane so the coverage cell stops racing its 10-minute timeout

CHANGELOG.md

@@ -36,6 +36,9 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **Indexed bypass could return every node for `rows()` over an unnamed pattern**: the boundary accepted a `rows()` call carrying no binding ops over a middle with no aliases. That call reads the traversal-narrowed node table, but the bypass hands it the full graph, so the query would have returned all nodes. The gate now requires that the rows call actually consumes the whole middle as binding ops — either it already carries them, or the named-middle rewrite installs exactly them.
 - **Seeded property-RETURN dtype divergence on cuDF**: the lean projection applied the pandas rows-pivot artifact (int → float64, bool → object) on every engine, but cuDF's canonical pivot preserves the source dtypes — so the fast path returned `float64`/`object` where cuDF's own canonical path returns `int64`/`bool`. The cast rule is now engine-aware; the dtype-class decline guard is unchanged.
 
+### Infrastructure
+- **The polars CI lane runs in parallel, so the coverage cell stops racing its timeout**: `test-polars (3.12)` is the only cell that runs the coverage-instrumented pass *and* the per-file coverage audit, and it had already been cancelled twice at its 10-minute budget (615s, 611s) with zero test failures — pure budget exhaustion, with the same commit passing or failing depending on the runner it drew. `bin/test-polars.sh` now runs its main phase under `pytest-xdist` (`-n auto --maxprocesses 4 --dist load`); `pytest-xdist` was already in the `[test]` extra and `test-gfql-core` already runs `-n auto` under `--cov`, so no dependency changes. `auto` resolves to the runner's core count (4 on a GitHub-hosted `ubuntu-latest`) and scales *down* on a 2-vCPU runner where a fixed `-n 4` could be slower than serial; `--maxprocesses` keeps a many-core dev box from fanning out one polars process per core. Verified rather than assumed, on py3.12 / polars 1.43.1 / pandas 3.0.5 with `POLARS_COV=1`: the pass/skip **node-id sets** (not just counts) are identical serial vs parallel across 4-worker, 2-worker and `--dist loadfile` runs (2417 node ids, 2404 passed / 13 skipped every time); the merged coverage data is a strict superset of the serial data (28,480 vs 28,478 covered lines, zero lines lost, zero files dropped); the second `--cov-append` invocation still appends into the xdist-produced data file (+1,791 lines, nothing lost); and `bin/coverage_audit.py --profile gfql-polars` emits a byte-identical report (modulo its timestamp) from the parallel data, so the downstream `changed-line-coverage` check sees no change. Measured **on CI**, by this change's own run: the `test-polars (3.12)` cell goes **501s → 322s**, its script step **484s → 297s (1.63×)**, and the five non-coverage cells go 222–318s → 115–172s. (A local 4-CPU-pinned A/B with coverage on showed 326s → 109s, 2.98×; CI realizes less, because coverage tracing is per-worker CPU cost that does not parallelize away and the ~8s serial second phase plus 4× interpreter startup are fixed. The CI number is the one to quote.) `--dist load` rather than `loadfile` because one module (`test_engine_polars_chain.py`) is 69% of the lane's test time, capping file-level distribution at ~1.4×; `POLARS_XDIST=0` restores the serial path and `POLARS_XDIST_DIST` selects another distribution mode should a future order-dependent test need one.
+
 ### Documentation
 - **GFQL pay-as-you-go resident indexing user guide**: New :doc:`Pay-As-You-Go Resident Indexing <gfql/indexing>` page — the lifecycle guide to resident indexes (`gfql_index_all()` / `gfql_index_edges()` / `create_index()` / `show_indexes()` / `drop_index()`): what the node-id + CSR in/out adjacency sidecars are, what engages them on 0.58.0 (seeded typed-hop fast paths incl. property RETURNs and property-seeded lookups per #1768/#1770, direct `g.hop()`; the general polars chain traversal honestly noted as not yet covered), the staleness/validity contract (identity + fingerprint; rebind invalidates; declines are safe — identical results either way), engine notes (polars needs `gfql_index_all(engine='polars')` until #1767), 0.58.0-tag measured numbers, and a runnable end-to-end example. Wired into the GFQL toctree + recommended paths alongside :doc:`Seeded Traversal Indexes <gfql/index_adjacency>`.
 - **GFQL performance docs: 0.58.0 release-tag-verified numbers, siloed in one page**: `gfql/performance.rst` is now the canonical benchmark-numbers page (alongside `gfql/index_adjacency.rst` for the index benchmarks) — a benchmark rerun updates it alone. It carries the 0.58.0 tag sweep (DGX Spark GB10, warm medians N=30; four-engine numbers cross-engine parity-verified, competitor pairs validated against expected result rows): seeded typed-hop fast path across all four engines (e.g. pandas 29.9→2.46ms, 12.1×), native chain form, resident-index covered-shape lookups (with the `gfql_index_all(engine='polars')` caveat / PR #1767), flat seeded-hop scaling on pandas (0.159–0.164ms from 0.25M to 32M edges), the one-keyword `engine='polars'` LDBC SNB SF1 seed-lookup win (1,299.6→106.1ms, 12.3×), LDBC SNB interactive SF1 vs Neo4j 5.26 same-box pairs (GFQL 4 of 5; Neo4j wins recent-replies — reported as-is), OLAP multi-join vs embedded Kuzu (q8 200×, q9 14.2×) with the honest inverse (Kuzu wins single-table aggregates 2–4×, seeded property-projection lookups 2.4–64×), plus the prior Orkut/LiveJournal bulk sweep (moved from `engines.rst`, dated once) and its methodology. All other pages — `engines.rst`, `quick.rst`, `about.rst`, `overview.rst`, `index.rst` — now carry stable qualitative claims (e.g. "often an order of magnitude faster on query-heavy workloads") that link into `performance.rst` instead of inline figures, replacing the stale "up to ~38×" headlines and avoiding scattered per-claim version labels.

bin/test-polars.sh

@@ -5,6 +5,8 @@ set -ex
 # - Extra args are passed through to the pytest phase
 # - Set POLARS_COV=1 to collect coverage over graphistry/compute; the coverage
 #   data file location is taken from $COVERAGE_FILE (as the CI py3.12 lane sets it)
+# - Set POLARS_XDIST to pick the pytest-xdist worker spec (default `auto`); `0` forces
+#   the serial path, which is what an A/B of this lane compares against
 # - Non-zero exit code on fail
 
 # Assume [polars,test] installed
@@ -57,15 +59,40 @@ POLARS_TEST_FILES=(
     graphistry/tests/test_engine_frame_helpers.py
 )
 
+# PARALLELISM. The py3.12 cell of this lane is the coverage cell and has repeatedly run out
+# of its CI budget; xdist is the lever that does not require a workflow edit (pytest-xdist is
+# already in the [test] extra, and test-gfql-core already runs `-n auto` under --cov, so
+# coverage+xdist is an established combination in this repo).
+#   * worker spec `auto` = os.cpu_count(): 4 on a GitHub-hosted ubuntu-latest runner, and it
+#     scales DOWN on a 2-vCPU runner where a fixed `-n 4` could be slower than serial.
+#   * --maxprocesses caps the count so a 24-core dev box does not fan out 24 polars processes
+#     that then oversubscribe polars' own thread pool.
+#   * --dist load (xdist's default) balances per test. `loadfile` was measured too: it is
+#     bounded by the single largest module and only reaches 1.4x where `load` reaches 3.2x.
+#     No test in this lane depends on execution order or on cross-test module state, and the
+#     pass/skip node-id sets were compared serial-vs-parallel and are identical; POLARS_XDIST_DIST
+#     is the escape hatch if a future order-dependent test needs `loadfile`/`loadscope`.
+XDIST_ARGS=()
+if [ "${POLARS_XDIST:-auto}" != "0" ]; then
+    XDIST_ARGS=(
+        -n "${POLARS_XDIST:-auto}"
+        --maxprocesses "${POLARS_XDIST_MAX:-4}"
+        --dist "${POLARS_XDIST_DIST:-load}"
+    )
+fi
+
 COV_ARGS=()
 if [ -n "${POLARS_COV:-}" ]; then
     COV_ARGS=(--cov=graphistry --cov-report=)
 fi
 
-python -B -m pytest -vv "${COV_ARGS[@]}" "${POLARS_TEST_FILES[@]}" "$@"
+python -B -m pytest -vv "${XDIST_ARGS[@]}" "${COV_ARGS[@]}" "${POLARS_TEST_FILES[@]}" "$@"
 
 # cypher-lowering polars-parametrized cases (round ties, lower/upper, =~, numeric fns);
-# appended into the same coverage data file when POLARS_COV=1 (CI audit reads it)
+# appended into the same coverage data file when POLARS_COV=1 (CI audit reads it).
+# Left SERIAL on purpose: it is one module and ~8s of the lane, so worker startup would eat
+# the gain. Appending into the data file the xdist phase produced is verified — the merged
+# result is line-for-line identical to running the whole script in one go.
 COV_APPEND_ARGS=()
 if [ -n "${POLARS_COV:-}" ]; then
     COV_APPEND_ARGS=(--cov=graphistry --cov-report= --cov-append)