CI: the test-polars py3.12 coverage cell is at its timeout ceiling; every polars test addition re-triggers a TIMEOUT reported as `cancelled`

[lmeyerov]

Summary

The test-polars py3.12 cell is at its wall-clock ceiling. It is the only cell in the matrix that runs coverage, and it shares one timeout-minutes with five siblings that do half its work. Adding a single test file to bin/test-polars.sh is now enough to push it over, and when it goes over GitHub reports the job as cancelled — which reads as unrelated fail-fast collateral, not as a budget failure.

This is a structural capacity limit, not a one-off. It will re-fire on the next polars test addition.

Evidence

On PR #1794 (which appends one file to POLARS_TEST_FILES), test-polars (3.12) failed twice with zero test failures:

attempt	duration	outcome
2	10m15s	cancelled
3	10m11s	cancelled

Job log, attempt 3:

2377 passed, 13 skipped ... in 572.43s (0:09:32)
The operation was canceled.

The pytest phase alone was 9m32s of a 10m job budget. Siblings (3.9/3.10/3.11/3.13/3.14) finish in 4m09s–5m51s. A rerun does not clear it — it is 0-for-2.

Two things made this expensive to diagnose:

1. A timed-out job is reported as cancelled, indistinguishable at a glance from fail-fast collateral. The distinguishing evidence is the duration: a fail-fast cancel lands seconds after a sibling failure; a timeout lands at exactly timeout-minutes. Attempt 2 was 1 cancelled / 72 success / 1 skipped, so collateral was ruled out by evidence.
2. changed-line-coverage then reports SKIPPED, because it needs: the job that produces polars-coverage-py3.12. So one budget overrun silently disables a second gate.

Root cause

.github/workflows/ci.yml:

• test-polars declares timeout-minutes: 10 for the whole matrix job (was line 1579).
• Only the py3.12 cell runs Polars tests with coverage (POLARS_COV=1, was line 1618) plus the per-file Polars GFQL coverage audit plus an artifact upload.

So one cell carries roughly double the work of its siblings against a budget sized for the siblings. The timeout is simultaneously too tight for the cell that matters and too loose to be a useful hang detector for the ones that don't.

Fix applied in #1794

Split the coverage pass into its own job, test-polars-coverage (py3.12, timeout-minutes: 20), and drop 3.12 from the test-polars matrix. Net effect:

• test-polars cells now do identical work, so its 10-minute budget is a real hang detector again.
• The coverage pass has an explicit, separately visible budget that can grow without cancelling five unrelated cells.
• No python version loses a run: py3.12 was already only running the coverage step, so this is the previous step layout re-homed.
• changed-line-coverage needs: updated to include the new job (it consumes that job's artifact).

Note there is no required_status_checks configuration on master (verified via the branches API), so introducing a new check name does not block merges. If required checks are added later, test-polars-coverage should be in the list.

Still open after that fix

The 20 minutes buys headroom, it does not remove the ceiling. The polars lane is:

• serial — bin/test-polars.sh runs plain python -B -m pytest with no -n auto, unlike test-gfql-core which does use it. This is the single biggest available lever and is not taken.
• allowlist-driven — POLARS_TEST_FILES grows monotonically, and every addition lands on the coverage lane at ~2x cost. Related: test-polars.sh is an allowlist: a new polars test file is silently never run until someone remembers to register it #1795 (new test files not in the allowlist run in no polars lane at all).

Candidate follow-ups, roughly in order of value:

1. Add -n auto to the polars pytest invocations (coverage under xdist works via coverage combine; needs pytest-xdist confirmed in the test-polars-* locks).
2. Shard POLARS_TEST_FILES across two coverage cells and combine the two .coverage files before the audit.
3. Have the coverage lane report its own wall-clock into the step summary, so approaching the ceiling is visible before it fails.

Acceptance

• A polars test-file addition of typical size does not cancel any CI job.
• A cancelled polars job is attributable to a real cause, not to a shared budget.

[lmeyerov]

The patch implementing the split is posted on #1794 (#1794 (comment)) rather than committed to that branch: the token available to the agent lacks the workflow OAuth scope, so a commit touching .github/workflows/ci.yml is rejected at push time. It needs a maintainer to apply it. Until then test-polars (3.12) will keep timing out on any PR that adds polars test work.

[lmeyerov]

Measured evidence for follow-up 1 (-n auto)

Ran bin/test-polars.sh locally with cuDF import-blocked (so the engine set matches the CI polars lane), -p no:randomly, same tree, back to back:

workers	result	wall clock
serial (today)	2395 passed, 54 skipped, 0 failed	172.21s
-n 4 (GitHub runner core count)	2395 passed, 54 skipped, 0 failed	59.33s

2.9x, with an identical pass/skip set — no test-ordering or shared-state breakage at 4 workers. -n 4 rather than -n auto deliberately, to emulate a 4-core runner rather than report a wide-machine number that would not transfer.

Two caveats before anyone lands this:

1. Not measured under coverage. pytest-cov is not installed on the box I measured on, so POLARS_COV=1 could not be exercised. The combination is not speculative — test-gfql-core already runs pytest -n auto --cov=graphistry/compute in this same workflow (ci.yml:1166) and the audit reads the resulting data file — but the polars lane's own coverage plumbing plus the per-file audit floors should be confirmed on a runner before this is relied on.
2. Memory. Four workers multiply peak RSS, and parts of this suite build large frames. The gfql-core precedent suggests it is fine on a 16 GB runner, but it is the failure mode to watch.

Deliberately NOT included in #1794: that PR is about variable-length query semantics, and changing the parallelism of a lane used by six python versions is a separate decision from fixing the shared-budget defect. Filed here as the evidence, not the change.

[lmeyerov]

The lane is a coin flip, not a fixed problem — measured

#1794 came back fully green on run 30312253135, including test-polars (3.12). That is not evidence the ceiling is gone. Job durations from the GitHub API, same job, near-identical workload (the PR's new test file measured +1.1s over the one it replaces):

run	test-polars (3.12)	outcome
30304440110 attempt 2	615s	cancelled (timeout at 600s)
30304440110 attempt 3	611s	cancelled (timeout at 600s)
30312253135	431s	success, 169s headroom

The siblings show the same runner variance across those runs (3.11: 249s → 361s; 3.14: 351s → 329s), so the 611s → 431s move is scheduling/runner luck, not a workload change.

Concretely: this lane sits close enough to its 600s budget that the same commit can pass or time out depending on which runner it lands on. A green run must not be read as closing this issue — the fix is still the split (patch on #1794) and, longer term, follow-up 1.

[lmeyerov]

Corroborating measurement, from #1805 (which widens bin/test-polars.sh by ten files):

	test-polars (3.12) duration	budget
master 233b64c8	8m21s	timeout-minutes: 10
#1805 head 3de36710	9m57s	timeout-minutes: 10

So master is already at 84% of the budget with no change at all, and #1805 passes with a
3-second margin — green, but not meaningfully green. The other five cells are unaffected
(3.9 4m20s, 3.10 4m29s, 3.11 5m40s, 3.13 5m25s, 3.14 5m55s), which is exactly the asymmetry the
split in this issue is aimed at: one cell carries the coverage pass plus the per-file audit while
sharing a timeout sized for the plain ones.

Practical consequence for sequencing: #1805 should merge after the ci.yml split lands,
otherwise the next polars test addition — or a slower-than-usual runner on the same commit —
turns this cell into a cancelled with zero test failures in the log.

[lmeyerov]

PR #1814 implements the xdist lever from this issue, and it is now measured ON CI rather than projected.

The -n 4 number quoted when this issue was filed (59.33s vs 172.21s, 2.9x) was measured without coverage, because pytest-cov was absent from that environment — which was the whole risk, since 3.12 is the coverage cell. That gap is closed: coverage + xdist was verified end to end, and then the branch was run through CI.

Measured, test-polars cells, master run 30300891340 vs PR run 30322706850:

cell	master	PR #1814	speedup
3.12 (coverage + audit)	501s (script 484s)	322s (script 297s)	1.56x cell / 1.63x script
3.9	247s	139s	1.93x
3.10	266s	140s	2.03x
3.11	298s	145s	2.24x
3.13	340s	139s	2.63x
3.14	314s	186s	1.74x

The CI speedup is smaller than the local one (a 4-CPU-pinned local A/B with coverage on gave 2.98x): coverage tracing is per-worker CPU cost that does not parallelize away, the --cov-append second phase stays serial, and 4x interpreter startup is fixed. 1.63x is the number to quote for the coverage cell.

Correctness was verified rather than assumed, since a coverage regression caused by a harness bug would look like a real coverage regression in changed-line-coverage:

• node-id sets (not counts) identical serial vs parallel — 2417 node ids, 2404 passed / 13 skipped — across 4-worker, 2-worker, --dist loadfile and 4-CPU-pinned runs
• merged coverage is a strict superset of serial (28,480 vs 28,478 lines; zero lost, zero files dropped)
• the --cov-append phase still appends into the xdist-produced data file (+1,791 lines, nothing lost)
• bin/coverage_audit.py --profile gfql-polars emits a byte-identical report from the parallel data, and coverage combine + coverage report (what changed-line-coverage does) gives identical totals

Effect on the ordering constraint: #1805's 3.12 cell measured 597s against the 600s timeout — green by 3 seconds. At the measured 1.63x that becomes ~373s (~62% of budget), i.e. ~3.8 minutes of margin instead of 3 seconds. #1805 no longer has to wait on the ci.yml job split, which remains a good idea but is no longer a prerequisite that only a human with workflow scope can unblock.