Commit 1907be8
authored
test: regenerate shard timings + add regenerate_timings.py (#1582)
This PR does NOT change adapter behavior — test-only.
## Problem
The integration matrix's wall-clock is set by its slowest shard.
`scripts/shard_assign.py` balances shards with greedy-LPT weighted by
per-file historical wall time from `.github/test_timings.json` — but
that file was last regenerated **2026-05-07** (89 files/profile), before
1.12.1's and 1.12.2's tests existed. New/changed test files were absent
from the timings, so the balancer mean-filled them and could not see
which files are heavy → shards drifted out of balance. In a recent run
the slowest `uc_cluster` shard ran **~59 min** while its siblings
finished in ~35.
`scripts/refresh_timings.py` exists to rebuild the data but is wired
into no workflow and only aggregates one profile from one run at a time.
## What this PR does
1. **Regenerates `.github/test_timings.json`** from recent green
`integration.yml` runs — now **117 files/profile** (up from 89), so the
~28 newer test files get real weights.
2. **Adds `scripts/regenerate_timings.py`** — an automatable,
all-profiles-at-once front end to `refresh_timings.py`. It discovers
green integration runs (dedup by SHA), downloads every profile's
per-shard junit artifacts (with a transient-401 retry), and writes the
**median** per-file wall time across runs — median rather than mean so
one slow-warehouse run doesn't skew a file's weight. Output format is
byte-identical to `refresh_timings.py`, whose junit-parsing it reuses.
## Projected rebalance (greedy-LPT max-shard wall, scored against the
new timings)
| Profile | Old max | New max | Δ |
|---|---|---|---|
| databricks_cluster | 362.7m | 340.8m | −21.9m |
| databricks_uc_cluster | 354.8m | 278.8m | **−76.0m** |
| databricks_uc_sql_endpoint | 297.3m | 281.3m | −16.0m |
uc_cluster shard loads go from 355/246/236 (a 45% imbalance) to
279/279/279. (These are summed junit test-times — no
fixture/cluster-startup overhead — so absolute minutes run higher than
real wall-clock, but the relative rebalance is what matters.)
## Verification
- Regenerated deterministically (median + sorted output → byte-identical
on re-run).
- All 117 keys/profile resolve to real test files; schema unchanged
(`shard_assign.py` consumes it verbatim).
- `ruff check` + `ruff format` clean.
- `refresh_timings.py` and `shard_assign.py` are byte-identical to
`main` (only the JSON data + the new script).
## Notes
- Sourced from the 2 distinct-SHA green runs whose artifacts were still
within GitHub's retention window at regeneration time (median-of-2).
Re-running the script over 3+ runs when more are retained is a
one-command refresh.
- Follow-up (not here): wire `regenerate_timings.py` into the nightly so
the timings never go stale again.1 parent e318a44 commit 1907be8
2 files changed
Lines changed: 624 additions & 267 deletions
0 commit comments