Skip to content

Commit 1907be8

Browse files
authored
test: regenerate shard timings + add regenerate_timings.py (#1582)
This PR does NOT change adapter behavior — test-only. ## Problem The integration matrix's wall-clock is set by its slowest shard. `scripts/shard_assign.py` balances shards with greedy-LPT weighted by per-file historical wall time from `.github/test_timings.json` — but that file was last regenerated **2026-05-07** (89 files/profile), before 1.12.1's and 1.12.2's tests existed. New/changed test files were absent from the timings, so the balancer mean-filled them and could not see which files are heavy → shards drifted out of balance. In a recent run the slowest `uc_cluster` shard ran **~59 min** while its siblings finished in ~35. `scripts/refresh_timings.py` exists to rebuild the data but is wired into no workflow and only aggregates one profile from one run at a time. ## What this PR does 1. **Regenerates `.github/test_timings.json`** from recent green `integration.yml` runs — now **117 files/profile** (up from 89), so the ~28 newer test files get real weights. 2. **Adds `scripts/regenerate_timings.py`** — an automatable, all-profiles-at-once front end to `refresh_timings.py`. It discovers green integration runs (dedup by SHA), downloads every profile's per-shard junit artifacts (with a transient-401 retry), and writes the **median** per-file wall time across runs — median rather than mean so one slow-warehouse run doesn't skew a file's weight. Output format is byte-identical to `refresh_timings.py`, whose junit-parsing it reuses. ## Projected rebalance (greedy-LPT max-shard wall, scored against the new timings) | Profile | Old max | New max | Δ | |---|---|---|---| | databricks_cluster | 362.7m | 340.8m | −21.9m | | databricks_uc_cluster | 354.8m | 278.8m | **−76.0m** | | databricks_uc_sql_endpoint | 297.3m | 281.3m | −16.0m | uc_cluster shard loads go from 355/246/236 (a 45% imbalance) to 279/279/279. (These are summed junit test-times — no fixture/cluster-startup overhead — so absolute minutes run higher than real wall-clock, but the relative rebalance is what matters.) ## Verification - Regenerated deterministically (median + sorted output → byte-identical on re-run). - All 117 keys/profile resolve to real test files; schema unchanged (`shard_assign.py` consumes it verbatim). - `ruff check` + `ruff format` clean. - `refresh_timings.py` and `shard_assign.py` are byte-identical to `main` (only the JSON data + the new script). ## Notes - Sourced from the 2 distinct-SHA green runs whose artifacts were still within GitHub's retention window at regeneration time (median-of-2). Re-running the script over 3+ runs when more are retained is a one-command refresh. - Follow-up (not here): wire `regenerate_timings.py` into the nightly so the timings never go stale again.
1 parent e318a44 commit 1907be8

2 files changed

Lines changed: 624 additions & 267 deletions

File tree

0 commit comments

Comments
 (0)