Commit f1277b2
committed
feat(cli): add time-barrier wave staggering to e2e batch fan-out
The e2e CodeBuild batch fans out a large number of shards that all depend
only on `upb` (Linux) or `build_windows`/`upb` (Windows). Dispatching that
many builds as a single instantaneous burst is unreliable, so this change
spreads dispatch over time.
This change spreads dispatch over time using TIME barriers rather than
completion chaining. Shards are grouped into waves of `E2E_WAVE_SIZE`
(per OS). Wave 1 keeps its original dependency on `upb`/`build_windows`.
Each later wave is gated behind a synthetic barrier build whose only job
is to sleep `(k-1) * E2E_WAVE_BARRIER_INTERVAL_SEC` seconds. Crucially,
every barrier depends ONLY on `upb`/`build_windows` — never on the prior
barrier or the prior wave — so all barriers start in parallel right after
the package upload and the stagger comes purely from their differing
sleep durations. This cuts the per-scheduling-decision dispatch burst
without serializing wave completion, deliberately avoiding the serial
completion chain that timed out at the 240m batch limit previously.
Both knobs are tunable constants near the top of the generator. Barriers
are excluded from the e2e report wait list (`wait_for_ids.json`) since
they are synthetic and must not be polled by `aggregate_e2e_reports`.
Verified by regenerating `e2e_workflow_generated.yml`: 4 barriers
present (l/w waves 2 and 3), waves reference their barrier, wave 1 still
on `upb`/`build_windows`, no shard depends on another shard, no barrier
depends on another barrier, and the yaml parses cleanly.
---
Prompt: implement barrier-wave staggering, draft PR from dev, run e2e1 parent aa9d525 commit f1277b2
2 files changed
Lines changed: 300 additions & 160 deletions
0 commit comments