chore(hub-e2e): restore parallel smoke-all workers (3)#384
Merged
Conversation
The smoke-all suite ran serially (workers: 1) on the theory that the 75s-timeout flakiness came from the WASM render pipeline stalling under CPU contention on a '2-core' runner. That diagnosis was wrong on both counts: the public-repo ubuntu-latest runner has 4 vCPUs, and the real cause was server-side sync contention — fixed by the samod-0.12 hub upgrade (PR #355) on 2026-07-03. Since #355 the nightly has been clean at workers: 1 for several consecutive nights. Dispatch stress runs on this branch then confirmed 78/78 with zero flaky at both 3 and 4 workers (~3.4m, vs the serial ~4.8m), proving the fix holds under concurrent load: - run 28840734138 — 4 workers, nproc=4, 78 passed, 0 flaky (3.4m) - run 28871855320 — 3 workers, nproc=4, 78 passed, 0 flaky (3.4m) Restore parallelism at 3 (reserves one core for the co-resident hub + vite-preview + node). SMOKE_ALL_WORKERS + the smoke-all-workers dispatch input remain as a knob to re-stress at 4 if flakiness ever returns.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Restore parallelism in the nightly
smoke-allE2E suite: default workers1→3, with aSMOKE_ALL_WORKERSenv /smoke-all-workersdispatch-inputoverride retained as a knob.
Why
The suite ran serially on the theory that the 75s-timeout flakiness came from
the WASM render pipeline stalling under CPU contention on a "2-core" runner.
That diagnosis was wrong on both counts:
ubuntu-latestrunner has 4 vCPUs, not 2 (confirmed vianprocin the stress runs);hub upgrade (PR chore(hub): migrate samod to quarto-dev/samod@access-policy (automerge 0.10) #355, 2026-07-03).
Since #355 the nightly has been clean at
workers: 1for several consecutivenights. Dispatch stress runs on this branch then confirmed the fix holds under
concurrent load:
vs the pre-#355 regime: 7–17 flaky per night, 14–30 min.
Choice of 3
Both 3 and 4 were clean and equally fast (~3.4m — the suite is I/O-bound at
this point, not core-starved). Landing on 3 reserves one core for the
co-resident hub +
vite preview+ node processes, keeping a robustness marginover the 4-vCPU ceiling. Re-stress at 4 any time via
workflow_dispatchwithsmoke-all-workers=4.Verification
4→ 4.smoke-all-workersinput default3.in-process tests).
Context:
claude-notes/research/2026-07-03-e2e-reliability-experiment-log.md(lead 2, samod-0.12).