fix(ci): raise AVM check-circuit per-tx timeout to 120s#24672
Closed
AztecBot wants to merge 1 commit into
Closed
Conversation
Collaborator
Author
|
Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Raise the per-transaction timeout in
avm_check_circuit_cmdsfrom30sto120s(yarn-project/end-to-end/bootstrap.sh).Root cause
The
avm-check-circuitjob of the AVM Circuit Inputs Collection and Check workflow failed onnext(run 29287694175, exit code124).ci-avm-check-circuitdownloads the cached AVM inputs and runsbb-avm avm_check_circuiton every dumped tx viaavm_check_circuit_cmds | parallelize 16. Each command runs underTIMEOUT=30s. In this run 643 of 644 checks passed in 1–4s each, but one heavy transaction tripped the timeout:The
e2e_multiple_blobstx has a large AVM trace and ran31s— just over the30scap.timeout -vkilled it (exit124), and becauseparallelizerunsparallel --halt now,fail=1, that single timeout failed the entire job.This is not caused by the head commit (a docs-only release PR, #24664); the check runs on the tree-hashed cached inputs and is unrelated to docs. It's a pre-existing borderline-heavy check running under a too-tight timeout — exactly the risk the code comment already flagged ("transactions could need more CPU and MEM than we allocate by default … they might start timing out"). A previous attempt to relieve this reduced parallelism (#24458, "It was timing out"); with the heaviest tx now sitting right at the boundary it regressed again.
Fix
Give the check generous headroom over the slowest observed tx (
31s) so a single slow run under 16-way CPU contention no longer fails the whole job.avm_check_circuitis a correctness gate — wall-time doesn't matter as long as it's bounded, and a120scap still catches genuine hangs (normal txs finish in seconds). Updated the stale comment to match.Note
This is a recurring failure that ClaudeBox has been auto-dispatched on many times (dozens of stale
cb/avm-check-circuit-timeout*/cb/avm-cc-timeout*branches exist, none merged —nextstill carries30s). Worth actually landing a headroom bump so the auto-dispatch stops firing on it.Testing
No local red/green repro is feasible: the check runs
bb-avm avm_check_circuiton an EC2 instance against the S3-cachede2e-avm-circuit-inputs-<tree-hash>.tar.gzwith the freshly built AVM binary — none of which is reproducible in this container. The change is a one-line timeout bump validated against the failing job's own log (the timed-out tx took31s, well within the new120s).Created by claudebox · group:
slackbot