Skip to content

fix(ci): raise AVM check-circuit per-tx timeout to 120s#24672

Closed
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-timeout-run29287694175
Closed

fix(ci): raise AVM check-circuit per-tx timeout to 120s#24672
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-timeout-run29287694175

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

What

Raise the per-transaction timeout in avm_check_circuit_cmds from 30s to 120s (yarn-project/end-to-end/bootstrap.sh).

Root cause

The avm-check-circuit job of the AVM Circuit Inputs Collection and Check workflow failed on next (run 29287694175, exit code 124).

ci-avm-check-circuit downloads the cached AVM inputs and runs bb-avm avm_check_circuit on every dumped tx via avm_check_circuit_cmds | parallelize 16. Each command runs under TIMEOUT=30s. In this run 643 of 644 checks passed in 1–4s each, but one heavy transaction tripped the timeout:

FAILED (…): bb-avm avm_check_circuit -v --avm-inputs .../e2e_multiple_blobs/avm-circuit-inputs-tx-0x127eb600….bin (31s) (code: 124)
parallel: This job failed:
run_test_cmd 'eda031bed1f51704:ISOLATE=1:TIMEOUT=30s:NAME=avm_cc_e2e_multiple_blobs_0x127eb600 …'
+ local code=124

The e2e_multiple_blobs tx has a large AVM trace and ran 31s — just over the 30s cap. timeout -v killed it (exit 124), and because parallelize runs parallel --halt now,fail=1, that single timeout failed the entire job.

This is not caused by the head commit (a docs-only release PR, #24664); the check runs on the tree-hashed cached inputs and is unrelated to docs. It's a pre-existing borderline-heavy check running under a too-tight timeout — exactly the risk the code comment already flagged ("transactions could need more CPU and MEM than we allocate by default … they might start timing out"). A previous attempt to relieve this reduced parallelism (#24458, "It was timing out"); with the heaviest tx now sitting right at the boundary it regressed again.

Fix

Give the check generous headroom over the slowest observed tx (31s) so a single slow run under 16-way CPU contention no longer fails the whole job. avm_check_circuit is a correctness gate — wall-time doesn't matter as long as it's bounded, and a 120s cap still catches genuine hangs (normal txs finish in seconds). Updated the stale comment to match.

Note

This is a recurring failure that ClaudeBox has been auto-dispatched on many times (dozens of stale cb/avm-check-circuit-timeout* / cb/avm-cc-timeout* branches exist, none merged — next still carries 30s). Worth actually landing a headroom bump so the auto-dispatch stops firing on it.

Testing

No local red/green repro is feasible: the check runs bb-avm avm_check_circuit on an EC2 instance against the S3-cached e2e-avm-circuit-inputs-<tree-hash>.tar.gz with the freshly built AVM binary — none of which is reproducible in this container. The change is a one-line timeout bump validated against the failing job's own log (the timed-out tx took 31s, well within the new 120s).


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 13, 2026
@AztecBot

Copy link
Copy Markdown
Collaborator Author

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.

@AztecBot AztecBot closed this Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant