Skip to content

ci: debug cycle-count old branch hang#944

Draft
fakedev9999 wants to merge 12 commits into
mainfrom
fakedev9999/debug-cycle-count-old-hang
Draft

ci: debug cycle-count old branch hang#944
fakedev9999 wants to merge 12 commits into
mainfrom
fakedev9999/debug-cycle-count-old-hang

Conversation

@fakedev9999

Copy link
Copy Markdown
Member

What

  • Add phase prints to test_cycle_count_diff to identify where the old-branch baseline hangs.
  • Add 30 minute timeouts to the default and EigenDA cycle-count test steps so the debug run does not burn the 6 hour default.

Why

  • PR chore: release v4.6.0 #940 reruns pass the new-branch measurement but hang in Run Test On Old Branch until the GitHub Actions 6 hour limit.

Validation

  • cargo fmt --check
  • git diff --check

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Performance Comparison (ELF: eigenda-range-elf-embedded)

Range 8147433~8147438

Metric Base Branch Current PR Diff (%)
Total Instructions 579112691 579059855 -0.01%
Oracle Verify Cycles 74151945 74151945 0.00%
Derivation Cycles 363310656 363310656 0.00%
Block Execution Cycles 6821335 6821335 0.00%
Blob Verification Cycles 624 624 0.00%
Total SP1 Gas 842666333 842591776 -0.01%
Cycles per Block 115822538 115811971 -0.01%
Cycles per Transaction 115822538 115811971 -0.01%
BN Pair Cycles 0 0 0.00%
BN Add Cycles 0 0 0.00%
BN Mul Cycles 0 0 0.00%
KZG Eval Cycles 0 0 0.00%
EC Recover Cycles 0 0 0.00%
P256 Verify Cycles 0 0 0.00%

@fakedev9999

Copy link
Copy Markdown
Member Author

RCA update (2026-07-17 UTC)

Observed

  • Release PR chore: release v4.6.0 #940 attempt 2: new branch passed; old branch used the same range and remained inside host.run until the 30-minute step timeout.
  • Diagnostic run 29555292686 attempt 2, range 20662845..20662850: new host.run completed in 90.4s (04:55:44-04:57:14); old entered host.run at 05:02:19 and was still there when cancelled at 05:07:15.
  • Five valid debug_executePayload calls against the same L2 endpoint completed in 6.5-6.8s while idle. During the old host.run, a valid concurrent call first took 14.25s/HTTP 200, then another returned HTTP 504 after 15.83s with body error code: 504. A post-cancel call still returned 504 after 15.86s. Lightweight RPC calls continued to work.

Mechanism

  • Kona v1.6.0 (optimism#21269) retains L2PayloadWitness and retries any debug_executePayload failure indefinitely in get_preimage, with no backoff or attempt/deadline cap. The Alloy HTTP provider also has no request timeout.
  • op-reth serves execution witnesses through a 3-permit semaphore and waits on the blocking payload-witness task without a server-side timeout. A slow/stuck worker or queue delay can cross the Conduit gateway timeout; Kona then immediately retries the expensive call, which can sustain saturation.

Conclusion
This is not an ELF/cycle-execution regression and increasing the GitHub Actions timeout will not fix it. The failing phase is the Conduit/op-reth execution-witness RPC path.

Conduit ask
Please inspect rpc-succinct-testnet-1.t.conduit.xyz for debug_executePayload during 2026-07-17 05:02:19-05:08:34 UTC, especially semaphore wait time, active witness workers, payload-witness duration, gateway 504s, and whether work continues after the client/gateway times out. The CI range was 20662845..20662850; candidate parent hashes are 0x6351f3e9b05ced974d29d80e9a712dc3f081bd510f8a55963b0514a1c0ed493d, 0xbdc04dc4cfd8674be3c02685d373898fb78ae66ce501ef117b758926f1ee0fdc, 0xba13ef9361bc85d41c1540673e1e93ab25f243831936a8949fea2a36f2fc32ac, 0x265fe96a77c542b4f63ae4e6b0ae1df808958f45c685d1b637520164d32ae79a, and 0xad40bd4b68fc3c18b06499a7538b4aa3c49557f82a9e7f9cbf3913c7b7aee28e.

Immediate recovery is likely a restart/worker drain plus a gateway timeout comfortably above healthy witness latency. The durable fix needs witness queue/runtime metrics, bounded server execution/cancellation, and enough dedicated capacity so a gateway timeout does not leave expensive work occupying all permits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant