feat(bench): Updated tx inclusion benchmarking#24266
Merged
Merged
Conversation
Phase 1 (A-1221) + Phase 2 (A-1222) of the nightly benchmarking revamp. Schema v4 (additive over v3, all v3 fields retained): - New optional sections provingInfra and saturation (metricSeriesMap: open slug -> timeSeries map), plus run.sweepId/sweepLabel so a night's 1/5/10 TPS points group as one sweep. - Version gate bumped to "4" in bench_output.schema.json and the network_bench_upload check in bootstrap.sh; index.json entry now carries sweepId/sweepLabel. (Dashboard SUPPORTED_RUN_VERSION lives in AztecProtocol/explorations and must be bumped there separately.) - 10tps-readiness-spec.md: the canonical spec — tx-lifecycle stage waterfall mapped to metrics, headline KPIs + thresholds, sweep/run-group notion. Scraper (bench_scrape.ts): - provingInfra: prover-node hint-gen (public_processor.* + prover_node block/checkpoint processing, scoped to the prover-node pod) and proving-queue series broken down by aztec_proving_job_type (size/active/job_duration/ timed-out/resolved). There is no prover_node.execution.duration metric; the re-execution is the public_processor.* path, mapped accordingly. - saturation: per-role ELU/CPU/memory as max (hottest pod) AND avg, never a single hand-picked pod. ELU/mem from nodejs.*, CPU from host-metrics. - Sections scrape independently so one failure cannot drop the others. CPU and ELU may be telemetry-gated in the bench env (empty series, non-fatal); flagged in the spec for live verification per A-1222 acceptance.
…ch point Phase 3 (A-1223), infra + runner. The CI workflow that fans out the 1/5/10 sweep is intentionally not included yet (concurrency model is an open decision). - bench-inclusion-sweep.env: bench-10tps's custom scrape pipeline + the tps-scenario topology trimmed to 150 full nodes (12x4 validators, 10 RPC), local eth-devnet, fake (fixed-delay) proving so proving never bottlenecks inclusion, P2P instrumentation on. Target TPS / sweep id are passed per point. - full-node-resources-spot.yaml: full-node profile that *requires* spot (nodeSelector pool:spot + required gke-spot affinity + toleration), copying the prover-resources-dev.yaml pattern. Validators/RPC/prover stay on-demand. - spartan/bootstrap.sh bench_inclusion_point: one sweep point at TARGET_TPS in its own namespace, all high-value load, scraped with a shared BENCH_SWEEP_ID (schema v4) via the existing bench_scrape path; reuses network_bench_upload. Usable for local runs now: TARGET_TPS=5 BENCH_SWEEP_ID=incl-<date> \ spartan/bootstrap.sh bench_inclusion_point bench-inclusion-sweep
Phase 3 (A-1223) CI wiring for the Set A inclusion sweep.
- nightly-bench-inclusion-sweep.yml: runs the three points SEQUENTIALLY (one
~150-full-node network at a time to bound cluster load), each in its own
namespace (bench-incl-sweep-{1,5,10}tps), all tagged with a shared sweepId
computed once in select-image. Each point chains deploy -> wait -> bench ->
teardown; the next point's deploy waits on the previous point's teardown, so
peak load stays at one network. A failed point drops only itself.
- ci.sh network-inclusion-sweep: threads TARGET_TPS / BENCH_SWEEP_ID /
BENCH_SWEEP_LABEL into the EC2 bootstrap command (mirrors network-bench-10tps).
- bootstrap.sh ci-network-inclusion-sweep: runs one point via
spartan/bootstrap.sh bench_inclusion_point; v4 run JSON (tagged sweepId) is
uploaded to GCS inside that function. SKIP_NETWORK_DEPLOY=1 (deploy/teardown
done by the workflow), mirroring the ci-network-bench-10tps path.
Run #95's proving sweep stalled with the agent fleet parked around ~50 replicas: under fake-realistic delays (PUBLIC_VM ~180s/job) the ~50 agents could not drain a 32-checkpoint epoch's jobs within the proof-submission window, so the proven tip never advanced past epoch 1 and the run ended in a proof-window-expiry reorg. The KEDA bands only requested the full 200 agents once the unassigned queue exceeded 200; at moderate backlog the fleet was capped at 10. Ramp to 75 agents at queue>25 and the full 200 at queue>100 so a fresh epoch's job burst pulls in the fleet it needs. Tuning hypothesis to validate on the next run; node-pool/spot autoscaling may still gate how quickly 200 agents actually schedule.
(cherry picked from commit ef43cb7)
… series
Verified against a live smoke run: the v4 scraper's memory and duration-histogram
series came back empty because the OTel->Prometheus exporter appends unit
suffixes the queries omitted:
- nodejs heap gauge is nodejs_memory_v8_heap_usage_bytes (was ..._usage)
- duration histograms carry a _milliseconds infix, e.g.
aztec_proving_queue_job_duration_milliseconds_bucket,
aztec_public_processor_tx_duration_milliseconds_bucket,
aztec_prover_node_{block,checkpoint}_processing_duration_milliseconds_bucket
With these, memory (all roles), hint-gen, and proving-queue job_duration series
now populate. ELU + CPU were already correct. Full-node saturation remains
empty by design (FULL_NODE_INCLUDE_METRICS restricts full nodes to gossip).
…test fails bench_inclusion_point ran the scrape only if the load test succeeded — a non-zero test exit aborted the function first, discarding the inclusion records and per-role/proving metrics it had already captured. A degraded point (e.g. a higher-TPS point that misses its target, or any failed assertion) is exactly where that data is most useful. Capture the test exit code, always scrape + upload, then re-surface the failure via the return code so job status still reflects it.
Two robustness changes to n_tps.test.ts, surfaced by a smoke run where the inclusion benchmark mined 300/300 txs but still exited code 1: - Replace the strict "successCount === totalHighValueSent" assertion with metrics.recordInclusionOutcome(mined, failed). A degraded point (a TPS target the network can't fully include) should report its mined-vs-failed counts + success_ratio, not fail the run — the ratio is itself the headline result we track. A total submission failure (nothing sent) is still a hard error. - Wrap the afterAll wallet-cleanup loop and uninstallChaosMesh in try/catch so a teardown error (dead PXE, missing chaos-mesh CRDs on a run that didn't install it) is logged, not thrown — it was turning successful runs red. tx_metrics.ts: TxInclusionMetrics gains recordInclusionOutcome, emitting inclusion/mined_count, inclusion/failed_count, inclusion/success_ratio.
…value + low-value fill) Each inclusion-sweep point now drives a fixed 1 TPS of high-value txs (the measured inclusion lane) plus (TARGET_TPS - 1) TPS of low-value background traffic, instead of all-high-value at the target. This matches bench_10tps and measures what we actually care about: whether a properly-paying user's tx is included promptly while the network runs at the target total TPS. The 10 TPS point is now identical to bench-10tps (1 high + 9 low).
…eorgs The high-value success/fail count came from waitForTx(waitForStatus: PROPOSED), which throws/times out if a reorg drops a mined tx back to pending or the pool evicts it — so a tx that WAS included got counted as failed under reorg churn (inflating failed_count on reorg-heavy high-TPS runs). Inclusion is a first-sighting event: the block-watcher already stamps minedAtMs idempotently (never cleared on reorg). Add TxInclusionMetrics.wasMined(txHash) and, in the waitForTx catch path, count the tx as included if it was ever observed in a block — we don't care what happens to it afterwards.
Add ingressTxDurationP50/P99 from aztec.node.receive_tx.duration (histogram, RPC-side tx-ingest cost on the submission path). This is the metric to watch when RPC ingress is the bottleneck — it climbs as the RPC saturates, distinct from the ingress-TPS rate. Flows into the existing timeSeries section (open map, no schema change).
Reuse MEMPOOL_TX_MINED_DELAY but derive the value from the tx's persisted receivedAt at the mined transition in handleMinedBlock, instead of the add/remove timestamp map. The map-based recording fired on any removal (eviction/deletion too), conflating mined delay with evicted delay; it was also in-memory and lost across restarts. handleMinedBlock now passes a per-tx minedDelayMs (now - receivedAt) through onTxsMined, left undefined for txs hydrated from the DB on restart (receivedAt === 0) so unknown receive times don't pollute the histogram. The shared PoolInstrumentation map mechanism is retained for the attestation pool.
The pending-to-mined delay (aztec.mempool.tx_mined_delay) routinely exceeds the 60s ceiling of the shared ms-unit histogram view under load, saturating every percentile at 60000. Add a name-scoped view with buckets from 1s to 10min, placed before the generic ms view so it wins the bucket boundaries (the SDK keeps the first-registered compatible storage; other ms histograms are unaffected). Also fix the now-stale metric description.
…e event-loop delay Three additive metrics in the v4 scrape output: - Per-tx block build decomposition in summary: blockBuildPerTxMsAggregate, blockBuildMarginalMsPerTx (OLS slope), blockBuildFixedOverheadMsPerBlock (OLS intercept), blockBuildAvgTxsPerBlock — computed from per-block records. - Pool-side pending-to-mined delay (aztec_mempool_tx_mined_delay) as both timeSeries (mempoolTxMinedDelayP50/P95/P99) and summary scalars, distinct from the client-observed inclusionLatency*. - Event-loop delay (nodejs_eventloop_delay_mean) added to per-role saturation, surfacing the prover node's multi-second main-thread blockage at 10 TPS.
Validator and RPC pods previously requested only 0.5 CPU with a node-type=network
selector, so the scheduler bin-packed them across the shared cluster's mixed
2-/4-core network pools. Validators landing on 2-core nodes ran their 4 attesters
on too little CPU, fell behind on re-execution, and failed to attest ('Timed out
waiting for block with archive matching checkpoint proposal') — dragging the
committee below quorum and causing checkpoint prunes (reorgs) at 10 TPS.
Add bench resource profiles (request=limit, Guaranteed QoS) so every node gets
dedicated, un-contended cores and lands on an adequately-sized node:
- validator: 3.5 CPU / 12Gi (one pod, 4 attesters, per dedicated 4-core node)
- rpc: 3.5 CPU / 8Gi (dedicated 4-core; drops prod's preferred cores=2 affinity)
- full-node: 3.5 CPU / 8Gi (kept on spot; packs onto 8-/32-core spot nodes)
Point both bench-inclusion-sweep envs at VALIDATOR/RPC/FULL_NODE_RESOURCE_PROFILE=bench.
Prover/broker keep dev-hi-tps (already explicitly sized, never bin-packed).
Adds a dedicated 4-core spot pool (spot_nodes_4core, t2d-standard-4) to the GKE cluster. Benchmark full nodes request 3.5 CPU but there was no 4-core spot pool (spot pools were 2/8/32-core), so they packed several onto 8-core spot boxes. Pin the bench full-node profile to the new pool via a cores=4 nodeSelector so each full node gets a dedicated 4-core spot node, without consuming the on-demand pool that validators / RPC / prover run on. Requires a terraform apply of the gke-cluster/cluster module before the next bench deploy, otherwise the cores=4 + pool=spot full-node pods stay Pending.
…2s slots The inclusion sweep (1/5/10 TPS) is a superset of the single-point 10 TPS nightly (it reuses the bench-10tps load model), so retire nightly-bench-10tps.yml and let nightly-bench-inclusion-sweep.yml be the nightly TPS/inclusion benchmark. Rebase bench-inclusion-sweep.env on the smoke env (same 12x4 validator / 10 RPC / 10 full-node topology on dedicated guaranteed-QoS bench profiles, RPC ingress off, fake proving) but at a representative 72s slot duration and 800 tx per checkpoint. 8-slot epochs keep nightly committee warm-up short. Test duration is already 10 min (bench_inclusion_point default TEST_DURATION_SECONDS=600; the workflow does not override it).
Replace bench-inclusion-sweep.env with the smoke env's topology (12x4 validators, 10 RPC, 10 full nodes on dedicated guaranteed-QoS bench profiles, RPC ingress off, fake proving, 8-slot epochs) but at a representative 72s slot duration and 800 tx per checkpoint. Drops the prior production timing / RPC LoadBalancer / per-block cap that the sweep no longer uses.
PhilWindle
changed the title
The inclusion-sweep workflow repeated the deploy -> wait -> bench -> teardown block once per TPS point (~250 lines of near-duplicate jobs). Extract it into a reusable workflow (bench-inclusion-point.yml, workflow_call) parametrized by tps/namespace/image/ref/sweep_id, and call it 3x from the sweep with needs chaining for the sequential one-network-at-a-time ordering. An `if: !cancelled` guard on the later points preserves point independence (a failed point drops only itself). Sweep workflow drops 357 -> 161 lines; per-point logic defined once.
…l/bench-integration-v5 # Conflicts: # .github/workflows/nightly-bench-inclusion-sweep.yml
…etwork Previously each TPS point deployed and tore down its own network (3x deploy + teardown). Switch to deploying a single network, running the 1/5/10 points sequentially against it (the scraper drains the mempool to zero between points), and tearing it down once. Cheaper, faster, and matches how it's run locally. deploy/wait/teardown move to the top-level workflow (run once); the reusable bench-inclusion-point.yml is now just the bench step (one point, against the existing network, SKIP_NETWORK_DEPLOY=1), called 3x with needs chaining so the points don't overlap. Later points are gated on `wait` success so a failed point still drops only itself.
Capture each pod's aztec-container CPU/memory requests and limits during the infrastructure scrape, exposed as `resources` per node and a deduped `resourceProfiles` per role, so a run records the machine sizing the validators/rpc/full-node/prover pods actually got (request==limit for the guaranteed-QoS bench profiles). Also anchor the role-classification patterns to the namespace-stripped pod name so the L1 infra pods (eth-validator / eth-execution / eth-beacon) are no longer misread as aztec roles — an unanchored /-validator/ matched "<ns>-eth-validator-0" and polluted the validator role.
The node emits aztec.validator.attestation_failed_node_issue_count (the attester-side timeout/missing-data counter that diagnosed the 10 TPS reorgs) alongside the bad-proposal and success counters, but the bench scraper never queried them — only the proposer-side collect-duration gauges reached the JSON. Add three timeSeries (node-issue and bad-proposal failures broken down by error_type, plus a success rate for a failure-ratio denominator) and three summary scalars totalling each outcome over the observed window, so the attestation node-issue count — the signal upstream of committee quorum loss — is recorded per run.
prune_count was incremented only by the proven/epoch prune (handleEpochPrune), so the pending-chain reorgs that drive world-state's "Chain pruned to block N" — proposed blocks dropped because their slot ended without a checkpoint, because they conflicted with an L1 checkpoint, or because their matching proposed checkpoint never arrived before the deadline (pruneOrphanBlocks) — incremented no metric and were observable only in logs. Add a prune_type attribute (unproven / uncheckpointed / l1_conflict / orphan) to prune_count and tag all four emit sites. Scraped per pod, this distinguishes a single straggler node's local re-sync from a network-wide reorg.
…breakdown reorgCount counted `Chain pruned` log events network-wide, which conflates one straggler node repeatedly re-syncing its local view with a genuine network reorg. The 10 TPS sweep showed this: 3 "reorgs" that were all one saturated validator pruning its own view three times while the canonical chain never moved. Re-point reorgCount to count DISTINCT PODS that experienced a pending-chain prune (archiver prune_count, prune_type in uncheckpointed/l1_conflict/orphan), so a straggler reads as 1 and a network reorg as many. Add a prunesByType summary breakdown and a pruneCountByType time series. deepestReorgBlocks stays log-derived (no depth metric). Requires the prune_type-attributed metric (new image); reads 0 on older images, cross-checked by the chainPruned events.
Add a throughputTps series carrying three labeled sub-series (source = gossip | rpc_receive | mined) so the dashboard overlays tx rate at three stages on one chart: gossip propagation (avg per-node accepted tx-topic gossipsub messages — every node accepts each unique tx once, so avg approximates the unique-tx propagation rate; sum would overcount by pod count), RPC ingress (receiveTx on the submission node), and mined (archiver block tx rate, avg across archivers). Query validated against retained Prometheus data.
Add p2pBandwidthBytesPerSec carrying four labeled series (recv/sent × avg/max per pod) from the gossipsub RPC byte counters. Per pod rather than summed so the hottest-pod (max) series surfaces gossip bandwidth that isn't evenly spread — at 10 TPS the hottest pod moved ~4 MB/s each way vs ~2.3 MB/s average. No libp2p-level transport metric is exported, so the gossipsub byte counters are the bandwidth signal. Query validated against retained Prometheus data.
…licate-tf-blocks The spot_nodes_4core node pool definition belongs with the gke-cluster terraform cleanup, not the bench branch. Removed here and applied to stack/chore-remove-duplicate-tf-blocks. The pool is already applied to the live aztec-gke-private cluster, and the bench full-node resource profile that targets it (cores=4 selector) stays on this branch.
NETWORK_BENCH_BUCKET overrides the upload target (default unchanged: gs://aztec-testnet/network_bench) so the upload path can be validated against a test prefix without polluting the production dashboard index. Add a network_bench_upload dispatch entry so a run JSON can be (re)uploaded manually.
Strip internal tracker issue IDs and the 'Set A' project label from code and workflow comments; the repo is public so these convey nothing to external readers. Behaviour unchanged (comment-only edits).
PhilWindle
changed the title
The headline inclusion TPS divided total mined by the whole observed window, which includes the warmup ramp and the post-load drain tail — diluting it well below the achieved rate (e.g. 0.84 at a 1 TPS target that actually kept up). Compute it instead from the block log over [first block with txs, load stop]: trims the startup ramp (start at first real inclusion) and the cooldown tail (end at load stop, not drain end). headlineKpi now reads ~1.0 on healthy runs (1/5/10 TPS: 0.97/0.99/0.99) and drops below 1 only on genuine shortfall. The old whole-window figure is retained as inclusionTpsWindowMean.
Tag each chainPruned event with podCount (distinct pods that pruned to the same block within the dedupe window). reorgCount now counts only genuine chain reorgs (podCount >= 2 — the canonical chain rewound across nodes); single-node local prunes (a straggler that diverged and re-synced) are counted separately as nodeLocalPruneCount and no longer inflate reorgCount or deepestReorgBlocks. This is log-derived, so it is accurate regardless of node image. Verified on the 1/5/10 TPS runs: 0 chain reorgs, with the 5/10 TPS prunes correctly classified as node-local.
Refine reorg detection: tag each chainPruned event with validatorPruneCount (distinct validator pods that pruned to that block) and compare against the validator fleet size (distinct validators that logged l2-block-handled). reorgCount now counts only events where ALL following validators rewound to the same block — a true canonical reorg. A prune reported by a subset of validators (or by non-validator followers) is a local divergence (nodeLocalPruneCount), not a reorg. Verified on 1/5/10 TPS: validatorCount=12, 0 chain reorgs (prunes had 0-1 validators each).
Adds an optional run.benchmarkType enum (ingress-inclusion, simulated-proving, real-proving, block-capacity) so the dashboard can group a day's sweeps by benchmark kind. Bumps the run schema 4->5 and the index schema 1->2. - bench_scrape.ts: --benchmark-type arg / BENCH_BENCHMARK_TYPE env, emitted in run - bootstrap.sh: inclusion point passes ingress-inclusion; benchmarkType copied into each index.json entry; upload gate now expects schema "5"
PhilWindle
commented
Jun 30, 2026
| RPC_RESOURCE_PROFILE="bench" | ||
| RPC_INGRESS_ENABLED=false | ||
|
|
||
| FULL_NODE_REPLICAS=10 |
Collaborator
Author
There was a problem hiding this comment.
Kept low until we know it all works
PhilWindle
commented
Jun 30, 2026
|
|
||
| AZTEC_EPOCH_DURATION=8 | ||
| AZTEC_SLOT_DURATION=72 | ||
| AZTEC_PROOF_SUBMISSION_EPOCHS=100 |
Collaborator
Author
There was a problem hiding this comment.
Effectively never re-org
TestDateProvider.now() tracks the real clock, so minedDelayMs was the 5s advance plus the real elapsed time between addPendingTxs and handleMinedBlock — which could exceed the < 6000ms bound on a slow/loaded CI box. Switch to ManualDateProvider (frozen clock, advances only via advanceTime) so the delay is deterministically exactly 5000ms.
…tstrap The single-shot 10 TPS bench (bench_10tps / bench_10tps_cmds) is superseded by the inclusion sweep (bench_inclusion_point, run at 1/5/10 TPS). Removes the dead functions and the case-dispatch entry; the shared scraper at scripts/bench_10tps/bench_scrape.ts is still used by the sweep and is untouched.
PhilWindle
marked this pull request as ready for review
June 30, 2026 11:09
PhilWindle
changed the title
spalladino
reviewed
Jul 1, 2026
spalladino
left a comment
spalladino
left a comment
Contributor
There was a problem hiding this comment.
Looks good! Some scripts I skimmed through, and I flagged one possible incompatibility on schema version that Claude spotted, plus a prune category missing.
| // | ||
| // Scrape a completed bench-10tps run into a schema-conformant JSON payload. | ||
| // Contract: bench_output.schema.json (v3). Invoked by the bench_10tps function | ||
| // Contract: bench_output.schema.json (v4). Invoked by the bench_10tps function |
Contributor
There was a problem hiding this comment.
Heads up bench_output.schema.json shows v5, not v4.
Collaborator
Author
There was a problem hiding this comment.
Great catch, thanks for this.
Comment on lines
2246
to
2249
| if (payload.schemaVersion !== "4") { | ||
| throw new Error( | ||
| `schemaVersion must be "3", got ${String(payload.schemaVersion)}`, | ||
| `schemaVersion must be "4", got ${String(payload.schemaVersion)}`, | ||
| ); |
Contributor
There was a problem hiding this comment.
Claude found this incompatibility: schema version if 5 (as created a few lines below), but here we request 4.
Contributor
There was a problem hiding this comment.
This is missing a recordPrune in L785 (next to a call to removeCheckpointsAfter), which triggers when the latest checkpointed tip in the archiver does not match the one on L1, which could be caused by either an L1 reorg or a missed proof.
And I'm happy that I found it, and Claude missed it. How sad is that?
Comment on lines
+683
to
+691
| // Once a tx has been observed in a block we count it as included, even if | ||
| // waitForTx then threw because a reorg dropped it back to pending or the | ||
| // pool evicted it. Inclusion is a first-sighting event here; we don't care | ||
| // what happens to the tx afterwards. | ||
| if (metrics.wasMined(txHash)) { | ||
| logger.info(`${txName} was mined (ignoring post-inclusion reorg/drop)`, { txName, txHash }); | ||
| results.push({ success: true, txHash }); | ||
| return; | ||
| } |
Contributor
There was a problem hiding this comment.
IIUC the only thing that can set the wasMined for a tx is recordMinedTx, which is never reached if waitForTx throws.
Collaborator
Author
There was a problem hiding this comment.
There are 2 places that update the underlying store for mined tx data. recordMinedTx and an observer that watches blocks for mined transactions.
…atch prune - bench_scrape.ts asserted schemaVersion "4" while emitting "5", so the scraper threw on its own output; align the check and refresh the stale header comment (bench_10tps -> bench_inclusion_point). - Archiver: the checkpointed-tip-vs-L1 mismatch rewind (removeCheckpointsAfter in l1_synchronizer) emitted no prune_count. Add a new 'l1_mismatch' prune_type for it (distinct from the proposed-block causes) and surface it in prunesByType + schema.
alexghr
approved these changes
Jul 2, 2026
Comment on lines
+87
to
+89
| this.pruneCount = createUpDownCounterWithDefault(meter, Metrics.ARCHIVER_PRUNE_COUNT, { | ||
| [Attributes.PRUNE_TYPE]: ['unproven', 'uncheckpointed', 'l1_conflict', 'orphan'], | ||
| }); |
Contributor
There was a problem hiding this comment.
We'll need to update our alerts so they sum over all prune types, but this is 🔥
spalladino
approved these changes
Jul 2, 2026
l1_mismatch was added to recordPrune and emitted from the l1 synchronizer, but omitted from the prune_count default-values list, so the prune_count{prune_type="l1_mismatch"} series only appeared after the first such prune instead of from startup like the other causes. Add it to the zero-seed list and to the PRUNE_TYPE attribute doc comment.
PhilWindle
enabled auto-merge (squash)
July 2, 2026 19:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lands the inclusion sweep on the v5 line — a nightly benchmark that deploys a live network and drives sustained 1 / 5 / 10 TPS mixed-priority load, measuring transaction inclusion under load. Supersedes the stacked PRs #24260 and #24265.
Changes
Benchmark harness & output
saturation(ELU / CPU / mem / event-loop delay), per-pod node resource sizing, RPC ingress duration, per-tx block-build decomposition, a pool-side pending→mined delay, and validator attestation-failure metrics (the signal upstream of quorum loss).run.benchmarkTypelets the dashboard group a day's sweeps by kind.inclusionTpsMeanis a steady-state rate (warmup ramp + drain tail trimmed), soheadlineKpi ≈ 1.0on healthy runs.reorgCountcounts genuine chain reorgs only — events where the whole validator set pruned; a single/subset divergence is reported separately asnodeLocalPruneCount. Addsprune_countprune_typeattribution for cause breakdown.throughputTps(gossip / rpc / mined funnel) andp2pBandwidthBytesPerSecseries.Node-side metrics (aztec image)
now − receivedAtat the mined transition, no eviction conflation), widened mined-delay histogram buckets, andprune_countattributed byprune_typeso pending-chain reorgs are countable from metrics instead of logs.Deployment / CI
bench-inclusion-sweep.env(12×4 validators, 10 RPC, 10 full nodes, 72s slots, fake proving) + guaranteed-QoS resource profiles so every node gets reserved cores on an adequately-sized instance.nightly-bench-inclusion-sweep.yml: deploy once, run 1/5/10 sequentially (own namespace each, shared sweepId), scrape-on-failure, GCS upload. Retires the single-pointnightly-bench-10tpsand the obsoletebench_10tpsscenario.Validation (smoke network, all fixes applied)
Under-provisioned validators (bin-packed onto 2-core nodes) were the original reorg cause: attesters couldn't re-execute proposed blocks fast enough to hold quorum. The dedicated-core profiles fix it at the root — at 10 TPS, 13–15 reorgs → 0 and ~57% → ~93% checkpoint publish. The few single-node divergences seen under load self-healed with 0 tx loss and now count as
nodeLocalPruneCount, not chain reorgs.Follow-ups (not in this PR)
AztecProtocol/explorations): renders the v5 schema (companion PR Engineer can add new leaves to any tree and obtain the new root and hash path informtation #22).