Skip to content

Commit 3a68d68

Browse files
authored
fix(ci3): run dedicated benches on bare metal (m6a.metal) to cut variance (#24298)
## Problem The dedicated bench box (`m6a.32xlarge`) still produced bimodal benchmark numbers — e.g. `avm_bulk` wire commitments flipping between ~2.1s and ~3.2s — despite CPU isolation, HT-off, and (separately tried) NUMA memory binding. ## Root cause (confirmed by a live controlled experiment) On a fresh `m6a.32xlarge` I ran `avm_bulk` (a) alone and (b) while 3 more `avm_bulk` saturated the other NUMA nodes: | Phase | isolated | contended (all 64 cores) | Δ | |---|---|---|---| | Wire commitments (MSM) | ~2,070 ms | ~2,098 ms | +1.4% | | Sumcheck (compute) | ~2,770 ms | ~2,873 ms | +4% | Saturating the whole box barely moved the numbers — so it's **not** code, **not** contention from our own concurrent benches, and **not** NUMA locality (memory binding made no difference). Across every observed run the **compute phases are constant** while **only the memory-bandwidth-bound MSM phase varies box-to-box**. That's the signature of **per-physical-instance memory bandwidth differing**: `m6a.32xlarge` is ~2/3 of a host and shares it with other tenants, so a memory-heavy co-tenant steals bandwidth, and placement is random per launch → the bimodal flip. ## Fix Run the dedicated bench box on **`m6a.metal`** — sole-tenant bare metal, no co-tenant to contend for memory bandwidth. Unchanged: spot-first with on-demand fallback (same sole-tenant hardware either way; mid-run reclaim handled by `bootstrap_ec2`'s `NO_SPOT` retry). Bonus: 96 physical cores vs the 32xl's 64, so more bench parallelism. Supersedes #24276 (the NUMA-binding attempt, now closed — the experiment showed binding wasn't the lever).
2 parents f66377a + ef88e18 commit 3a68d68

1 file changed

Lines changed: 11 additions & 7 deletions

File tree

ci.sh

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -180,17 +180,21 @@ case "$cmd" in
180180
;;
181181
bench)
182182
# Launched by the build instance on uploadable runs to produce stable benchmark
183-
# numbers on a dedicated instance of a FIXED type. AWS_INSTANCE pins the exact type
184-
# (bypasses spot pool diversification) — that's what keeps numbers comparable. Spot
185-
# vs on-demand is the same hardware, so we try spot first and fall back to on-demand
186-
# (the default fleet behaviour); a mid-run spot reclaim is handled by bootstrap_ec2's
187-
# internal on-demand retry. CI_DASHBOARD and PARENT_LOG_ID are inherited from the
188-
# launching run so it nests as a sibling job.
183+
# numbers on a dedicated, FIXED-type instance. AWS_INSTANCE pins the exact type
184+
# (bypasses spot pool diversification) — that's what keeps numbers comparable. We use
185+
# bare metal (m6a.metal): a sub-host instance (e.g. m6a.32xlarge) shares its physical
186+
# host with other tenants, and a memory-heavy neighbour steals memory bandwidth, which
187+
# showed up as bimodal variance in the bandwidth-bound proving phases (MSM commitments)
188+
# while compute phases stayed flat. Metal is sole-tenant, so there's no neighbour.
189+
# Spot vs on-demand is the same hardware, so we try spot first and fall back to
190+
# on-demand (the default fleet behaviour); a mid-run spot reclaim is handled by
191+
# bootstrap_ec2's internal on-demand retry. CI_DASHBOARD and PARENT_LOG_ID are
192+
# inherited from the launching run so it nests as a sibling job.
189193
# Timestamp the instance-request output. pipefail (in a subshell, since ci.sh doesn't
190194
# set it globally) keeps bootstrap_ec2's exit code through add_timestamps — the
191195
# launching build instance waits on this fatally.
192196
( set -o pipefail
193-
AWS_INSTANCE=m6a.32xlarge JOB_ID=x-bench INSTANCE_POSTFIX=x-bench \
197+
AWS_INSTANCE=m6a.metal JOB_ID=x-bench INSTANCE_POSTFIX=x-bench \
194198
bootstrap_ec2 "./bootstrap.sh ci-bench" 2>&1 | add_timestamps )
195199
;;
196200
socket-fix)

0 commit comments

Comments
 (0)