Skip to content

Commit ef88e18

Browse files
committed
fix(ci3): run dedicated benches on bare metal to cut variance
The dedicated bench box used m6a.32xlarge, which is only ~2/3 of a physical host and shares it with other tenants. A memory-heavy co-tenant steals memory bandwidth, which showed up as bimodal variance in the bandwidth-bound proving phases (e.g. avm_bulk wire commitments flipping ~2.1s vs ~3.2s) while compute phases (sumcheck, trace gen) stayed flat. A controlled experiment confirmed it's not contention from our own concurrent benches, not NUMA locality, and not code. Switch to m6a.metal: sole-tenant bare metal, no co-tenant to contend for memory bandwidth. Still spot-first with on-demand fallback (same sole-tenant hardware either way), and 96 physical cores gives more bench parallelism than the 32xl.
1 parent 1bbe4a6 commit ef88e18

1 file changed

Lines changed: 11 additions & 7 deletions

File tree

ci.sh

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -180,17 +180,21 @@ case "$cmd" in
180180
;;
181181
bench)
182182
# Launched by the build instance on uploadable runs to produce stable benchmark
183-
# numbers on a dedicated instance of a FIXED type. AWS_INSTANCE pins the exact type
184-
# (bypasses spot pool diversification) — that's what keeps numbers comparable. Spot
185-
# vs on-demand is the same hardware, so we try spot first and fall back to on-demand
186-
# (the default fleet behaviour); a mid-run spot reclaim is handled by bootstrap_ec2's
187-
# internal on-demand retry. CI_DASHBOARD and PARENT_LOG_ID are inherited from the
188-
# launching run so it nests as a sibling job.
183+
# numbers on a dedicated, FIXED-type instance. AWS_INSTANCE pins the exact type
184+
# (bypasses spot pool diversification) — that's what keeps numbers comparable. We use
185+
# bare metal (m6a.metal): a sub-host instance (e.g. m6a.32xlarge) shares its physical
186+
# host with other tenants, and a memory-heavy neighbour steals memory bandwidth, which
187+
# showed up as bimodal variance in the bandwidth-bound proving phases (MSM commitments)
188+
# while compute phases stayed flat. Metal is sole-tenant, so there's no neighbour.
189+
# Spot vs on-demand is the same hardware, so we try spot first and fall back to
190+
# on-demand (the default fleet behaviour); a mid-run spot reclaim is handled by
191+
# bootstrap_ec2's internal on-demand retry. CI_DASHBOARD and PARENT_LOG_ID are
192+
# inherited from the launching run so it nests as a sibling job.
189193
# Timestamp the instance-request output. pipefail (in a subshell, since ci.sh doesn't
190194
# set it globally) keeps bootstrap_ec2's exit code through add_timestamps — the
191195
# launching build instance waits on this fatally.
192196
( set -o pipefail
193-
AWS_INSTANCE=m6a.32xlarge JOB_ID=x-bench INSTANCE_POSTFIX=x-bench \
197+
AWS_INSTANCE=m6a.metal JOB_ID=x-bench INSTANCE_POSTFIX=x-bench \
194198
bootstrap_ec2 "./bootstrap.sh ci-bench" 2>&1 | add_timestamps )
195199
;;
196200
socket-fix)

0 commit comments

Comments
 (0)