Commit 3a68d68
authored
fix(ci3): run dedicated benches on bare metal (m6a.metal) to cut variance (#24298)
## Problem
The dedicated bench box (`m6a.32xlarge`) still produced bimodal
benchmark numbers — e.g. `avm_bulk` wire commitments flipping between
~2.1s and ~3.2s — despite CPU isolation, HT-off, and (separately tried)
NUMA memory binding.
## Root cause (confirmed by a live controlled experiment)
On a fresh `m6a.32xlarge` I ran `avm_bulk` (a) alone and (b) while 3
more `avm_bulk` saturated the other NUMA nodes:
| Phase | isolated | contended (all 64 cores) | Δ |
|---|---|---|---|
| Wire commitments (MSM) | ~2,070 ms | ~2,098 ms | +1.4% |
| Sumcheck (compute) | ~2,770 ms | ~2,873 ms | +4% |
Saturating the whole box barely moved the numbers — so it's **not**
code, **not** contention from our own concurrent benches, and **not**
NUMA locality (memory binding made no difference). Across every observed
run the **compute phases are constant** while **only the
memory-bandwidth-bound MSM phase varies box-to-box**. That's the
signature of **per-physical-instance memory bandwidth differing**:
`m6a.32xlarge` is ~2/3 of a host and shares it with other tenants, so a
memory-heavy co-tenant steals bandwidth, and placement is random per
launch → the bimodal flip.
## Fix
Run the dedicated bench box on **`m6a.metal`** — sole-tenant bare metal,
no co-tenant to contend for memory bandwidth. Unchanged: spot-first with
on-demand fallback (same sole-tenant hardware either way; mid-run
reclaim handled by `bootstrap_ec2`'s `NO_SPOT` retry). Bonus: 96
physical cores vs the 32xl's 64, so more bench parallelism.
Supersedes #24276 (the NUMA-binding attempt, now closed — the experiment
showed binding wasn't the lever).1 file changed
Lines changed: 11 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
189 | 193 | | |
190 | 194 | | |
191 | 195 | | |
192 | 196 | | |
193 | | - | |
| 197 | + | |
194 | 198 | | |
195 | 199 | | |
196 | 200 | | |
| |||
0 commit comments