Fix flaky CodSpeed microbenchmarks (#8742)

connortsui20 · claude · web-flow · commit 5abaf9823dee · 2026-07-14T01:24:25.000+01:00
A small pool of microbenchmarks flips ±10-35% between two fixed values on unrelated PRs (including docs-only and lockfile-only changes), spamming every CodSpeed report. This PR fixes them (and removes only 1 benchmark). No `#[cfg(codspeed)]` gating; CI and local runs stay identical. One commit per benchmark. ## Changes - **mimalloc as global allocator** in the 5 flaky `vortex-array` bench files: `chunk_array_builder`, `dict_compress`, `varbinview_compact`, `compare`, `binary_ops` - **`bitwise_not_vortex_buffer_mut`**: drop the 128/1024/2048 sizes (measured only harness overhead) - **`slice_empty_vortex`**: rewrite as a 1024-iteration tight loop, renamed `slice_empty_tight_loop_vortex` - **`rebuild_naive` (vortex-zstd)**: the one benchmark removed instead of fixed — its cost is dominated by zstd-internal copies (glibc `ifunc`-resolved `memcpy`) that no bench-level change can stabilize, its fixture is degenerate (a 4-string dictionary with all-zero offsets), and ListView rebuild is already benchmarked across element types and list shapes in `vortex-array/benches/listview_rebuild.rs`. The crate's now-unused `divan` dev-dependency goes with it. <details> <summary>Which benchmarks were flaky, and the evidence</summary> Identified by reading the CodSpeed comments on the ~47 PRs merged since June 25 (post-#8490). The tell: the same benchmark flipping between the same two values, in both directions, on PRs that can't have affected it — including deny.toml-only (#8712, #8716), uv.lock-only (#8732), docs-only (#8737, #8728, #8685), and CI-YAML-only (#8660, #8683) changes. | Benchmark | Evidence | |---|---| | `bitwise_not_vortex_buffer_mut[128/1024/2048]` | ~half of all PRs — worst offender | | `chunked_varbinview_*` ×4 | ~20 PRs, both directions | | `chunked_bool_canonical_into[(1000,10)]` | ~2× flips (16µs ↔ 35µs) on 4 PRs | | `encode_varbin`, `encode_varbinview` | ~12 PRs; `encode_varbinview[(10000,4)]` also flipped on an earlier revision of this PR | | `compact`, `compact_sliced` (90%-utilization args) | ~8 PRs | | `compare_int_constant` | ±11.1% verbatim on ≥9 PRs | | `eq_i64_constant` | same ±11% signature incl. docs-only PR | | `slice_empty_vortex` | -14.66% verbatim on ~13 PRs | | `rebuild_naive` (vortex-zstd) | ~10 PRs, both directions | Watch list (left alone, below the ≥3-independent-sightings bar): `copy_nullable`/`copy_non_nullable[65536]` in `cast_decimal.rs`, `true_count_vortex_buffer[128]`. </details> <details> <summary>Root causes and why each fix matches</summary> - **Allocation in the timed region** → glibc malloc's code differs across CodSpeed runner images, so alloc-heavy benchmarks trace differently for byte-identical Vortex code. Vendored mimalloc removes glibc malloc from the trace. Empirical support: the only three bench files already using mimalloc (`single_encoding_throughput`, `common_encoding_tree_throughput`, `row_encode`) are the most alloc-heavy suites in the repo and were never flagged once in the 47-PR window. - **Sub-microsecond work** → the measurement is fixed harness overhead plus binary code layout, which shifts with any unrelated change. Fix by making the operation dominate: the in-place NOT (no alloc, no copy) keeps only sizes where the loop dominates; the empty slice runs 1024× per iteration, mirroring the neighboring `slice_tight_loop_vortex`. - **Environment-bound and low-signal** → `rebuild_naive`, per the justification above: unfixable at the bench level and redundant with the vortex-array ListView rebuild suite, so removed. </details> <details> <summary>Validation: A/A reruns and a stacked canary PR</summary> - **Expected one-time step changes on this PR**: swapping the allocator changes the trace of every benchmark in the touched files, so this PR's report shows a few ±10-20% level shifts (including on never-flaky `encode_primitives`, which just shares a file). These need a one-time acknowledgment on the CodSpeed dashboard; after merge every PR compares mimalloc-vs-mimalloc. - **A/A reruns** (same bench binaries measured three times, on separate runners and commits): every value reproduced exactly — 211.5µs, 137.1µs, 14.6µs, 26.3µs — with zero new flags across 1656 benchmarks. - **Canary #8743** (a #8681-style cold-string change stacked on this branch — the class of change that used to collect five false flags): **Performance Gate Passed**, `✅ 1660 untouched`, zero changes reported. </details> No public API changes; benchmark-only. https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/encodings/zstd/Cargo.toml b/encodings/zstd/Cargo.toml
@@ -34,10 +34,5 @@ vortex-session = { workspace = true }
 zstd = { workspace = true }
 
 [dev-dependencies]
-divan = { workspace = true }
 rstest = { workspace = true }
 vortex-array = { workspace = true, features = ["_test-harness"] }
-
-[[bench]]
-name = "listview_rebuild"
-harness = false
diff --git a/encodings/zstd/benches/listview_rebuild.rs b/encodings/zstd/benches/listview_rebuild.rs
diff --git a/vortex-array/Cargo.toml b/vortex-array/Cargo.toml
@@ -85,6 +85,7 @@ serde = ["dep:serde", "vortex-buffer/serde", "vortex-mask/serde"]
 divan = { workspace = true }
 futures = { workspace = true, features = ["executor"] }
 insta = { workspace = true }
+mimalloc = { workspace = true }
 rand_distr = { workspace = true }
 rstest = { workspace = true }
 serde_json = { workspace = true }
diff --git a/vortex-array/benches/binary_ops.rs b/vortex-array/benches/binary_ops.rs
@@ -11,6 +11,7 @@ use std::sync::LazyLock;
 
 use divan::Bencher;
 use divan::counter::ItemsCount;
+use mimalloc::MiMalloc;
 use vortex_array::ArrayRef;
 use vortex_array::Executable;
 use vortex_array::IntoArray;
@@ -23,6 +24,9 @@ use vortex_array::builtins::ArrayBuiltins;
 use vortex_array::scalar_fn::fns::operators::Operator;
 use vortex_session::VortexSession;
 
+#[global_allocator]
+static GLOBAL: MiMalloc = MiMalloc;
+
 fn main() {
     LazyLock::force(&SESSION);
     divan::main();
diff --git a/vortex-array/benches/chunk_array_builder.rs b/vortex-array/benches/chunk_array_builder.rs
@@ -4,6 +4,7 @@
 use std::sync::LazyLock;
 
 use divan::Bencher;
+use mimalloc::MiMalloc;
 use rand::RngExt;
 use rand::SeedableRng;
 use rand::prelude::StdRng;
@@ -22,6 +23,9 @@ use vortex_array::dtype::DType;
 use vortex_error::VortexExpect;
 use vortex_session::VortexSession;
 
+#[global_allocator]
+static GLOBAL: MiMalloc = MiMalloc;
+
 fn main() {
     LazyLock::force(&SESSION);
     divan::main();
diff --git a/vortex-array/benches/compare.rs b/vortex-array/benches/compare.rs
@@ -4,6 +4,7 @@
 #![expect(clippy::unwrap_used)]
 
 use divan::Bencher;
+use mimalloc::MiMalloc;
 use rand::RngExt;
 use rand::SeedableRng;
 use rand::distr::Uniform;
@@ -25,6 +26,9 @@ use vortex_array::scalar_fn::fns::operators::Operator;
 use vortex_array::validity::Validity;
 use vortex_buffer::Buffer;
 
+#[global_allocator]
+static GLOBAL: MiMalloc = MiMalloc;
+
 fn main() {
     divan::main();
 }
diff --git a/vortex-array/benches/dict_compress.rs b/vortex-array/benches/dict_compress.rs
@@ -6,6 +6,7 @@
 use std::sync::LazyLock;
 
 use divan::Bencher;
+use mimalloc::MiMalloc;
 use rand::distr::Distribution;
 use rand::distr::StandardUniform;
 use vortex_array::Canonical;
@@ -20,6 +21,9 @@ use vortex_array::builders::dict::dict_encode;
 use vortex_array::dtype::NativePType;
 use vortex_session::VortexSession;
 
+#[global_allocator]
+static GLOBAL: MiMalloc = MiMalloc;
+
 fn main() {
     LazyLock::force(&SESSION);
     divan::main();
diff --git a/vortex-array/benches/varbinview_compact.rs b/vortex-array/benches/varbinview_compact.rs
@@ -4,6 +4,7 @@
 use std::sync::LazyLock;
 
 use divan::Bencher;
+use mimalloc::MiMalloc;
 use rand::RngExt;
 use rand::SeedableRng;
 use rand::rngs::StdRng;
@@ -19,6 +20,9 @@ use vortex_buffer::Buffer;
 use vortex_error::VortexExpect;
 use vortex_session::VortexSession;
 
+#[global_allocator]
+static GLOBAL: MiMalloc = MiMalloc;
+
 fn main() {
     divan::main();
 }
diff --git a/vortex-buffer/benches/vortex_bitbuffer.rs b/vortex-buffer/benches/vortex_bitbuffer.rs
@@ -246,7 +246,12 @@ fn bitwise_not_vortex_buffer(bencher: Bencher, length: usize) {
         .bench_values(|buffer| !&buffer);
 }
 
-#[divan::bench(args = INPUT_SIZE)]
+/// The in-place NOT on an owned `BitBufferMut` performs no allocation and no copy, so below a
+/// few thousand bits the measurement is fixed harness overhead and binary code layout rather
+/// than the loop itself. Only the sizes where the loop dominates are worth measuring.
+const NOT_MUT_INPUT_SIZE: &[usize] = &[16_384, 65_536];
+
+#[divan::bench(args = NOT_MUT_INPUT_SIZE)]
 fn bitwise_not_vortex_buffer_mut(bencher: Bencher, length: usize) {
     bencher
         .with_inputs(|| BitBufferMut::from_iter((0..length).map(|i| i % 2 == 0)))
diff --git a/vortex-buffer/benches/vortex_buffer.rs b/vortex-buffer/benches/vortex_buffer.rs
@@ -170,10 +170,17 @@ fn slice_tight_loop_arrow(bencher: Bencher, len: usize) {
     });
 }
 
+/// Loops like `slice_tight_loop_vortex` above: a single empty slice is a few nanoseconds, far
+/// below the measurement noise floor, so the loop amortizes fixed overhead until the slice
+/// itself dominates.
 #[divan::bench]
-fn slice_empty_vortex(bencher: Bencher) {
+fn slice_empty_tight_loop_vortex(bencher: Bencher) {
     let buf = Buffer::<i32>::from_iter((0..1024).map(|i| i % i32::MAX));
-    bencher.bench(|| divan::black_box(buf.slice(8..8)));
+    bencher.bench(|| {
+        for _ in 0..1024 {
+            divan::black_box(buf.slice(8..8));
+        }
+    });
 }
 
 #[divan::bench(args = INPUT_SIZE)]
