Skip to content

Fix remaining flaky CodSpeed microbenchmarks#8861

Merged
connortsui20 merged 3 commits into
developfrom
claude/codspeed-flaky-benchmarks-4vapbm
Jul 20, 2026
Merged

Fix remaining flaky CodSpeed microbenchmarks#8861
connortsui20 merged 3 commits into
developfrom
claude/codspeed-flaky-benchmarks-4vapbm

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 20, 2026

Copy link
Copy Markdown
Member

Rationale for this change

Three benchmarks stayed flaky after #8742, flipping between the same two values on PRs that can't affect them. Same root causes and fixes as #8742:

Benchmark Seen flaky on Why Fix
true_count_vortex_buffer[128] ±11.17% on 9 unrelated PRs (#8805, #8811, #8812, #8820, #8843, #8803, …) a 128-bit popcount measures harness overhead and code layout, not the count drop the 128 size
runend compress[(100000, 4)] ±11.9% on #8805, #8750, #8856 allocates in the timed region; glibc malloc differs across runner images mimalloc as global allocator
cast_decimal copy_*[65536] identical flags on #8838 and #8724 same glibc-malloc cause (512 KB alloc per iteration) mimalloc as global allocator

Left alone: compact_sliced[(4096, 90)] (single sighting) and the CUDA walltime benches (hosted-runner walltime noise, a runner config issue).

The allocator swap shifts every benchmark in the two touched binaries once — see the comment below. Needs a one-time CodSpeed acknowledgment, like #8742.

What changes are included in this PR?

One commit per benchmark; bench files only. Ran cargo check + clippy on the three bench targets, smoke-ran the binaries, cargo +nightly fmt.

claude added 3 commits July 20, 2026 15:30
`true_count_vortex_buffer[128]` flipped between the same two values
(580.6ns <-> 522.2ns, +/-11.17%) on at least nine unrelated PRs since
#8742, including java-only (#8811, #8812), Cargo.toml-only (#8820),
website-removal-only (#8805), zstd-only (#8843), and geo-only (#8803)
changes. A 128-bit true_count is a popcount over two u64 words, so the
measurement is fixed dispatch/harness overhead and binary code layout
rather than the count itself. Keep only the sizes where the popcount
loop dominates, mirroring the bitwise_not_vortex_buffer_mut fix from
#8742.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
`copy_nullable[65536]` and `copy_non_nullable[65536]` were on the #8742
watch list and have now been flagged with byte-identical values on two
unrelated open PRs (#8838 union scalars, #8724 decimal add/sub kernels;
-24.3% / -16.59% each). The copy casts allocate their output buffer in
the timed region, so glibc malloc differences across CodSpeed runner
images show up as benchmark changes. Install vendored mimalloc as the
global allocator, as #8742 did for the other alloc-heavy vortex-array
bench files.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
`compress[(100000, 4)]` flipped between the same two values (1.8ms <->
1.6ms, ~+/-11.9%) on three unrelated PRs since #8742: #8805 (deletes a
JS website), #8750, and #8856 (list min/max expressions). runend_encode
allocates its output buffers inside the timed region and the runend
bench binaries still use glibc malloc, whose code differs across
CodSpeed runner images. Install vendored mimalloc as the global
allocator, matching the #8742 fix for the flaky vortex-array benches.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 1902072 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.9%
Engines: DataFusion No clear signal (+5.3%, low confidence) · DuckDB No clear signal (-1.5%, low confidence)
Vortex (geomean): 0.974x ➖
Parquet (geomean): 0.966x ➖
Shifts: Parquet (control) -3.4% · Median polish -1.6%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (0.987x ➖, 0↑ 0↓)
name PR 1902072 (ns) base b7b01d3 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 9615865 9831831 0.98
vortex_q01/datafusion:vortex-file-compressed 6190978 6216077 1.00
datafusion / parquet (0.937x ➖, 0↑ 0↓)
name PR 1902072 (ns) base b7b01d3 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 19834449 21543706 0.92
vortex_q01/datafusion:parquet 4496080 4713514 0.95
duckdb / vortex-file-compressed (0.980x ➖, 0↑ 0↓)
name PR 1902072 (ns) base b7b01d3 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 9973448 9900408 1.01
vortex_q01/duckdb:vortex-file-compressed 5688309 5965326 0.95
duckdb / parquet (0.995x ➖, 0↑ 0↓)
name PR 1902072 (ns) base b7b01d3 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23287826 23497829 0.99
vortex_q01/duckdb:parquet 9394154 9407972 1.00

No file size changes detected.

@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 16.43%

❌ 6 regressed benchmarks
✅ 1662 untouched benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (1000, 16)] 22.6 µs 31.1 µs -27.3%
Simulation decompress[u8, (1000, 4)] 24.5 µs 31.3 µs -21.55%
Simulation decompress[u8, (10000, 4)] 96 µs 111.6 µs -13.93%
Simulation decompress_utf8[(1000, 4)] 37.5 µs 43.5 µs -13.71%
Simulation take_indices[(10000, 256)] 57.3 µs 64.1 µs -10.63%
Simulation take_indices[(1000, 4)] 49.2 µs 54.7 µs -10.03%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/codspeed-flaky-benchmarks-4vapbm (1902072) with develop (b7b01d3)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Copy link
Copy Markdown
Member Author

On the 6 "regressions" in the CodSpeed report: they are the expected one-time step change from the allocator swap, not real regressions.

All 6 are in run_end_compress.rs — the binary that switched to mimalloc. CodSpeed's Simulation mode counts instructions for the whole trace, allocator included, and the decompress/take benchmarks also allocate inside the timed region; their traces now include mimalloc's allocation path (more instructions on these sizes) instead of glibc's. No library code changed, so nothing actually got slower — the values just move to a new baseline once.

#8742 had the same ±10–20% level shifts on benchmarks sharing files with the fixed ones and was resolved by a one-time acknowledgment on the dashboard. After that, every PR compares mimalloc-vs-mimalloc, and these benchmarks measure only Vortex changes — that stability is the point of the swap.


Generated by Claude Code

@connortsui20
connortsui20 enabled auto-merge (squash) July 20, 2026 15:44
@connortsui20
connortsui20 merged commit 5b688a9 into develop Jul 20, 2026
80 of 82 checks passed
@connortsui20
connortsui20 deleted the claude/codspeed-flaky-benchmarks-4vapbm branch July 20, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove remaining codspeed flakes

3 participants