Skip to content

Commit fa5e974

Browse files
evertlammertsevertlammerts
authored
Add benchmark suite (#526)
2 parents d7e138f + 79b5e6d commit fa5e974

23 files changed

Lines changed: 2938 additions & 1 deletion

‎.github/workflows/codspeed.yml‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Instruction-count (Callgrind) perf-regression gate against a COMMITTED baseline. No CodSpeed account/token/runner:
2+
# compare_baseline.py parses raw callgrind dumps and diffs each benchmark against benchmarks/baseline.json. Counts
3+
# are near-deterministic with PYTHONHASHSEED pinned (~0.1% noise), so the 5% gate threshold sits far above it.
4+
# Details + rationale: benchmarks/README.md and benchmarks/PLAN.md.
5+
#
6+
# Triggers: nightly schedule + manual workflow_dispatch (no pull_request/push). A dispatch on a feature branch
7+
# compares that branch's counts vs the baseline.json committed on it, answering "did my branch regress vs main".
8+
#
9+
# Modes (workflow_dispatch input `regen`):
10+
# regen=false (default) -> COMPARE + report. Report-only for now (never fails); flip to --enforce once trusted.
11+
# regen=true -> write a fresh baseline.json + upload as an artifact to commit deliberately. Bump
12+
# requirements-bench.txt FIRST (separate commit) if the pins should change.
13+
#
14+
# The concurrency module is excluded from the sweep (Callgrind serializes threads, so its signal is meaningless).
15+
# Memory mode (a second sweep for produce peak-RSS) is deferred (see PLAN.md).
16+
17+
name: Benchmarks
18+
19+
on:
20+
schedule:
21+
- cron: "0 3 * * *" # nightly at 03:00 UTC
22+
workflow_dispatch:
23+
inputs:
24+
regen:
25+
description: "Regenerate benchmarks/baseline.json (upload as artifact) instead of comparing"
26+
type: boolean
27+
default: false
28+
29+
concurrency:
30+
group: codspeed-${{ github.ref }}
31+
cancel-in-progress: true
32+
33+
jobs:
34+
benchmarks:
35+
runs-on: ubuntu-latest
36+
timeout-minutes: 90 # ~25 min sweep at BENCH_SCALE=10 (12-core Linux) + ~10 min cold build; margin for CI
37+
permissions:
38+
contents: read
39+
env:
40+
PYTHONHASHSEED: "0" # stable instruction counts for dict/struct paths
41+
CODSPEED_ENV: "1" # activates pytest-codspeed's instrument hooks
42+
# shrink the O(rows) benches so the sweep fits under timeout-minutes. Local runs leave this unset -> full N.
43+
# Recorded in baseline.json meta.bench_scale; a baseline only compares to a run at the SAME scale.
44+
BENCH_SCALE: "10"
45+
steps:
46+
- uses: actions/checkout@v4
47+
with:
48+
submodules: recursive # the DuckDB engine submodule is needed to build
49+
fetch-depth: 0 # setuptools_scm needs history for version detection
50+
51+
- name: Resolve DuckDB submodule SHA
52+
id: duckdb_sha
53+
# used for the sccache key AND passed to compare_baseline.py for the engine-bump guard
54+
run: echo "sha=$(git rev-parse HEAD:external/duckdb)" >> "$GITHUB_OUTPUT"
55+
56+
- name: Install uv
57+
uses: astral-sh/setup-uv@v5
58+
with:
59+
python-version: "3.13"
60+
61+
- name: Install valgrind
62+
run: sudo apt-get update && sudo apt-get install -y valgrind
63+
64+
- name: Cache sccache
65+
uses: actions/cache@v4
66+
with:
67+
path: ~/.cache/sccache
68+
key: sccache-codspeed-${{ steps.duckdb_sha.outputs.sha }}
69+
restore-keys: sccache-codspeed-
70+
71+
- name: Install sccache
72+
run: |
73+
curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.8.2/sccache-v0.8.2-x86_64-unknown-linux-musl.tar.gz \
74+
| tar -xz --strip-components=1 -C /usr/local/bin sccache-v0.8.2-x86_64-unknown-linux-musl/sccache
75+
76+
- name: Build the extension (release) + pinned benchmark deps
77+
env:
78+
CMAKE_C_COMPILER_LAUNCHER: sccache
79+
CMAKE_CXX_COMPILER_LAUNCHER: sccache
80+
run: |
81+
# step 1: build deps only (needed for --no-build-isolation), no project
82+
uv sync --only-group build --no-install-project -p 3.13
83+
# step 2: the frozen bench pins (exact ==), so the only cross-run delta is the binding. MUST precede the
84+
# build: numpy>=2.0 is a [build-system].requires (numpy C API headers), which --no-build-isolation does
85+
# not auto-install and which is not in the `build` group, so CMake's find_package(... NumPy) fails first.
86+
uv pip install -r benchmarks/requirements-bench.txt
87+
# step 3: build+install the project (release), no default `dev` group (torch/tensorflow/pyspark). uv pip
88+
# install is additive; uv sync here would prune numpy back out before the build and re-break the config.
89+
uv pip install --no-build-isolation --no-deps --reinstall -C cmake.build-type=Release .
90+
91+
- name: Collect gate node-ids
92+
# the gate/informational marker split; regen uses it to classify each benchmark
93+
run: uv run --no-sync pytest benchmarks/ -m gate --collect-only -q -o addopts= -p no:cacheprovider \
94+
| grep '::' > gate_list.txt || true
95+
96+
- name: Run benchmarks under Callgrind (per-benchmark instruction counts)
97+
# ONE sweep over gate+informational EXCEPT the concurrency module (thread-serialized, expensive). Each
98+
# benchmark emits a callgrind dump keyed by its uri.
99+
run: |
100+
mkdir -p profiles
101+
CODSPEED_PROFILE_FOLDER="$PWD/profiles" valgrind --tool=callgrind --instr-atstart=no \
102+
--callgrind-out-file="$PWD/profiles/cg.%p.%n" \
103+
uv run --no-sync pytest benchmarks/ \
104+
--ignore=benchmarks/test_concurrency_perf.py \
105+
-m "gate or informational" --codspeed -o addopts= -p no:cacheprovider
106+
107+
- name: Compare against committed baseline (report-only)
108+
if: ${{ !inputs.regen }}
109+
# report-only: prints the delta table, never fails the job. Add --enforce once trusted.
110+
run: |
111+
uv run --no-sync python benchmarks/compare_baseline.py compare \
112+
--profiles profiles --baseline benchmarks/baseline.json \
113+
--submodule-sha "${{ steps.duckdb_sha.outputs.sha }}" \
114+
--pins benchmarks/requirements-bench.txt
115+
116+
- name: Regenerate baseline (upload artifact to commit deliberately)
117+
if: ${{ inputs.regen }}
118+
run: |
119+
uv run --no-sync python benchmarks/compare_baseline.py regen \
120+
--profiles profiles --out benchmarks/baseline.json --gate-list gate_list.txt \
121+
--git-commit "${{ github.sha }}" --submodule-sha "${{ steps.duckdb_sha.outputs.sha }}" \
122+
--pins benchmarks/requirements-bench.txt
123+
124+
- name: Upload regenerated baseline
125+
if: ${{ inputs.regen }}
126+
uses: actions/upload-artifact@v4
127+
with:
128+
name: baseline-update
129+
path: benchmarks/baseline.json

‎.github/workflows/packaging_wheels.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
strategy:
3131
fail-fast: false
3232
matrix:
33-
python: [ cp311, cp314 ]
33+
python: [ cp314 ]
3434
platform:
3535
- { os: windows-2022, arch: amd64, cibw_system: win }
3636
- { os: windows-11-arm, arch: ARM64, cibw_system: win }

‎benchmarks/PLAN.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Benchmark suite plan
2+
3+
Design rationale for the binding micro-benchmarks. The suite is implemented in `benchmarks/`; CI lives in
4+
`../.github/workflows/codspeed.yml`; conventions, markers, and the two data-pattern traps are in
5+
[README.md](README.md).
6+
7+
Priority: **P0** = known-regression or cutover-reworked path (narrow-numeric common case); **P1** = high-traffic
8+
conversion or per-element Python work; **P2** = correctness-relevant, lower-traffic or engine-dominated.
9+
10+
## Scenarios
11+
12+
PRODUCE (duckdb to Python) is the highest regression risk: `Fetchone` builds a `TupleBuilder` per row and calls
13+
`FromValue` per cell (O(rows x cols), the shape of the historical ~15% fetchall regression).
14+
15+
- **OUT-row** (`test_fetch_perf`, `test_types_roundtrip_perf`): fetchall / fetchone / fetchmany per type. P0
16+
narrow numeric; P1 varchar, list, struct, and the expensive per-row types (decimal `Decimal()`, timestamptz
17+
pytz, hugeint string round-trip, uuid). Small-N `*_gate` probes isolate the compile+fetch fixed cost.
18+
- **OUT-col** (`test_produce_numpy_perf`): df() / fetchnumpy() reworked columnar path. P0 numeric no-null vs
19+
REAL-null (the masked_array branch); plus string, timestamp, and wide-internal (hugeint/uuid/decimal128).
20+
- **OUT-arrow / polars** (`test_arrow_perf`): to_arrow_table / reader / pl(). Informational (engine-parallel,
21+
GIL-released).
22+
- **Cardinality** (`test_cardinality_perf`): a LIMIT-n sweep giving a clean per-row conversion slope.
23+
24+
INGEST (Python to duckdb):
25+
26+
- **numpy / pandas** (`test_ingest_numpy_perf`, `test_pandas_perf`): numpy-backed scan (NaN-to-NULL, masked),
27+
object-string transcode ladder, arrow-backed zero-copy, and the per-bind PandasAnalyzer.
28+
- **arrow** (`test_arrow_perf`): Table + RecordBatchReader + dictionary sweep.
29+
- **native** (`test_ingest_native_perf`): values() list/tuple/dict per-cell TransformPythonValue, executemany.
30+
31+
UDF (`test_udf_perf`, zero coverage before this suite): native scalar per-row (P0, the biggest untested per-call
32+
path) and vectorized arrow per-chunk.
33+
34+
## Type x direction matrix
35+
36+
Directions: IN-native (TransformPythonValue), IN-numpy (NumpyScan), OUT-row (FromValue), OUT-col (ArrayWrapper),
37+
OUT-arrow.
38+
39+
| Type | IN-native | IN-numpy | OUT-row | OUT-col | OUT-arrow |
40+
|------|-----------|----------|---------|---------|-----------|
41+
| int32/int64 | P1 | **P0** | **P0** | **P0** | P1 |
42+
| double | P1 | **P0** (NaN->NULL) | P0 | P0 | P1 |
43+
| varchar | P1 | **P0** (PyUnicode) | P1 | P1 | P1 |
44+
| bool | P2 | P1 | P2 | P1 | P2 |
45+
| decimal64/128 | P2 | n/a | **P1** (Python Decimal) | P1 | P2 |
46+
| date | P2 | P1 | P1 | P1 | P2 |
47+
| timestamp(tz) | P1 | P1 | **P1** (pytz/row) | P1 | P1 |
48+
| LIST/STRUCT | P2 | P2 | P1 (recursive) | P1 | P2 |
49+
| hugeint/uuid | P2 | P2 | **P1** (round-trip) | P1 | P2 |
50+
| blob/map | P2 | P2 | P2 | P2 | P2 |
51+
| NULL-heavy | n/a | **P1** | P2 | **P0** (masked_array) | P1 |
52+
53+
## Mechanics
54+
55+
- **Walltime vs instruction-count.** Local A/B is walltime only (no Valgrind on macOS arm64). CI is
56+
instruction-count via self-hosted Callgrind (near-deterministic, PYTHONHASHSEED pinned), diffed against a
57+
committed baseline. Report-only until trusted.
58+
- **Marker split + auto-move.** Every benchmark is `gate` or `informational` (see README). At baseline regen,
59+
each numeric-produce gate's binding fraction `= 1 - floor_Ir / bench_Ir` is computed against its engine floor
60+
(`test_engine_control_perf`); a gate below the ~25% cutoff is auto-moved to informational (a threshold on an
61+
engine-diluted total is not meaningful). OUT-row fetch and UDFs are ~all binding; numeric produce is a bulk
62+
memcpy of ~engine magnitude (auto-move candidate).
63+
- **Guards.** compare_baseline.py warns and stops enforcing when BENCH_SCALE, the pin file, or the DuckDB
64+
submodule SHA differ from the baseline's (any of those makes the counts non-comparable).
65+
- **Sustained-leak guard** (`tests/fast/test_binding_pressure_leak.py`): a plain RSS + object-count test for the
66+
object-pinning paths, since a per-call refcount imbalance is invisible to a steady-state benchmark.
67+
- **Memory mode** (a second Callgrind sweep for O(rows) produce peak-RSS) is designed but deferred; the
68+
`test_mem_df_with_nulls` tracemalloc guard is the local stand-in.
69+
70+
## Cross-check vs iqmo-org/bareduckdb
71+
72+
Their suite is a SQL-file-driven A/B comparing two clients (production `duckdb` vs the C-API prototype), arrow-in
73+
/ arrow-out only, no fetchall/df/numpy/native/UDF coverage. So our binding suite is far broader; their genuine
74+
deltas concentrate in PRODUCE/types. Actionable additions they suggest:
75+
76+
- **hugeint / uuid in the produce matrix** (they select both): OUT-row does a per-value string round-trip, distinct
77+
from narrow int. Now in `test_produce_numpy_perf` / `test_fetch_perf`.
78+
- **int128-internal decimal** (`DECIMAL(28,x)`) alongside the int64-internal one: hits a wider cast path. Added.
79+
- **heterogeneous mixed-type row**: exercises per-cell type dispatch in the Fetchone loop, unlike homogeneous
80+
columns. Added as `test_fetchall_mixed_wide`.
81+
- **long varchar (>64 char)** alongside the short string: shifts string copy / transcode toward copy-bound. Added
82+
as `varchar_long` in the matrix.
83+
- **result-cardinality (top-N) sweep**: holds engine work ~constant while sweeping rows-to-Python. Adopted as
84+
`test_cardinality_perf` (plain LIMIT, no ORDER BY; the sort swamped the signal).
85+
- **peak-memory guard** on the O(rows) produce paths: a conversion regression is often memory-shaped. Partially
86+
covered by the tracemalloc guard; full coverage waits on memory mode.
87+
88+
Out of scope (theirs, not adopted): pure-engine filter/group/window workloads; 100M+ row scale (IO/engine
89+
dominated); the free-threading category (unsupported by this client). Do NOT adopt their no-warmup single-run
90+
methodology (charges import-cache population into the measurement).

‎benchmarks/README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Benchmark suite
2+
3+
CodSpeed micro-benchmarks for the binding hot paths (produce, ingest, UDF).
4+
Design rationale: [PLAN.md](PLAN.md). CI: [../.github/workflows/codspeed.yml](../.github/workflows/codspeed.yml).
5+
6+
## Markers
7+
8+
Every benchmark carries exactly one (registered in `conftest.py`):
9+
10+
- **gate**: binding-dominated, GIL-held, deterministic under Callgrind. A threshold breach is a binding regression.
11+
- **informational**: engine/library/streaming-diluted. Reported, never gated (would false-positive on engine bumps).
12+
13+
## Local A/B (walltime)
14+
15+
Only walltime runs locally (no Valgrind on macOS arm64; instruction-count gating is Linux/CI-only, and walltime is
16+
noisy on sub-ms benches). Pin the data libs identically across both builds so the delta is pure binding:
17+
18+
```bash
19+
for P in ../main/.venv-release/bin/python .venv-release/bin/python; do
20+
$P -m pytest benchmarks/<module>.py --codspeed --codspeed-mode=walltime -o addopts= -p no:cacheprovider
21+
done
22+
```
23+
24+
## Conventions
25+
26+
- READ aggregates real columns (`sum`/`length`), never `count(*)` (answered from metadata).
27+
- WRITE fully materializes the result or drains the lazy reader.
28+
- Warm once before measuring.
29+
- `con` fixture pins `threads=1` (see `conftest.py`).
30+
31+
Two traps (a benchmark that skips these silently measures the wrong thing):
32+
33+
- OUT-col null benches need REAL nulls (`CASE WHEN ... THEN NULL`), else the cheap `std::move` path is taken.
34+
- IN-numpy string benches need mixed ASCII + non-ASCII + a null sentinel, else the transcode/null ladder is skipped.

‎benchmarks/_scale.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
"""Env-gated row-count scaling for the benchmark suite.
2+
3+
Callgrind is 20-50x, so the O(rows) benches at full N make the CI sweep too slow. `scaled(n)` shrinks row counts
4+
ONLY when `BENCH_SCALE=<divisor>` is set (which the CI sweep sets); unset -> full N, so local walltime A/B is
5+
unchanged. A gate bench and the engine floor it is compared against share a base N, so routing BOTH through
6+
`scaled()` keeps them at an identical scaled N and the binding fraction stays valid. Scaling reduces row counts
7+
only; it must never change the data patterns the benches depend on (real nulls, mixed ASCII, LIMIT-no-ORDER-BY).
8+
A floor keeps a scaled bench row-dominated so per-element work still dominates; the small-N `*_gate` probes are
9+
already fast and are NOT scaled.
10+
"""
11+
12+
from __future__ import annotations
13+
14+
import os
15+
16+
FLOOR = 20_000 # a scaled bench never drops below this (stays row-dominated, ~10x the range(2048) probes)
17+
18+
19+
def bench_scale() -> int:
20+
"""Return the divisor from `BENCH_SCALE` (>=1); 1 (no scaling) if unset/invalid."""
21+
v = os.environ.get("BENCH_SCALE")
22+
if not v:
23+
return 1
24+
try:
25+
return max(int(v), 1)
26+
except ValueError:
27+
return 1
28+
29+
30+
def scaled(n: int) -> int:
31+
"""Return `n` at full scale, or `max(n // BENCH_SCALE, min(n, FLOOR))` when scaling is enabled."""
32+
d = bench_scale()
33+
if d <= 1:
34+
return n
35+
return max(n // d, min(n, FLOOR))

0 commit comments

Comments
 (0)