Redesign cardinality benchmark + add tokenless CodSpeed CI

Cardinality: the ORDER BY ... LIMIT n sweep let the engine top-N sort dominate
and swamp the per-row conversion signal (numbers were non-monotone). Replace it
with a pre-materialized fixed source + plain LIMIT n (no sort): the scan
early-stops at n rows, so rows-to-Python conversion is the dominant n-varying
cost and the slope is monotone; the A/B delta at each n isolates the binding.

CI: .github/workflows/codspeed.yml runs the suite under CodSpeed simulation
(instruction-count) mode on Linux, tokenless (no dashboard upload; enable the
hosted gate later via a CodSpeed project + OIDC/token). Instruction counts are
deterministic for every benchmark, so no gated/informational split is needed.
Not yet run in CI; the build steps mirror the dev build and need a shakeout.

Author: evertlammerts

.github/workflows/codspeed.yml

@@ -0,0 +1,74 @@
+# Performance-regression benchmarks via CodSpeed, in deterministic instruction-count (simulation) mode.
+#
+# TOKENLESS FOR NOW: the CodSpeed action's token is only needed to UPLOAD results to the CodSpeed
+# dashboard (the hosted PR-gate). Without it the action still RUNS every benchmark under Valgrind and
+# reports the instruction counts in the job log. To turn on the hosted regression gate later: create a
+# CodSpeed project for the repo and either (public repo) rely on the OIDC `id-token: write` permission
+# below, or add a `CODSPEED_TOKEN` repo secret and pass `token: ${{ secrets.CODSPEED_TOKEN }}` to the action.
+#
+# Why simulation (instruction-count) and not walltime: instruction counts are deterministic even for the
+# multi-threaded / engine-heavy paths (Valgrind serializes and counts), so the whole suite is gate-able and
+# there is no need to split "gated vs informational" the way noisy local walltime required. Instruction count
+# is exactly the signal that would have caught the LIST/ARRAY df() regression cleanly.
+#
+# NOTE: this workflow has not been run in CI yet; the build steps mirror the documented dev build (CLAUDE.md)
+# and will likely need a shakeout run. Valgrind is slow (~20-50x); if the full suite is too slow, trim the
+# largest-N benchmarks or run a curated subset.
+
+name: Benchmarks
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+concurrency:
+  group: codspeed-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  benchmarks:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write # enables tokenless (OIDC) upload once a CodSpeed project is linked; harmless otherwise
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive # the DuckDB engine submodule is needed to build
+          fetch-depth: 0 # setuptools_scm needs history for version detection
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.13"
+
+      - name: Cache sccache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/sccache
+          key: sccache-codspeed-${{ hashFiles('external/duckdb') }}
+          restore-keys: sccache-codspeed-
+
+      - name: Install sccache
+        run: |
+          curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.8.2/sccache-v0.8.2-x86_64-unknown-linux-musl.tar.gz \
+            | tar -xz --strip-components=1 -C /usr/local/bin sccache-v0.8.2-x86_64-unknown-linux-musl/sccache
+
+      - name: Build the extension (release) + benchmark deps
+        env:
+          CMAKE_C_COMPILER_LAUNCHER: sccache
+          CMAKE_CXX_COMPILER_LAUNCHER: sccache
+        run: |
+          uv sync --only-group build --no-install-project -p 3.13
+          uv sync --no-build-isolation --no-editable --reinstall -p 3.13
+          # benchmark deps: keep these pinned in lockstep with any baseline you compare against, so the only
+          # cross-run delta is the binding (numpy/pandas/pyarrow/polars/pytz + the codspeed plugin).
+          uv pip install pytest pytest-codspeed numpy pandas pyarrow polars pytz
+
+      - name: Run benchmarks (instruction-count)
+        uses: CodSpeedHQ/action@v4
+        with:
+          mode: simulation
+          run: uv run pytest benchmarks/ --codspeed -o addopts= -p no:cacheprovider

benchmarks/test_cardinality_perf.py

@@ -1,47 +1,50 @@
-"""Standalone CodSpeed benchmark module: the RESULT-CARDINALITY (top-N) sweep — NOT integrated (not in
-pyproject, not in CI, not committed). Run under each build's interpreter and compare:
+"""Standalone CodSpeed benchmark: the RESULT-CARDINALITY (rows-to-Python) sweep. Run under each build:
 
   M=/Users/evert/projects/duckdb-python/main/.venv-release/bin/python
   C=/Users/evert/projects/duckdb-python/wt-codspeed/.venv-release/bin/python
   cd /Users/evert/projects/duckdb-python/wt-codspeed
   $M -m pytest benchmarks/test_cardinality_perf.py --codspeed --codspeed-mode=walltime -o addopts= -p no:cacheprovider
   $C -m pytest benchmarks/test_cardinality_perf.py --codspeed --codspeed-mode=walltime -o addopts= -p no:cacheprovider
 
-WHY THIS MODULE (adopted from iqmo-org/bareduckdb): hold the SOURCE fixed and sweep only the number of rows
-materialized to Python via ORDER BY ... LIMIT n for n in {100, 1k, 10k, 100k}, through fetchall / df /
-to_arrow_table. The engine cost (scan the fixed SRC + top-N heap) stays ~constant, so the walltime delta
-across n is dominated by the per-row binding conversion -> a clean per-row slope. The n=100 end is the
-noise-free overhead regime (the natural instruction-count-gate point); the n=100k end is throughput.
-
-A clean monotone slope (and ~parity slope between the two builds) is the signal we report; a build whose slope
-is steeper has a per-row conversion regression. Source held constant rules out scan-cost as the confound (a
-cleaner axis than varying range(), which also changes scan cost).
-
-numpy/pandas/pyarrow are pinned to the SAME versions in both .venv-release, so the A/B delta is purely the binding.
+REDESIGN NOTE: the first version swept `ORDER BY a DESC LIMIT n` over a fixed source. That was wrong:
+the engine's full top-N SORT (~3-14ms, itself variable) dominated and swamped the per-row conversion
+signal, and the numbers came out non-monotone. This version pre-materializes the fixed source table ONCE
+and sweeps `SELECT * FROM src LIMIT n` with NO ORDER BY: a plain LIMIT early-stops the scan at n rows, so
+the engine cost is cheap and monotone in n, and the rows-to-Python CONVERSION is the dominant n-varying
+cost. That gives a clean, monotone per-row slope; the A/B delta at each n isolates the binding, and a build
+whose slope is steeper has a per-row conversion regression. n=100 is the overhead regime (the natural
+instruction-count-gate point); n=100_000 is throughput.
+
+3 columns (BIGINT, DOUBLE, VARCHAR) so per-row conversion is non-trivial. numpy/pandas/pyarrow are pinned to
+the SAME versions in both .venv-release, so the A/B delta is purely the binding.
 """
 
 import duckdb
 import pytest
 
-SRC = 200_000  # fixed source size -> constant engine scan + top-N across all n
+SRC_ROWS = 200_000
 LIMITS = [100, 1_000, 10_000, 100_000]
 
-# 3 columns (BIGINT, DOUBLE, VARCHAR) so the per-row conversion is non-trivial; source is a fixed inline
-# subquery (no table state) and ORDER BY forces a full scan + top-N of the same SRC rows every time.
-_SRC_SUBQ = f"(SELECT i::BIGINT AS a, (i * 1.5)::DOUBLE AS b, ('s_' || i) AS s FROM range({SRC}) t(i))"
-
-
-def _query(n):
-    return f"SELECT a, b, s FROM {_SRC_SUBQ} ORDER BY a DESC LIMIT {n}"
 
-
-@pytest.fixture
+@pytest.fixture(scope="module")
 def con():
+    # Fixed source materialized ONCE (module-scoped): building it per test would add noise, and it must be
+    # identical across the n sweep. `SELECT * FROM src LIMIT n` then reads only the first n rows.
     c = duckdb.connect()
+    c.execute(
+        "CREATE TABLE src AS "
+        f"SELECT i::BIGINT AS a, (i * 1.5)::DOUBLE AS b, ('s_' || i) AS s FROM range({SRC_ROWS}) t(i)"
+    )
     yield c
     c.close()
 
 
+def _query(n):
+    # No ORDER BY: a plain LIMIT early-stops the scan at n rows -> engine cost cheap and monotone in n, so the
+    # per-row binding conversion dominates the n-varying signal (unlike the old ORDER BY top-N sort).
+    return f"SELECT a, b, s FROM src LIMIT {n}"
+
+
 @pytest.mark.parametrize("n", LIMITS)
 def test_limit_fetchall(benchmark, con, n):
     q = _query(n)