bench: internal performance benchmark suite + CodSpeed CI (#781)

Author: FBumann

.github/dependabot.yml

@@ -12,3 +12,30 @@ updates:
     github-actions:
       patterns:
       - '*'
+
+# Pinned ``[benchmarks]`` extra in pyproject.toml. One PR per dep bump
+# → CodSpeed CI runs and attributes any perf delta to that specific
+# bump. Keeps the cross-version ``sweep`` baseline (lockfile-pinned)
+# stable while still surfacing upstream perf changes per-PR with
+# eyes-open review. Loose ``[project.dependencies]`` (numpy, scipy, ...)
+# have no version specifier so Dependabot leaves them alone — only the
+# ``==`` pins in ``[benchmarks]`` produce PRs.
+- package-ecosystem: pip
+  directory: /
+  schedule:
+    interval: monthly
+  open-pull-requests-limit: 5
+  groups:
+    # Measurement scaffolding + CLI/notebook tooling. Perf-irrelevant —
+    # they don't move CodSpeed signal, so batching into one PR cuts
+    # review noise. Perf-relevant deps (numpy, xarray, highspy, …) stay
+    # un-grouped so each gets its own attributed CodSpeed delta.
+    benchmark-tooling:
+      patterns:
+        - pytest
+        - pytest-benchmark
+        - pytest-memray
+        - pytest-codspeed
+        - nbconvert
+        - typer
+        - plotly

.github/workflows/benchmark-smoke.yml

@@ -0,0 +1,39 @@
+name: Benchmark smoke
+
+# Builds every spec and fires every phase once (--benchmark-disable):
+# a "did a refactor break a spec?" check, not timing.
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ '*' ]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  smoke:
+    name: Benchmark smoke (quick)
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        fetch-depth: 0  # setuptools_scm
+
+    - name: Set up Python 3.13
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.13"
+
+    - name: Install package and benchmark dependencies
+      run: |
+        python -m pip install uv
+        uv pip install --system -e ".[dev,benchmarks]"
+
+    - name: Run benchmark smoke
+      # Every spec builds at one size and every phase fires once, no timings.
+      run: |
+        pytest benchmarks/ --benchmark-disable -q

.github/workflows/codspeed.yml

@@ -0,0 +1,76 @@
+name: CodSpeed
+
+# Both instruments in one workflow (CodSpeed aggregates per-commit results from a
+# single workflow). Memory: heap tracking on a free GitHub runner, every PR +
+# master. Walltime: bare-metal macro runner, master + the maintainer-gated
+# `trigger:benchmark` label only.
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+    types: [ opened, synchronize, reopened, labeled ]
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  memory:
+    name: Memory (heap)
+    runs-on: ubuntu-latest
+    continue-on-error: true   # informational, never blocks a merge
+    permissions:
+      contents: read   # actions/checkout
+      id-token: write  # OIDC auth with CodSpeed — no token secret
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        fetch-depth: 0  # setuptools_scm
+    - name: Set up Python 3.13
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.13"
+    - name: Install pinned benchmark environment
+      run: |
+        python -m pip install uv
+        uv pip install --system -e ".[dev,benchmarks]"
+    - name: Run benchmarks under CodSpeed (memory)
+      uses: CodSpeedHQ/action@v4
+      with:
+        mode: memory
+        run: |
+          pytest benchmarks/ --codspeed
+
+  walltime:
+    name: Walltime (macro runner)
+    # Master push / dispatch always; PRs only when explicitly labelled — macro
+    # minutes are metered and bare-metal shouldn't run arbitrary PR code.
+    if: >-
+      ${{ github.event_name != 'pull_request' ||
+          contains(github.event.pull_request.labels.*.name, 'trigger:benchmark') }}
+    runs-on: codspeed-macro
+    continue-on-error: true
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        fetch-depth: 0  # setuptools_scm
+    - name: Set up Python 3.13
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.13"
+    - name: Install pinned benchmark environment
+      run: |
+        python -m pip install uv
+        uv pip install --system -e ".[dev,benchmarks]"
+    - name: Run benchmarks under CodSpeed (walltime)
+      uses: CodSpeedHQ/action@v4
+      with:
+        mode: walltime
+        run: |
+          pytest benchmarks/ --codspeed

.gitignore

@@ -45,6 +45,10 @@ benchmark/scripts/__pycache__
 benchmark/scripts/benchmarks-pypsa-eur/__pycache__
 benchmark/scripts/leftovers/
 
+# Benchmarks (internal suite): regenerable .ipynb viewing artifacts
+benchmarks/walkthrough.ipynb
+benchmarks/.ipynb_checkpoints/
+
 # IDE
 .idea/
 

benchmarks/README.md

@@ -1,94 +1,81 @@
 # Internal Performance Benchmarks
 
-Measures linopy's own performance (build time, LP write speed, memory usage) across problem sizes using [pytest-benchmark](https://pytest-benchmark.readthedocs.io/) and [pytest-memray](https://pytest-memray.readthedocs.io/). Use these to check whether a code change introduces a regression or improvement.
+End-to-end performance tracking for `linopy` — build → matrix generation →
+LP / netCDF (de)serialization → solver handoff → a fixed PyPSA model. Solver
+algorithm runtime is out of scope.
 
-> **Note:** The `benchmark/` directory (singular) contains *external* benchmarks comparing linopy against other modeling frameworks. This directory (`benchmarks/`) is for *internal* performance tracking only.
+The suite is a set of `pytest-benchmark` tests driven by a model registry.
+**CodSpeed** measures them in CI (walltime on dedicated runners, memory on every
+PR); locally you just run `pytest`.
 
-## Setup
+> `benchmark/` (singular) is the legacy external-framework suite.
+> `benchmarks/` (plural) is this internal suite.
 
-```bash
-pip install -e ".[benchmarks]"
-```
+## Layout
 
-## Running benchmarks
+- `registry.py`, `phases.py`, `conftest.py` — the harness (specs, measured
+  verbs, pytest wiring).
+- `models/`, `patterns/` — the subjects; each file self-registers one `BenchSpec`.
+- `drivers/` — one `test_<phase>.py` per measured phase.
 
-```bash
-# Quick smoke test (small sizes only)
-pytest benchmarks/ --quick
+## Models vs patterns
 
-# Full timing benchmarks
-pytest benchmarks/test_build.py benchmarks/test_lp_write.py benchmarks/test_matrices.py
+Two kinds of benchmark spec, same harness and same phases, distinguished by
+their sweep axis:
 
-# Run a specific model
-pytest benchmarks/test_build.py -k basic
-```
+- **Models** (`models/`, `REGISTRY`) — whole `linopy.Model`s swept over
+  `size` (axis `n`): "how does cost scale with the problem?"
+- **Patterns** (`patterns/`, `PATTERNS`) — fragments of realistic modelling
+  code (a balance constraint, a KVL contraction) swept over `severity`
+  (0–100, axis `severity`): "how does cost respond as one data shape goes
+  from benign to pathological?"
 
-## Comparing timing between branches
+Both kinds build a complete `linopy.Model`, so both run the **same phases** and
+share the phase drivers (`drivers/test_build.py`, `drivers/test_matrices.py`, …)
+— they're just more `(spec, value)` rows, tagged by `axis`. There is no separate
+pattern driver. Running a pattern through `build` *and* `to_lp` shows whether a
+dense-`_term` blow-up propagates to export or collapses.
 
-```bash
-# Save baseline results on master
-git checkout master
-pytest benchmarks/test_build.py --benchmark-save=master
+Patterns target the operations where the dense-`_term` representation forces
+materialisation — `groupby().sum()` padding, sparse `@` densification — so a
+`severity` sweep draws the cost cliff. Adding either kind is one file: drop it
+in `models/` or `patterns/`, call `register(...)` / `register_pattern(...)`.
 
-# Switch to feature branch and compare
-git checkout my-feature
-pytest benchmarks/test_build.py --benchmark-save=my-feature --benchmark-compare=0001_master
+## Install
 
-# Compare saved results without re-running
-pytest-benchmark compare 0001_master 0002_my-feature --columns=median,iqr
+```bash
+uv sync --extra dev --extra benchmarks
+source .venv/bin/activate
 ```
 
-Results are stored in `.benchmarks/` (gitignored).
+`pypsa` is optional — `pypsa_scigrid` and `drivers/test_pypsa_carbon_management.py`
+skip gracefully without it: `uv pip install pypsa`.
 
-## Memory benchmarks
+The `[benchmarks]` extra in `pyproject.toml` pins every direct dep that affects
+measurement (`numpy`, `scipy`, `xarray`, `pandas`, `polars`, `dask`, …) so
+run-to-run deltas reflect linopy changes, not dependency bumps.
 
-`memory.py` runs each test in a separate process with pytest-memray to get accurate per-test peak memory (including C/numpy allocations). Results are saved as JSON and can be compared across branches.
-
-By default, only the build phase (`test_build.py`) is measured. Unlike timing benchmarks where `benchmark()` isolates the measured function, memray tracks all allocations within a test — including model construction in setup. This means LP write and matrix tests would report build + phase memory combined, making the phase-specific contribution impossible to isolate. Since model construction dominates memory usage, measuring build alone gives the most actionable numbers.
+## Running
 
 ```bash
-# Save baseline on master
-git checkout master
-python benchmarks/memory.py save master
-
-# Save feature branch
-git checkout my-feature
-python benchmarks/memory.py save my-feature
-
-# Compare
-python benchmarks/memory.py compare master my-feature
-
-# Quick mode (smaller sizes, faster)
-python benchmarks/memory.py save master --quick
-
-# Measure a specific phase (includes build overhead)
-python benchmarks/memory.py save master --test-path benchmarks/test_lp_write.py
+pytest benchmarks/                       # the suite
+pytest benchmarks/ --benchmark-disable -q   # smoke: every spec builds once
+pytest benchmarks/ --pipeline            # + the opt-in end-to-end pipeline test
 ```
 
-Results are stored in `.benchmarks/memory/` (gitignored). Requires Linux or macOS (memray is not available on Windows).
-
-> **Note:** Small tests (~5 MiB) are near the import-overhead floor and may show noise of ~1 MiB between runs. Focus on larger tests for meaningful memory comparisons. Do not combine `--memray` with timing benchmarks — memray adds ~2x overhead that invalidates timing results.
-
-## Models
-
-| Model | Description | Sizes |
-|-------|-------------|-------|
-| `basic` | Dense N*N model, 2*N^2 vars/cons | 10 — 1600 |
-| `knapsack` | N binary variables, 1 constraint | 100 — 1M |
-| `expression_arithmetic` | Broadcasting, scaling, summation across dims | 10 — 1000 |
-| `sparse_network` | Ring network with mismatched bus/line coords | 10 — 1000 |
-| `pypsa_scigrid` | Real power system (requires `pypsa`) | 10 — 200 snapshots |
-
-## Phases
+Each spec declares one `sizes` (models) / `severities` (patterns) tuple — a
+small representative set, kept tight because CodSpeed measures it on every PR.
+Need a scaling curve? That's a local pytest-benchmem job, not this suite.
 
-| Phase | File | What it measures |
-|-------|------|------------------|
-| Build | `test_build.py` | Model construction (add_variables, add_constraints, add_objective) |
-| LP write | `test_lp_write.py` | Writing the model to an LP file |
-| Matrices | `test_matrices.py` | Generating sparse matrices (A, b, c, bounds) from the model |
+## CI
 
-## Adding a new model
+- **Smoke** (`benchmark-smoke.yml`) — every PR: every spec builds and every
+  phase fires once under `--benchmark-disable`. A "did a refactor break a
+  spec?" check, not timing.
+- **CodSpeed** (`codspeed.yml`) — two jobs: **memory** (heap-allocation
+  tracking, every PR, free GitHub runner) and **walltime** (bare-metal macro
+  runner, on `master` or a PR labelled `trigger:benchmark`). Informational,
+  non-gating.
 
-1. Create `benchmarks/models/my_model.py` with a `build_my_model(n)` function and a `SIZES` list
-2. Add parametrized tests in the relevant `test_*.py` files
-3. Add a quick threshold in `conftest.py`
+Activating CodSpeed upstream needs a maintainer to connect the repo to the
+CodSpeed app (OIDC auth, no token secret); the workflows are already wired.

benchmarks/__init__.py

@@ -1 +1,15 @@
-"""Linopy benchmark suite — run with ``pytest benchmarks/`` (use ``--quick`` for smaller sizes)."""
+"""
+Linopy benchmark suite — run with ``pytest benchmarks/``.
+
+The model registry it drives is reusable on its own::
+
+    from benchmarks import REGISTRY
+    model = REGISTRY["basic"].build(100)
+"""
+
+# Importing the models / patterns packages triggers each module's
+# ``register(...)`` / ``register_pattern(...)`` call at import time.
+from benchmarks import models, patterns  # noqa: F401
+from benchmarks.registry import PATTERNS, REGISTRY
+
+__all__ = ["PATTERNS", "REGISTRY"]