Add in-process pg_bench benchmarking for pgrx extensions (pgcentralfoundation#2263)

Introduce a new `#[pg_bench]` benchmarking workflow for pgrx extensions,
backed by a new optional pgrx-bench crate and a new cargo pgrx bench
subcommand.

This adds Criterion-driven, in-process benchmarks that run inside a
Postgres backend, using the same managed cluster lifecycle as cargo pgrx
run rather than the ephemeral pg_test harness. Benchmarks are installed
into a managed database named $extname_benches, and each cargo pgrx
bench invocation is recorded as a persistent benchmark group with
environment metadata, git metadata, pg_settings snapshots, and
normalized benchmark results.

Highlights

- add #[pg_bench] proc macro in pgrx-macros
- require benchmarks to live in #[cfg(feature = "pg_bench")]
#[pg_schema] mod benches
- add per-benchmark setup support via #[pg_bench(setup = some_path)]
- add transaction modes:
  - shared
  - subtransaction_per_batch
  - subtransaction_per_iteration
- add new pgrx-bench runtime crate exposing:
  - Bencher
  - BatchSize
  - black_box
- add cargo pgrx bench command
- build/install bench-enabled extensions through the normal install
path, including control files and generated SQL scripts
- refresh the extension in $extname_benches with DROP EXTENSION / CREATE
EXTENSION while preserving runner-owned benchmark history
- store benchmark history in persistent pgrx_bench schema tables and
views
- stream per-benchmark results as each benchmark finishes
- emit Criterion-like CLI output for time/change/slope/mean/median
- persist raw Criterion artifacts and replay them as baselines for later
comparisons
- switch comparison reporting to Criterion-driven change data instead of
host-side approximations
- warn, but do not fail, when pg_bench is explicitly enabled for cargo
pgrx run/install/package

Result storage

Benchmark history is stored outside extension ownership in the
pgrx_bench schema so it survives extension refreshes. Each run group
captures:

- benchmark group name and comparison group
- extension/package/profile metadata
- full command line
- host and toolchain metadata
- git metadata
- pg_settings snapshot
- benchmark case/run/estimate/sample/throughput rows
- raw Criterion JSON artifacts for later comparison replay

Comparison behavior

Comparisons now use persisted Criterion artifacts from a prior benchmark
group. The backend reconstructs Criterion’s base/ layout, reruns the
benchmark, reads Criterion’s change/estimates.json, and mirrors
Criterion 0.5.1’s mixed-bootstrap T-test path for p-value reporting.
This keeps change intervals and status messages aligned with Criterion’s
own model.

Examples, docs, and CI

- add new pgrx-examples/benching example crate
- add example README showing #[pg_bench], setup functions, and
transaction handling
- add PG_BENCH-SPEC.md design/spec document
- expand pgrx-bench README with feature overview, workflow guidance, and
black_box discussion
- update cargo-pgrx README to document cargo pgrx bench
- add CI smoke coverage for the new benching example
- update publish.sh for the new pgrx-bench crate

Tests

- add unit tests in pgrx-bench for:
  - Criterion new-vs-base directory selection
  - baseline artifact materialization
  - artifact collection
  - Criterion-style summary classification
  - deterministic p-value behavior
  - comparison parsing from persisted baseline artifacts

This establishes pg_bench as the in-process benchmarking counterpart to
pg_test for pgrx extensions, with persistent historical storage and
Criterion-compatible comparison reporting.

--- 

NOTE: 100% AI, but this is a super cool feature. Something I've always
wanted to do but just never had the time. Only took a few hours with
Codex.

Author: eeeebbbbrrrr

.github/workflows/tests.yml

@@ -279,6 +279,14 @@ jobs:
             CARGO_TARGET_DIR="$(pwd)/target" cargo test --manifest-path=pgrx-examples/$example/Cargo.toml  --features "pg$PG_VER" --no-default-features
           done
 
+      - name: Smoke test cargo pgrx bench against the benching example
+        if: ${{ matrix.postgres == 16 && matrix.rust == null }}
+        run: |
+          CARGO_TARGET_DIR="$(pwd)/target" cargo pgrx bench pg$PG_VER \
+            bench_normalize_phrase \
+            --manifest-path=pgrx-examples/benching/Cargo.toml \
+            --group-name ci_benching_smoke
+
       - name: Run custom_types without schema generation example tests
         run: CARGO_TARGET_DIR="$(pwd)/target" cargo test --manifest-path=pgrx-examples/custom_types/Cargo.toml  --features "pg$PG_VER no-schema-generation" --no-default-features
 

.gitignore

@@ -10,4 +10,7 @@ cmake*/
 .direnv
 pgrx-examples/*/target
 /bin
-rustup-init.sh
+rustup-init.sh
+.claude/
+.codex/
+.agents/

Cargo.lock

@@ -13,6 +13,7 @@ resolver = "3"
 members = [
     "cargo-pgrx",
     "pgrx",
+    "pgrx-bench",
     "pgrx-macros",
     "pgrx-pg-config",
     "pgrx-pg-sys",
@@ -27,6 +28,7 @@ exclude = [
     "pgrx-examples/arrays",
     "pgrx-examples/bad_ideas",
     "pgrx-examples/bgworker",
+    "pgrx-examples/benching",
     "pgrx-examples/bytea",
     "pgrx-examples/composite_type",
     "pgrx-examples/custom_libname",
@@ -68,6 +70,7 @@ version = "0.16.0"
 cargo-pgrx = { path = "cargo-pgrx" }
 
 [workspace.dependencies]
+pgrx-bench = { path = "./pgrx-bench", version = "=0.17.0" }
 pgrx-macros = { path = "./pgrx-macros", version = "=0.17.0" }
 pgrx-pg-sys = { path = "./pgrx-pg-sys", version = "=0.17.0" }
 pgrx-sql-entity-graph = { path = "./pgrx-sql-entity-graph", version = "=0.17.0" }

Cargo.toml

@@ -53,12 +53,15 @@ tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }
 bzip2 = "0.5.2"
 env_proxy = "0.4.1"
 serde.workspace = true
+serde_json.workspace = true
 serde-xml-rs = "0.6.0"
 tar = "0.4.44"
 ureq = { version = "3.0.10", default-features = false, features = ["gzip", "platform-verifier", "rustls"] }
 url.workspace = true
 which = "7.0.3"
 zip-extract = "0.2.2"
+postgres = { version = "0.19.10", features = ["with-serde_json-1", "with-uuid-1"] }
+uuid = { version = "1.16.0", features = ["v4"] }
 
 # SQL schema generation
 proc-macro2.workspace = true