Merge pull request #1818 from graphistry/docs/bench-numbers-consumer

docs(gfql): print the benchmark numbers pyg-bench publishes, and re-verify them

Author: lmeyerov

CHANGELOG.md

@@ -44,6 +44,8 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **The polars CI lane runs in parallel, so the coverage cell stops racing its timeout**: `test-polars (3.12)` is the only cell that runs the coverage-instrumented pass *and* the per-file coverage audit, and it had already been cancelled twice at its 10-minute budget (615s, 611s) with zero test failures — pure budget exhaustion, with the same commit passing or failing depending on the runner it drew. `bin/test-polars.sh` now runs its main phase under `pytest-xdist` (`-n auto --maxprocesses 4 --dist load`); `pytest-xdist` was already in the `[test]` extra and `test-gfql-core` already runs `-n auto` under `--cov`, so no dependency changes. `auto` resolves to the runner's core count (4 on a GitHub-hosted `ubuntu-latest`) and scales *down* on a 2-vCPU runner where a fixed `-n 4` could be slower than serial; `--maxprocesses` keeps a many-core dev box from fanning out one polars process per core. Verified rather than assumed, on py3.12 / polars 1.43.1 / pandas 3.0.5 with `POLARS_COV=1`: the pass/skip **node-id sets** (not just counts) are identical serial vs parallel across 4-worker, 2-worker and `--dist loadfile` runs (2417 node ids, 2404 passed / 13 skipped every time); the merged coverage data is a strict superset of the serial data (28,480 vs 28,478 covered lines, zero lines lost, zero files dropped); the second `--cov-append` invocation still appends into the xdist-produced data file (+1,791 lines, nothing lost); and `bin/coverage_audit.py --profile gfql-polars` emits a byte-identical report (modulo its timestamp) from the parallel data, so the downstream `changed-line-coverage` check sees no change. Measured **on CI**, by this change's own run: the `test-polars (3.12)` cell goes **501s → 322s**, its script step **484s → 297s (1.63×)**, and the five non-coverage cells go 222–318s → 115–172s. (A local 4-CPU-pinned A/B with coverage on showed 326s → 109s, 2.98×; CI realizes less, because coverage tracing is per-worker CPU cost that does not parallelize away and the ~8s serial second phase plus 4× interpreter startup are fixed. The CI number is the one to quote.) `--dist load` rather than `loadfile` because one module (`test_engine_polars_chain.py`) is 69% of the lane's test time, capping file-level distribution at ~1.4×; `POLARS_XDIST=0` restores the serial path and `POLARS_XDIST_DIST` selects another distribution mode should a future order-dependent test need one.
 
 ### Documentation
+- **The filter -> PageRank -> filter benchmark is measured again, and the docs print it from data.** `gfql/benchmark_filter_pagerank.rst` published Twitter and GPlus figures whose charts were generated from a results directory that has never existed in any commit, so nobody could re-derive them. The benchmark now lives in `graphistry/pyg-bench`, which re-ran it on dgx-spark under the shared perf lock against a named PyGraphistry commit and committed both the raw artifacts and the published document. This page renders that document instead of restating it: Twitter GFQL GPU 0.24 s and CPU 1.58 s against Neo4j + GDS 11.72 s (49.5x and 7.4x), GPlus GFQL GPU 2.42 s and CPU 32.10 s (13.3x). Every arm reproduces the earlier run's final subgraphs exactly.
+- **Benchmark numbers are now referenced, not retyped.** `docs/source/_data/gfql_benchmarks.json` is a vendored copy of pyg-bench's published artifact and `gfql_benchmarks.contract.json` a vendored copy of the contract it satisfies. The new `gfql_bench` Sphinx extension resolves `:bench:` and `:bench-diag:` against it and re-verifies the contract independently before rendering anything, so the docs build fails on a key that is not published, a measurement older than the policy allows, a diagnostic-only figure printed as a result, or a page that drops a number's provenance or disclosures. `docs/test_bench_numbers.py` runs the same checks in the ordinary test lane, so a number going stale fails CI rather than a nightly.
 - **GFQL pay-as-you-go resident indexing user guide**: New :doc:`Pay-As-You-Go Resident Indexing <gfql/indexing>` page — the lifecycle guide to resident indexes (`gfql_index_all()` / `gfql_index_edges()` / `create_index()` / `show_indexes()` / `drop_index()`): what the node-id + CSR in/out adjacency sidecars are, what engages them on 0.58.0 (seeded typed-hop fast paths incl. property RETURNs and property-seeded lookups per #1768/#1770, direct `g.hop()`; the general polars chain traversal honestly noted as not yet covered), the staleness/validity contract (identity + fingerprint; rebind invalidates; declines are safe — identical results either way), engine notes (polars needs `gfql_index_all(engine='polars')` until #1767), 0.58.0-tag measured numbers, and a runnable end-to-end example. Wired into the GFQL toctree + recommended paths alongside :doc:`Seeded Traversal Indexes <gfql/index_adjacency>`.
 - **GFQL performance docs: 0.58.0 release-tag-verified numbers, siloed in one page**: `gfql/performance.rst` is now the canonical benchmark-numbers page (alongside `gfql/index_adjacency.rst` for the index benchmarks) — a benchmark rerun updates it alone. It carries the 0.58.0 tag sweep (DGX Spark GB10, warm medians N=30; four-engine numbers cross-engine parity-verified, competitor pairs validated against expected result rows): seeded typed-hop fast path across all four engines (e.g. pandas 29.9→2.46ms, 12.1×), native chain form, resident-index covered-shape lookups (with the `gfql_index_all(engine='polars')` caveat / PR #1767), flat seeded-hop scaling on pandas (0.159–0.164ms from 0.25M to 32M edges), the one-keyword `engine='polars'` LDBC SNB SF1 seed-lookup win (1,299.6→106.1ms, 12.3×), LDBC SNB interactive SF1 vs Neo4j 5.26 same-box pairs (GFQL 4 of 5; Neo4j wins recent-replies — reported as-is), OLAP multi-join vs embedded Kuzu (q9 1.26×) with the honest inverse (Kuzu wins single-table aggregates 2–4×, seeded property-projection lookups 2.4–64×), plus the prior Orkut/LiveJournal bulk sweep (moved from `engines.rst`, dated once) and its methodology. All other pages — `engines.rst`, `quick.rst`, `about.rst`, `overview.rst`, `index.rst` — now carry stable qualitative claims (e.g. "often an order of magnitude faster on query-heavy workloads") that link into `performance.rst` instead of inline figures, replacing the stale "up to ~38×" headlines and avoiding scattered per-claim version labels.
 - **Withdrew the OLAP q8 magnitude claim from `gfql/performance.rst`** (no replacement figure published): the page read "**q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200×)**". Reconciled rather than merely re-measured — our own side held up (5.0 ms published vs 5.06 ms re-measured), but the 1,004 ms was not canonical Kuzu: the old harness replaced q8/q9 path enumeration with an `OPTIONAL MATCH` degree-product rewrite and fed Kuzu pre-lowercased columns built *outside* the timing loop, while GFQL paid `toLower()` in-query. Re-running that same handicapped configuration reproduces **964.60 ms on the same box**, i.e. a 98.9× handicap applied to the competitor; canonical Kuzu 0.11.3 on a position-balanced, row-validated run at 100k persons / 2.4M FOLLOWS is **9.76 ms vs 5.06 ms = 1.93×**. The directional claim (`engine='polars'` wins q8) survives and is kept; the magnitude is withdrawn.

docs/.rstcheck.cfg

@@ -20,7 +20,9 @@ ignore_roles =
     py:attr,
     py:exc,
     py:obj,
-    py:data
+    py:data,
+    bench,
+    bench-diag
 
 # Ignore Sphinx-specific directives
 ignore_directives =
@@ -43,7 +45,9 @@ ignore_directives =
     glossary,
     productionlist,
     include,
-    graphviz
+    graphviz,
+    bench-provenance,
+    bench-disclosures
 
 # Ignore common informational messages
 ignore_messages = (Hyperlink target "[^"]*" is not referenced\.$)

docs/source/_data/gfql_benchmarks.contract.json

@@ -0,0 +1,102 @@
+{
+  "contract_version": 1,
+  "title": "pyg-bench published docs numbers",
+  "summary": [
+    "The boundary between this repository and the pygraphistry docs. pyg-bench MEASURES and",
+    "PUBLISHES; pygraphistry pretty-prints. Everything a consumer is allowed to assume about",
+    "published/docs-numbers.json is written down here, checked on the producer side before the",
+    "artifact is written, and re-verified independently on the consumer side before it renders.",
+    "Two independent implementations of these rules is the point: a shared library would fail",
+    "the same way on both sides of the boundary."
+  ],
+  "artifact_path": "published/docs-numbers.json",
+  "top_level_required": [
+    "contract_version",
+    "schema_version",
+    "generated_by",
+    "generated_at",
+    "policy",
+    "runs",
+    "cells"
+  ],
+  "policy_required": ["max_age_days", "managed_docs"],
+  "run_required": [
+    "measured_at",
+    "host",
+    "perf_lock_held",
+    "quiet_host",
+    "reps",
+    "pygraphistry_commit",
+    "pyg_bench_commit",
+    "runtime",
+    "dataset",
+    "artifact",
+    "row_validation"
+  ],
+  "cell_required": [
+    "run",
+    "workload",
+    "engine",
+    "value",
+    "unit",
+    "decimals",
+    "status",
+    "comparison_allowed",
+    "board_quotable",
+    "disclosures"
+  ],
+  "cell_status": ["ok", "partial", "adapter_workaround", "result_mismatch"],
+  "cell_unit": ["ms", "s", "x"],
+  "key_pattern": "^[a-z0-9][a-z0-9_]*(\\.[a-z0-9][a-z0-9_]*)+$",
+  "date_pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$",
+  "max_decimals": 4,
+  "invariants": [
+    {
+      "id": "cell_names_a_known_run",
+      "rule": "every cell.run must be a key of runs",
+      "why": "a number whose provenance record is absent is an unprovenanced number"
+    },
+    {
+      "id": "quotable_implies_comparable",
+      "rule": "board_quotable implies comparison_allowed",
+      "why": "a diagnostic-only figure must never reach a board"
+    },
+    {
+      "id": "quotable_implies_ok",
+      "rule": "board_quotable implies status == 'ok'",
+      "why": "partial / adapter_workaround / result_mismatch cells are not results"
+    },
+    {
+      "id": "caveat_travels_with_the_number",
+      "rule": "status != 'ok' or comparison_allowed is false implies disclosures is non-empty",
+      "why": "a bare ratio without its asterisk launders the caveat"
+    },
+    {
+      "id": "ratios_are_comparisons",
+      "rule": "unit 'x' implies comparison_allowed",
+      "why": "a ratio between two numbers that were never established as comparable is not a speedup"
+    },
+    {
+      "id": "value_is_rounded_as_published",
+      "rule": "value == round(value, decimals), value is finite and >= 0, 0 <= decimals <= max_decimals",
+      "why": "the consumer prints value verbatim; rounding must not happen at render time"
+    },
+    {
+      "id": "engine_is_named",
+      "rule": "cell.engine is a non-empty string",
+      "why": "pandas and polars are not interchangeable and the reader must be told which one ran"
+    },
+    {
+      "id": "measurement_is_dated",
+      "rule": "run.measured_at matches date_pattern",
+      "why": "staleness is checked against the MEASUREMENT date, not the export date"
+    }
+  ],
+  "consumer_obligations": [
+    "Refuse to render a key the artifact does not contain — a missing number is a build failure, never blank text.",
+    "Refuse to render a cell whose run.measured_at is older than policy.max_age_days.",
+    "Refuse to render a non-board_quotable cell as a bare number; it may appear only through a role that labels it diagnostic-only.",
+    "Render every referenced cell's disclosures and its run's provenance on the same page.",
+    "Re-verify every invariant above before rendering, with an implementation that does not import this repository."
+  ]
+}