docs: cite the corrected benchmark numbers publicly

Puts the measured circuit-level results in the README as a table readers can
check against `official_benchmark_results.csv`, and records the correction in
the CHANGELOG so the supersession is on the public record rather than only in
git history.

Both findings are stated, including the unflattering one. PyMatching led on
throughput at every distance measured; `qector_blossom` and PyMatching returned
identical logical-failure counts on identical samples at d=3 and d=5 (1891 and
1596 in 100,000). Neither is generalised beyond the cells shown, per
docs/REPRODUCIBILITY_CHECKLIST.md, and the not-quiesced host and
`git_tree_dirty: true` provenance are stated alongside.

All 14 cited rows were machine-checked against the CSV before commit -
throughput, LER and both Wilson bounds - so the README cannot drift from the
artifact it points at.

The reports were re-rendered with `--from-json` rather than re-measured, so the
underlying JSON is byte-identical to the run these figures come from. The only
change is presentation: a ~1000x-slower decoder rendered as "0.00x", which reads
as zero throughput instead of as the accurate number.

Author: qectorlab

CHANGELOG.md

@@ -12,6 +12,33 @@ has been published. `src/*.rs` is `.gitignore`d, so `git log v0.6.9..HEAD` shows
 none of the Rust work recorded here — it is verified by `cargo test` and by
 reading the tree.
 
+### Benchmarks — corrected artifacts, and what they say
+- **`official_benchmark_results.{json,csv,md,pdf}` were regenerated.** The first
+  v0.7.0 versions reproduced the defect that got six pre-v0.7.0 artifacts
+  withdrawn (todo6 A1-03), and additionally stamped `_provenance`'s canned
+  methodology note — which asserts `ler.estimate_ler_circuit_level` scoring and
+  `ler.assert_comparable` validation — into files produced by neither. They
+  reported "LER" as a syndrome-consistency check `(H·ĉ = s)`, which reads 0.000%
+  for every decoder at every distance because it never asks whether the logical
+  observable flipped; they timed QECTOR through a native batch call against
+  PyMatching through a 20-shot Python loop extrapolated to 100,000; and their
+  LER chart plotted hardcoded analytic curves rather than measurements.
+- **What replaced them.** Every row now comes from
+  `ler.estimate_ler_circuit_level` — one circuit, one decomposed DEM, one
+  detector/observable sample set per cell, one `decode_batch` resolver for
+  QECTOR, PyMatching and ldpc alike — and `ler.assert_comparable` gates the rows
+  before writing. Nothing is extrapolated: 63 cells that exceeded the per-cell
+  decode budget are recorded as *not measured*, with their probe rate and
+  projected cost.
+- **The numbers, per-cell and not generalised.** `qector_blossom` and PyMatching
+  2 returned identical logical-failure counts on identical samples at `d = 3`
+  and `d = 5` (1891 and 1596 in 100,000). On throughput PyMatching led at every
+  distance measured, consistent with this project's long-standing note that it
+  leads on plain MWPM. Both statements are in the artifacts and in the README.
+- `scripts/run_custom_comparison_benchmark.py` gained `--from-json`, which
+  re-renders the reports from a stamped artifact without re-measuring, so a
+  presentation fix cannot silently move the numbers underneath a citation.
+
 ### Added — CLI, diagnostics, and ecosystem entry points
 - **`qector` CLI** (`decode` / `bench` / `serve`) and **`qector-doctor`**, a
   15-check environment diagnostic that reports *why* a backend is unavailable

README.md

@@ -423,20 +423,46 @@ quiesced**, and its provenance block records `git_tree_dirty: true`. A further
 63 cells exceeded the per-cell decode budget and are listed as *not measured*,
 carrying their measured probe rate and projected cost — no cell is extrapolated.
 
+Largest shot count measured per cell. Throughput is decode time only; LER is
+per shot with a 95% Wilson interval. Every row is one
+`estimate_ler_circuit_level` call on the same circuit, DEM and samples.
+
+| d | Decoder | Shots | Throughput (dec/s) | LER | 95% CI |
+| ---: | --- | ---: | ---: | ---: | --- |
+| 3 | PyMatching 2 | 100,000 | 2,437,651 | 0.01891 | [0.01808, 0.01977] |
+| 3 | `qector_blossom` | 100,000 | 245,629 | 0.01891 | [0.01808, 0.01977] |
+| 3 | `qector_unionfind` | 100,000 | 825,726 | 0.02210 | [0.02121, 0.02303] |
+| 3 | ldpc BP-OSD | 50,000 | 2,259 | 0.01938 | [0.01821, 0.02063] |
+| 5 | PyMatching 2 | 100,000 | 249,065 | 0.01596 | [0.01520, 0.01676] |
+| 5 | `qector_blossom` | 100,000 | 9,121 | 0.01596 | [0.01520, 0.01676] |
+| 5 | `qector_unionfind` | 100,000 | 112,797 | 0.02645 | [0.02547, 0.02746] |
+| 7 | PyMatching 2 | 100,000 | 75,767 | 0.01220 | [0.01154, 0.01290] |
+| 7 | `qector_unionfind` | 100,000 | 22,643 | 0.02042 | [0.01956, 0.02132] |
+| 9 | PyMatching 2 | 100,000 | 29,244 | 0.00878 | [0.00822, 0.00938] |
+| 9 | `qector_unionfind` | 100,000 | 4,606 | 0.01732 | [0.01653, 0.01815] |
+| 11 | PyMatching 2 | 100,000 | 13,816 | 0.00647 | [0.00599, 0.00699] |
+| 13 | PyMatching 2 | 100,000 | 7,218 | 0.00445 | [0.00406, 0.00488] |
+| 15 | PyMatching 2 | 100,000 | 6,480 | 0.00314 | [0.00281, 0.00351] |
+
+The full 77-row table, including every `qector_blossom` and ldpc cell that fit
+the budget, is in `official_benchmark_results.md`.
+
 Two findings, stated per-cell and not generalised (see
 `docs/REPRODUCIBILITY_CHECKLIST.md`):
 
-- At `d = 3` and `d = 5`, `qector_blossom` and PyMatching 2 returned **the same
-  number of logical failures on the same 100,000 samples** — 1891 and 1596
-  respectively. On this workload the two agree exactly.
-- On throughput, **PyMatching was faster than every QECTOR decoder at every
-  distance measured here**, by roughly 3× at `d = 3` and by two orders of
-  magnitude for `qector_blossom` by `d = 11`. This is consistent with the
-  long-standing note elsewhere in this project that PyMatching leads on plain
-  MWPM; it is not a regression, and it is not something the artifacts hide.
-
-Do not quote these as a marketing comparison. Regenerate on quiesced hardware,
-and state the noise model, before any number here is used in a claim.
+- **Accuracy.** At `d = 3` and `d = 5`, `qector_blossom` and PyMatching 2
+  returned *the same number of logical failures on the same 100,000 samples* —
+  1891 and 1596 respectively, identical to the digit. On this workload the two
+  agree exactly, which is a checkable claim rather than a rounded one.
+- **Throughput.** PyMatching was faster than every QECTOR decoder at every
+  distance measured here — roughly 3× at `d = 3` against `qector_unionfind`,
+  and two orders of magnitude against `qector_blossom` by `d = 11`. That is
+  consistent with the long-standing note elsewhere in this project that
+  PyMatching leads on plain MWPM. It is not a regression, and the artifacts do
+  not hide it.
+
+Neither finding generalises beyond the cells above. Regenerate on quiesced
+hardware, and state the noise model, before any number here is used in a claim.
 
 ### Published, citable evidence
 

official_benchmark_results.md

@@ -16,26 +16,26 @@ Every row below is one `ler.estimate_ler_circuit_level` measurement: the same St
 | PyMatching v2 (C++) | 3 | 1,000 | 13 | 0.01300 | [0.00761, 0.02211] | 1,949,317.9 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 3 | 1,000 | 13 | 0.01300 | [0.00761, 0.02211] | 154,268.6 | 0.08x |
 | QECTOR Union-Find (CPU) | 3 | 1,000 | 16 | 0.01600 | [0.00987, 0.02583] | 222,321.0 | 0.11x |
-| ldpc BP-OSD | 3 | 1,000 | 12 | 0.01200 | [0.00688, 0.02086] | 2,266.7 | 0.00x |
+| ldpc BP-OSD | 3 | 1,000 | 12 | 0.01200 | [0.00688, 0.02086] | 2,266.7 | 0.001x |
 | PyMatching v2 (C++) | 3 | 5,000 | 84 | 0.01680 | [0.01359, 0.02075] | 1,865,810.9 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 3 | 5,000 | 84 | 0.01680 | [0.01359, 0.02075] | 236,337.3 | 0.13x |
 | QECTOR Union-Find (CPU) | 3 | 5,000 | 105 | 0.02100 | [0.01738, 0.02536] | 481,023.6 | 0.26x |
-| ldpc BP-OSD | 3 | 5,000 | 87 | 0.01740 | [0.01413, 0.02141] | 2,285.6 | 0.00x |
+| ldpc BP-OSD | 3 | 5,000 | 87 | 0.01740 | [0.01413, 0.02141] | 2,285.6 | 0.001x |
 | PyMatching v2 (C++) | 3 | 10,000 | 202 | 0.02020 | [0.01762, 0.02315] | 1,848,941.5 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 3 | 10,000 | 202 | 0.02020 | [0.01762, 0.02315] | 248,615.8 | 0.13x |
 | QECTOR Union-Find (CPU) | 3 | 10,000 | 233 | 0.02330 | [0.02052, 0.02644] | 571,931.9 | 0.31x |
-| ldpc BP-OSD | 3 | 10,000 | 199 | 0.01990 | [0.01734, 0.02283] | 2,329.7 | 0.00x |
+| ldpc BP-OSD | 3 | 10,000 | 199 | 0.01990 | [0.01734, 0.02283] | 2,329.7 | 0.001x |
 | PyMatching v2 (C++) | 3 | 50,000 | 960 | 0.01920 | [0.01803, 0.02044] | 2,234,836.6 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 3 | 50,000 | 960 | 0.01920 | [0.01803, 0.02044] | 250,396.0 | 0.11x |
 | QECTOR Union-Find (CPU) | 3 | 50,000 | 1128 | 0.02256 | [0.02129, 0.02390] | 812,805.9 | 0.36x |
-| ldpc BP-OSD | 3 | 50,000 | 969 | 0.01938 | [0.01821, 0.02063] | 2,259.1 | 0.00x |
+| ldpc BP-OSD | 3 | 50,000 | 969 | 0.01938 | [0.01821, 0.02063] | 2,259.1 | 0.001x |
 | PyMatching v2 (C++) | 3 | 100,000 | 1891 | 0.01891 | [0.01808, 0.01977] | 2,437,651.0 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 3 | 100,000 | 1891 | 0.01891 | [0.01808, 0.01977] | 245,628.9 | 0.10x |
 | QECTOR Union-Find (CPU) | 3 | 100,000 | 2210 | 0.02210 | [0.02121, 0.02303] | 825,725.6 | 0.34x |
 | PyMatching v2 (C++) | 5 | 1,000 | 19 | 0.01900 | [0.01220, 0.02948] | 266,410.9 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 5 | 1,000 | 19 | 0.01900 | [0.01220, 0.02948] | 8,690.2 | 0.03x |
 | QECTOR Union-Find (CPU) | 5 | 1,000 | 26 | 0.02600 | [0.01780, 0.03782] | 79,887.5 | 0.30x |
-| ldpc BP-OSD | 5 | 1,000 | 21 | 0.02100 | [0.01378, 0.03189] | 87.8 | — |
+| ldpc BP-OSD | 5 | 1,000 | 21 | 0.02100 | [0.01378, 0.03189] | 87.8 | 0x |
 | PyMatching v2 (C++) | 5 | 5,000 | 73 | 0.01460 | [0.01163, 0.01832] | 286,109.9 | 1.00x |
 | QECTOR Sparse Blossom (CPU) | 5 | 5,000 | 73 | 0.01460 | [0.01163, 0.01832] | 9,018.3 | 0.03x |
 | QECTOR Union-Find (CPU) | 5 | 5,000 | 148 | 0.02960 | [0.02525, 0.03467] | 91,362.6 | 0.32x |
@@ -73,7 +73,7 @@ Every row below is one `ler.estimate_ler_circuit_level` measurement: the same St
 | PyMatching v2 (C++) | 9 | 100,000 | 878 | 0.00878 | [0.00822, 0.00938] | 29,244.1 | 1.00x |
 | QECTOR Union-Find (CPU) | 9 | 100,000 | 1732 | 0.01732 | [0.01653, 0.01815] | 4,606.0 | 0.16x |
 | PyMatching v2 (C++) | 11 | 1,000 | 7 | 0.00700 | [0.00339, 0.01438] | 11,490.2 | 1.00x |
-| QECTOR Sparse Blossom (CPU) | 11 | 1,000 | 7 | 0.00700 | [0.00339, 0.01438] | 84.2 | 0.01x |
+| QECTOR Sparse Blossom (CPU) | 11 | 1,000 | 7 | 0.00700 | [0.00339, 0.01438] | 84.2 | 0.007x |
 | QECTOR Union-Find (CPU) | 11 | 1,000 | 13 | 0.01300 | [0.00761, 0.02211] | 367.3 | 0.03x |
 | PyMatching v2 (C++) | 11 | 5,000 | 24 | 0.00480 | [0.00323, 0.00713] | 12,516.9 | 1.00x |
 | PyMatching v2 (C++) | 11 | 10,000 | 77 | 0.00770 | [0.00617, 0.00961] | 13,693.2 | 1.00x |

official_benchmark_results.pdf

@@ -84,6 +84,18 @@
 PROBE_SHOTS = 256
 
 
+def _fmt_speedup(sp) -> str:
+    """Format a throughput ratio without collapsing small ones to '0.00x'.
+
+    A decoder ~1000x slower than the reference has a ratio of 0.001; printed with
+    two decimals that becomes '0.00x', which reads as zero throughput rather than
+    as the (accurate, unflattering) number it is.
+    """
+    if sp is None:
+        return "—"
+    return f"{sp:.2f}x" if sp >= 0.01 else f"{sp:.3g}x"
+
+
 def run_grid(distances, shot_list, p, seed, budget_s):
     """Measure every feasible (distance, decoder, shots) cell. Never extrapolate."""
     rows: list[dict] = []
@@ -236,7 +248,7 @@ def export_markdown(path: Path, rows, skipped, prov, p, budget_s):
         md.append(
             f"| {r['decoder']} | {r['distance']} | {r['shots']:,} | {r['errors']} | "
             f"{r['ler']:.5f} | [{r['ci95_lo']:.5f}, {r['ci95_hi']:.5f}] | "
-            f"{r['decodes_per_s']:,.1f} | {f'{sp:.2f}x' if sp else '—'} |"
+            f"{r['decodes_per_s']:,.1f} | {_fmt_speedup(sp)} |"
         )
 
     if skipped:
@@ -471,7 +483,7 @@ def export_pdf(path: Path, rows, skipped, prov, charts, out_dir: Path, p: float)
             Paragraph(f"{r['ler']:.5f}", cellb),
             Paragraph(f"[{r['ci95_lo']:.4f}, {r['ci95_hi']:.4f}]", cell),
             Paragraph(f"{r['decodes_per_s']:,.0f}", cellb),
-            Paragraph(f"{sp:.2f}x" if sp else "—", cell),
+            Paragraph(_fmt_speedup(sp), cell),
         ])
     t = Table(data, colWidths=[138, 22, 48, 38, 52, 94, 60, 40], repeatRows=1)
     t.setStyle(TableStyle([
@@ -525,6 +537,12 @@ def main():
              "skipped rather than extrapolated.",
     )
     ap.add_argument("--out-dir", default=".")
+    ap.add_argument(
+        "--from-json", metavar="PATH",
+        help="re-render CSV/Markdown/PDF/charts from an existing stamped artifact "
+             "instead of measuring. Use for presentation fixes so the numbers stay "
+             "byte-identical to what has already been cited.",
+    )
     args = ap.parse_args()
 
     distances = [int(x) for x in args.distances.split(",") if x.strip()]
@@ -536,6 +554,33 @@ def main():
     print(f"  distances={distances}  shots={shot_list}  p={args.p}  seed={args.seed}")
     print(f"  per-cell decode budget: {args.budget_seconds:g}s (no extrapolation)")
 
+    if args.from_json:
+        # Re-render the reports from an existing stamped artifact without
+        # re-measuring. A presentation fix must not silently change the numbers
+        # underneath it: re-running would produce a different (equally valid)
+        # sample, and any figure already quoted elsewhere would quietly stop
+        # matching the artifact it cites.
+        src = Path(args.from_json).resolve()
+        rows, prov = _provenance.load_artifact(src)
+        if prov is None:
+            print(f"{src} carries no provenance block - refusing to re-render it.")
+            return 1
+        skipped = prov.get("parameters", {}).get("skipped_cells", [])
+        p = prov.get("parameters", {}).get("physical_error_rate", args.p)
+        budget = prov.get("parameters", {}).get("per_cell_budget_seconds", args.budget_seconds)
+        assert_comparable(rows)
+        print(f"Re-rendering {len(rows)} measured / {len(skipped)} skipped cells from {src.name}")
+        export_csv(out_dir / "official_benchmark_results.csv", rows)
+        export_markdown(
+            out_dir / "official_benchmark_results.md", rows, skipped, prov, p, budget
+        )
+        charts = generate_charts(rows, out_dir, p)
+        export_pdf(
+            out_dir / "official_benchmark_results.pdf", rows, skipped, prov, charts, out_dir, p
+        )
+        print("\nDone - measurements untouched, reports re-rendered.")
+        return 0
+
     t0 = time.perf_counter()
     rows, skipped = run_grid(distances, shot_list, args.p, args.seed, args.budget_seconds)
     elapsed = time.perf_counter() - t0