Skip to content

Commit 61158de

Browse files
committed
docs: describe the circuit-level run that now exists, and its limits
The benchmark section still said the replacement publication run "has not been performed", while `official_benchmark_results.*` sat committed in the repo root. A README that contradicts the artifacts beside it is its own defect. Record what the run actually is - 77 cells, p=0.005, d in {3..15}, shots to 100,000 - and, just as importantly, what it is not: taken on a workstation that was not quiesced, with `git_tree_dirty: true` in its own provenance block, and 63 further cells over the per-cell decode budget that are listed as *not measured* rather than extrapolated. Both findings are stated per-cell rather than generalised, per docs/REPRODUCIBILITY_CHECKLIST.md, and that includes the unflattering one: PyMatching was faster than every QECTOR decoder at every distance measured. That is consistent with this project's own long-standing note that PyMatching leads on plain MWPM. The flattering one is equally concrete - qector_blossom and PyMatching returned an identical logical-failure count on identical samples at d=3 and d=5. Also points at both generators and spells out the shared pipeline, so the next reader does not have to take the methodology note on trust.
1 parent e300b42 commit 61158de

1 file changed

Lines changed: 36 additions & 6 deletions

File tree

README.md

Lines changed: 36 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -396,18 +396,48 @@ method that produced them cannot support the claim they were used to make.
396396
cross-model comparisons through `assert_comparable`, so this class of error
397397
cannot recur silently.
398398

399-
**What replaces them:** `scripts/regenerate_benchmark_artifacts.py` drives every
400-
decoder through one circuit-level pipeline and stamps the result with its
401-
methodology, git commit, tree-dirty flag, parameters, and dependency versions.
402-
The full publication run — 100,000 shots × 4 distances × 4 decoders, on a
403-
quiesced machine — **has not been performed**, so there is no replacement table
404-
to print yet. Run it yourself:
399+
**What replaces them:** `scripts/regenerate_benchmark_artifacts.py` and
400+
`scripts/run_custom_comparison_benchmark.py` both drive every decoder through
401+
one circuit-level pipeline — `ler.estimate_ler_circuit_level`, one Stim circuit,
402+
one decomposed DEM, one detector/observable sample set per cell, one
403+
`decode_batch` resolver, scored against the circuit's own logical observables —
404+
and stamp the result with its methodology, git commit, tree-dirty flag,
405+
parameters and dependency versions. `ler.assert_comparable` gates the rows
406+
before they are written.
405407

406408
```bash
407409
python scripts/regenerate_benchmark_artifacts.py --dry-run # show the plan
408410
python scripts/regenerate_benchmark_artifacts.py --yes # ~1.6M decodes
411+
412+
# QECTOR vs PyMatching vs ldpc, with a per-cell time budget:
413+
python scripts/run_custom_comparison_benchmark.py \
414+
--distances 3,5,7,9,11,13,15 --shots 1000,5000,10000,50000,100000 --p 0.005
409415
```
410416

417+
### Indicative circuit-level run (not a publication run)
418+
419+
`official_benchmark_results.{json,csv,md,pdf}` in the repo root hold a 77-cell
420+
run at `p = 0.005`, `seed = 1`, `d ∈ {3..15}`, shots up to 100,000. Read it as
421+
indicative only: it was taken on a **developer workstation that was not
422+
quiesced**, and its provenance block records `git_tree_dirty: true`. A further
423+
63 cells exceeded the per-cell decode budget and are listed as *not measured*,
424+
carrying their measured probe rate and projected cost — no cell is extrapolated.
425+
426+
Two findings, stated per-cell and not generalised (see
427+
`docs/REPRODUCIBILITY_CHECKLIST.md`):
428+
429+
- At `d = 3` and `d = 5`, `qector_blossom` and PyMatching 2 returned **the same
430+
number of logical failures on the same 100,000 samples** — 1891 and 1596
431+
respectively. On this workload the two agree exactly.
432+
- On throughput, **PyMatching was faster than every QECTOR decoder at every
433+
distance measured here**, by roughly 3× at `d = 3` and by two orders of
434+
magnitude for `qector_blossom` by `d = 11`. This is consistent with the
435+
long-standing note elsewhere in this project that PyMatching leads on plain
436+
MWPM; it is not a regression, and it is not something the artifacts hide.
437+
438+
Do not quote these as a marketing comparison. Regenerate on quiesced hardware,
439+
and state the noise model, before any number here is used in a claim.
440+
411441
### Published, citable evidence
412442

413443
Until that run lands, the reproducible accuracy and throughput evidence for this

0 commit comments

Comments
 (0)