Skip to content

Commit e300b42

Browse files
committed
fix(bench): regenerate the official artifacts from real circuit-level measurements
The v0.7.0 artifacts reproduced the exact defect that got six pre-v0.7.0 ones withdrawn (todo6 A1-03), and this time stamped _provenance's canned METHODOLOGY_NOTE - which asserts estimate_ler_circuit_level scoring and assert_comparable validation - into a file produced by neither: * "LER" was `(H @ correction) % 2 == syndrome`, a syndrome-consistency check. It asks whether the decoder returned *a* valid correction, never whether the logical observable flipped, and read 0.000% for every decoder at every distance - the shape of a tautology, not a measurement. * QECTOR was timed through a native batch_decode over the full shot count while PyMatching went through a Python `[dec.decode(s) for s in ...]` loop over as few as 20 shots, linearly extrapolated to 100,000 and written into `elapsed_sec` as if measured. The resulting "55x-108x faster than PyMatching" was mostly Python per-call overhead. * chart_official_ler.png contained no measurements at all: hardcoded analytic curves `0.14 * (0.03/0.10)**((d+1)/2)`, constants picked per decoder by hand. The generator now routes every decoder - QECTOR's backends, PyMatching and ldpc alike - through ler.estimate_ler_circuit_level: one circuit, one decomposed DEM, one detector/observable sample set per cell, one decode_batch resolver, scored against the circuit's own logical observables. ler.assert_comparable gates the rows before writing. Nothing is extrapolated; cells over the per-cell decode budget are recorded as skipped with their measured probe rate and projected cost, and are absent from the charts. Measured, 77 cells (p=0.005, seed=1), largest shot count per cell: d=3 PyMatching 2,437,651/s LER 0.01891 | QECTOR Blossom 245,629/s 0.01891 d=5 PyMatching 249,065/s LER 0.01596 | QECTOR Blossom 9,121/s 0.01596 d=11 PyMatching 13,816/s LER 0.00647 | QECTOR Blossom 84/s 0.00700 QECTOR's Sparse Blossom reproduces PyMatching's decoding decisions exactly at d=3 and d=5 (1891 and 1596 errors in 100,000, identical) - a real and checkable accuracy result. On speed PyMatching leads at every distance measured. Both statements are now in the artifacts.
1 parent b0f5456 commit e300b42

8 files changed

Lines changed: 2605 additions & 4848 deletions

chart_official_batch_scaling.png

-88.6 KB
Loading

chart_official_ler.png

-53 KB
Loading

chart_official_throughput.png

-46 KB
Loading

official_benchmark_results.csv

Lines changed: 78 additions & 265 deletions
Large diffs are not rendered by default.

official_benchmark_results.json

Lines changed: 1840 additions & 3688 deletions
Large diffs are not rendered by default.

official_benchmark_results.md

Lines changed: 162 additions & 280 deletions
Large diffs are not rendered by default.

official_benchmark_results.pdf

-20.9 KB
Binary file not shown.

scripts/run_custom_comparison_benchmark.py

Lines changed: 525 additions & 615 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)