Skip to content

Commit 0186c8b

Browse files
author
Guillaume Lessard
committed
bench: replace fabricated circuit-level comparison with verified v0.7.0 benchmark set (54 cells, 0 unfaithful)
1 parent 2ec8b88 commit 0186c8b

11 files changed

Lines changed: 1260 additions & 5156 deletions

‎README.md‎

Lines changed: 18 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -410,42 +410,30 @@ python scripts/run_custom_comparison_benchmark.py \
410410
--distances 3,5,7,9,11,13,15 --shots 1000,5000,10000,50000,100000 --p 0.005
411411
```
412412

413-
### Indicative circuit-level run (not a publication run)
413+
### Verified v0.7.0 benchmark set
414414

415-
`official_benchmark_results.{json,csv,md,pdf}` in the repo root hold a 77-cell
416-
run at `p = 0.005`, `seed = 1`, `d ∈ {3..15}`, shots up to 100,000. Read it as
417-
indicative only: it was taken on a **developer workstation that was not
418-
quiesced**, and its provenance block records `git_tree_dirty: true`. A further
419-
93 cells exceeded the per-cell decode budget and are listed as *not measured*,
420-
carrying their measured probe rate and projected cost — no cell is extrapolated.
415+
`official_benchmark_results.{json,csv,md,pdf}` in the repo root hold the verified
416+
v0.7.0 benchmark set: **54 measured cells, zero unfaithful corrections**, taken
417+
through the package MCP server (stdio, JSON-RPC 2.0) on `2026-08-02T05:59:13Z`
418+
(Linux glibc 2.35, Python 3.12.13).
421419

422-
Largest shot count measured per cell. Throughput is decode time only; LER is
423-
per shot with a 95% Wilson interval. Every row is one
424-
`estimate_ler_circuit_level` call on the same circuit, DEM and samples.
420+
- **Peak throughput:** 11,540,387 shots/s (FastUnionFind, 5-qubit repetition code, 8,000 samples)
421+
- **54/54 benchmark points** with zero unfaithful corrections (repetition n=5–65, ring n=16–48; unionfind, fastunionfind, blossom, sparseblossom, bposd, auto)
422+
- **42/42 syndrome-faithfulness cases** passed; **13 MCP tools** operational
423+
- **No speedup multiplier is claimed.** Apple-to-apple vs PyMatching (see `benchmarks/v0.7.0/VERIFIED_APPLE_TO_APPLE_REPORT.pdf`): comparable, with PyMatching often slightly ahead on the synchronized batch.
425424

426-
> **Benchmark figures are not published for this release.**
425+
The previous `official_benchmark_results.*` (2026-08-01 circuit-level comparison,
426+
generated from a dirty working tree at `b436f04e`, carrying per-row
427+
speedup-vs-PyMatching multipliers) is **withdrawn** — it is not part of the
428+
verified release and must not be cited.
429+
430+
> **Benchmark figures are not published beyond the verified set for this release.**
427431
> Decoder throughput and logical error rate depend on your hardware, code family,
428432
> distance and noise model, so any table printed here would describe a machine that
429433
> is not yours. The benchmark harness ships with the package and writes JSON carrying
430-
> its own environment and parameter block — run it on your target hardware and compare
431-
> decoders under the conditions you actually care about.
432-
433-
434-
The complete 187-row table — every shot count, together with the 93 cells that
435-
exceeded the per-cell decode budget and are therefore recorded as *not measured*
436-
rather than estimated — is in `official_benchmark_results.md`.
437-
438-
Findings apply to the cells listed above and are not generalised beyond them
439-
(see `docs/REPRODUCIBILITY_CHECKLIST.md`).
440-
441-
**Accuracy parity with PyMatching.** At the distances where both decoders were
442-
measured on identical sample sets, `qector_blossom` and PyMatching 2 recorded
443-
identical logical-failure counts. The counts live in the artifact; they are not
444-
restated here because this release publishes no performance figures.
445-
446-
**Throughput.** PyMatching 2 led at every distance measured in that run. This is
447-
consistent with the position recorded elsewhere in this project that PyMatching
448-
leads on plain MWPM.
434+
> its own environment and parameter block — run it on your target hardware
435+
> (`qector benchmark --verify` or `python -m qector.validate`) and compare decoders
436+
> under the conditions you actually care about.
449437
450438
**GPU accuracy depends on whether matching weights are supplied.**
451439
`CUDABatchDecoder` and `OpenCLBatchDecoder` accept an optional `edge_weights`

‎benchmarks/v0.7.0/REPORT.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# QECTOR MCP Benchmark Report
2+
3+
Generated (UTC): 2026-08-02T05:59:13.294039+00:00
4+
Server: `{'name': 'qector-decoder-v3', 'version': '0.7.0'}`
5+
Host: Linux-6.6.122+-x86_64-with-glibc2.35 | Python 3.12.13
6+
7+
## Tools
8+
9+
- `decode_syndrome`
10+
- `batch_decode`
11+
- `decode_hyperedge`
12+
- `decode_syndrome_blossom`
13+
- `batch_decode_blossom`
14+
- `decode_syndrome_cascade`
15+
- `benchmark_decoder`
16+
- `run_ler_benchmark`
17+
- `get_decoder_info`
18+
- `get_backend_health`
19+
- `clear_decoder_cache`
20+
- `get_server_env`
21+
- `recommend_decoder`
22+
23+
## Benchmarks (repetition / ring)
24+
25+
| code | n | decoder | samples | throughput | mean_us | unfaithful |
26+
|------|---|---------|---------|------------|---------|------------|
27+
| repetition | 5 | unionfind | 8000 | 11462979 | 0.087 | 0 |
28+
| repetition | 5 | fastunionfind | 8000 | 11540387 | 0.087 | 0 |
29+
| repetition | 5 | blossom | 8000 | 8262646 | 0.121 | 0 |
30+
| repetition | 5 | sparseblossom | 8000 | 5236849 | 0.191 | 0 |
31+
| repetition | 5 | bposd | 8000 | 314525 | 3.179 | 0 |
32+
| repetition | 5 | auto | 8000 | 8261043 | 0.121 | 0 |
33+
| repetition | 9 | unionfind | 4000 | 9877914 | 0.101 | 0 |
34+
| repetition | 9 | fastunionfind | 4000 | 10000813 | 0.1 | 0 |
35+
| repetition | 9 | blossom | 4000 | 5070210 | 0.197 | 0 |
36+
| repetition | 9 | sparseblossom | 4000 | 1980725 | 0.505 | 0 |
37+
| repetition | 9 | bposd | 4000 | 94640 | 10.566 | 0 |
38+
| repetition | 9 | auto | 4000 | 3177478 | 0.315 | 0 |
39+
| repetition | 17 | unionfind | 2500 | 4242251 | 0.236 | 0 |
40+
| repetition | 17 | fastunionfind | 2500 | 4472615 | 0.224 | 0 |
41+
| repetition | 17 | blossom | 2500 | 1695841 | 0.59 | 0 |
42+
| repetition | 17 | sparseblossom | 2500 | 1136156 | 0.88 | 0 |
43+
| repetition | 17 | bposd | 2500 | 79299 | 12.61 | 0 |
44+
| repetition | 17 | auto | 2500 | 3357187 | 0.298 | 0 |
45+
| repetition | 25 | unionfind | 1500 | 6178188 | 0.162 | 0 |
46+
| repetition | 25 | fastunionfind | 1500 | 6300252 | 0.159 | 0 |
47+
| repetition | 25 | blossom | 1500 | 2159946 | 0.463 | 0 |
48+
| repetition | 25 | sparseblossom | 1500 | 1521550 | 0.657 | 0 |
49+
| repetition | 25 | bposd | 1500 | 52477 | 19.056 | 0 |
50+
| repetition | 25 | auto | 1500 | 2151542 | 0.465 | 0 |
51+
| repetition | 33 | unionfind | 1000 | 4932787 | 0.203 | 0 |
52+
| repetition | 33 | fastunionfind | 1000 | 4953127 | 0.202 | 0 |
53+
| repetition | 33 | blossom | 1000 | 1536840 | 0.651 | 0 |
54+
| repetition | 33 | sparseblossom | 1000 | 1165277 | 0.858 | 0 |
55+
| repetition | 33 | bposd | 1000 | 45271 | 22.089 | 0 |
56+
| repetition | 33 | auto | 1000 | 1292897 | 0.773 | 0 |
57+
| repetition | 49 | unionfind | 600 | 3346108 | 0.299 | 0 |
58+
| repetition | 49 | fastunionfind | 600 | 3512604 | 0.285 | 0 |
59+
| repetition | 49 | blossom | 600 | 1010423 | 0.99 | 0 |
60+
| repetition | 49 | sparseblossom | 600 | 774424 | 1.291 | 0 |
61+
| repetition | 49 | bposd | 600 | 27953 | 35.773 | 0 |
62+
| repetition | 49 | auto | 600 | 985420 | 1.015 | 0 |
63+
| repetition | 65 | unionfind | 400 | 2900839 | 0.345 | 0 |
64+
| repetition | 65 | fastunionfind | 400 | 2754975 | 0.363 | 0 |
65+
| repetition | 65 | blossom | 400 | 172643 | 5.792 | 0 |
66+
| repetition | 65 | sparseblossom | 400 | 253455 | 3.945 | 0 |
67+
| repetition | 65 | bposd | 400 | 19176 | 52.148 | 0 |
68+
| repetition | 65 | auto | 400 | 251774 | 3.972 | 0 |
69+
| ring | 16 | unionfind | 3000 | 6791172 | 0.147 | 0 |
70+
| ring | 16 | fastunionfind | 3000 | 7687193 | 0.13 | 0 |
71+
| ring | 16 | blossom | 3000 | 3940735 | 0.254 | 0 |
72+
| ring | 16 | auto | 3000 | 4104082 | 0.244 | 0 |
73+
| ring | 32 | unionfind | 1500 | 5100896 | 0.196 | 0 |
74+
| ring | 32 | fastunionfind | 1500 | 5183528 | 0.193 | 0 |
75+
| ring | 32 | blossom | 1500 | 1877142 | 0.533 | 0 |
76+
| ring | 32 | auto | 1500 | 1883361 | 0.531 | 0 |
77+
| ring | 48 | unionfind | 800 | 3377505 | 0.296 | 0 |
78+
| ring | 48 | fastunionfind | 800 | 3448535 | 0.29 | 0 |
79+
| ring | 48 | blossom | 800 | 1064594 | 0.939 | 0 |
80+
| ring | 48 | auto | 800 | 1038234 | 0.963 | 0 |
81+
82+
## Summary
83+
84+
```json
85+
{
86+
"package": "qector-decoder-v3",
87+
"version": "0.7.0",
88+
"transport": "MCP stdio JSON-RPC 2.0",
89+
"timestamp_utc": "2026-08-02T05:59:13.294039+00:00",
90+
"tools_count": 13,
91+
"correctness_cases": 42,
92+
"correctness_faithful_all": true,
93+
"benchmark_points": 54,
94+
"zero_unfaithful_in_benchmarks": true,
95+
"peak_throughput_shots_per_s": 11540387
96+
}
97+
```

‎benchmarks/v0.7.0/benchmarks.csv‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
code,n_qubits,decoder,n_samples,throughput_shots_per_s,latency_mean_us,latency_p50_us,latency_p99_us,latency_min_us,latency_max_us,n_unfaithful,unfaithful_rate,phys_error,decoder_type_reported
2+
repetition,5,unionfind,8000,11462979,0.087,0.072,0.38,0.069,0.851,0,0.0,0.01,FastUnionFind
3+
repetition,5,fastunionfind,8000,11540387,0.087,0.072,0.377,0.071,1.335,0,0.0,0.01,FastUnionFind
4+
repetition,5,blossom,8000,8262646,0.121,0.064,1.179,0.063,3.304,0,0.0,0.01,Blossom
5+
repetition,5,sparseblossom,8000,5236849,0.191,0.072,2.984,0.069,6.706,0,0.0,0.01,SparseBlossom
6+
repetition,5,bposd,8000,314525,3.179,2.994,4.806,2.823,21.706,0,0.0,0.01,BPOSD
7+
repetition,5,auto,8000,8261043,0.121,0.064,1.176,0.063,2.543,0,0.0,0.01,Blossom
8+
repetition,9,unionfind,4000,9877914,0.101,0.074,0.428,0.073,1.145,0,0.0,0.01,FastUnionFind
9+
repetition,9,fastunionfind,4000,10000813,0.1,0.074,0.402,0.072,1.072,0,0.0,0.01,FastUnionFind
10+
repetition,9,blossom,4000,5070210,0.197,0.066,2.233,0.064,11.488,0,0.0,0.01,Blossom
11+
repetition,9,sparseblossom,4000,1980725,0.505,0.106,6.199,0.075,38.112,0,0.0,0.01,SparseBlossom
12+
repetition,9,bposd,4000,94640,10.566,9.449,41.347,5.914,65.997,0,0.0,0.01,BPOSD
13+
repetition,9,auto,4000,3177478,0.315,0.113,2.502,0.083,30.197,0,0.0,0.01,Blossom
14+
repetition,17,unionfind,2500,4242251,0.236,0.14,1.033,0.095,3.734,0,0.0,0.01,FastUnionFind
15+
repetition,17,fastunionfind,2500,4472615,0.224,0.134,1.034,0.089,1.564,0,0.0,0.01,FastUnionFind
16+
repetition,17,blossom,2500,1695841,0.59,0.131,4.838,0.069,59.882,0,0.0,0.01,Blossom
17+
repetition,17,sparseblossom,2500,1136156,0.88,0.125,14.435,0.085,37.448,0,0.0,0.01,SparseBlossom
18+
repetition,17,bposd,2500,79299,12.61,11.055,25.149,10.468,100.646,0,0.0,0.01,BPOSD
19+
repetition,17,auto,2500,3357187,0.298,0.072,2.603,0.069,21.525,0,0.0,0.01,Blossom
20+
repetition,25,unionfind,1500,6178188,0.162,0.082,0.73,0.079,2.449,0,0.0,0.01,FastUnionFind
21+
repetition,25,fastunionfind,1500,6300252,0.159,0.08,0.693,0.078,2.207,0,0.0,0.01,FastUnionFind
22+
repetition,25,blossom,1500,2159946,0.463,0.079,4.043,0.076,16.747,0,0.0,0.01,Blossom
23+
repetition,25,sparseblossom,1500,1521550,0.657,0.082,8.828,0.081,37.548,0,0.0,0.01,SparseBlossom
24+
repetition,25,bposd,1500,52477,19.056,16.446,33.281,15.936,70.269,0,0.0,0.01,BPOSD
25+
repetition,25,auto,1500,2151542,0.465,0.079,4.199,0.076,18.529,0,0.0,0.01,Blossom
26+
repetition,33,unionfind,1000,4932787,0.203,0.086,0.921,0.082,2.882,0,0.0,0.01,FastUnionFind
27+
repetition,33,fastunionfind,1000,4953127,0.202,0.086,0.914,0.082,2.759,0,0.0,0.01,FastUnionFind
28+
repetition,33,blossom,1000,1536840,0.651,0.087,5.986,0.082,20.099,0,0.0,0.01,Blossom
29+
repetition,33,sparseblossom,1000,1165277,0.858,0.087,9.543,0.085,31.342,0,0.0,0.01,SparseBlossom
30+
repetition,33,bposd,1000,45271,22.089,20.651,33.093,19.567,40.942,0,0.0,0.01,BPOSD
31+
repetition,33,auto,1000,1292897,0.773,0.088,6.443,0.083,19.6,0,0.0,0.01,Blossom
32+
repetition,49,unionfind,600,3346108,0.299,0.108,1.697,0.094,3.312,0,0.0,0.01,FastUnionFind
33+
repetition,49,fastunionfind,600,3512604,0.285,0.104,1.123,0.094,4.278,0,0.0,0.01,FastUnionFind
34+
repetition,49,blossom,600,1010423,0.99,0.102,7.752,0.095,19.206,0,0.0,0.01,Blossom
35+
repetition,49,sparseblossom,600,774424,1.291,0.1,17.163,0.094,33.222,0,0.0,0.01,SparseBlossom
36+
repetition,49,bposd,600,27953,35.773,31.803,60.328,29.717,212.207,0,0.0,0.01,BPOSD
37+
repetition,49,auto,600,985420,1.015,0.103,8.189,0.097,19.668,0,0.0,0.01,Blossom
38+
repetition,65,unionfind,400,2900839,0.345,0.17,1.444,0.108,5.004,0,0.0,0.01,FastUnionFind
39+
repetition,65,fastunionfind,400,2754975,0.363,0.168,1.284,0.108,4.867,0,0.0,0.01,FastUnionFind
40+
repetition,65,blossom,400,172643,5.792,0.23,106.096,0.11,205.109,0,0.0,0.01,Blossom
41+
repetition,65,sparseblossom,400,253455,3.945,0.202,42.842,0.134,68.018,0,0.0,0.01,SparseBlossom
42+
repetition,65,bposd,400,19176,52.148,46.936,82.35,39.011,251.277,0,0.0,0.01,BPOSD
43+
repetition,65,auto,400,251774,3.972,0.231,41.008,0.11,59.845,0,0.0,0.01,Blossom
44+
ring,16,unionfind,3000,6791172,0.147,0.086,0.727,0.076,3.181,0,0.0,0.01,FastUnionFind
45+
ring,16,fastunionfind,3000,7687193,0.13,0.077,0.519,0.076,13.039,0,0.0,0.01,FastUnionFind
46+
ring,16,blossom,3000,3940735,0.254,0.072,2.524,0.069,10.362,0,0.0,0.01,Blossom
47+
ring,16,auto,3000,4104082,0.244,0.071,1.717,0.069,7.144,0,0.0,0.01,Blossom
48+
ring,32,unionfind,1500,5100896,0.196,0.086,0.781,0.082,4.76,0,0.0,0.01,FastUnionFind
49+
ring,32,fastunionfind,1500,5183528,0.193,0.085,0.755,0.082,3.307,0,0.0,0.01,FastUnionFind
50+
ring,32,blossom,1500,1877142,0.533,0.087,5.391,0.083,11.289,0,0.0,0.01,Blossom
51+
ring,32,auto,1500,1883361,0.531,0.087,5.342,0.083,10.726,0,0.0,0.01,Blossom
52+
ring,48,unionfind,800,3377505,0.296,0.105,1.246,0.094,4.546,0,0.0,0.01,FastUnionFind
53+
ring,48,fastunionfind,800,3448535,0.29,0.103,1.307,0.094,4.905,0,0.0,0.01,FastUnionFind
54+
ring,48,blossom,800,1064594,0.939,0.103,9.887,0.096,16.447,0,0.0,0.01,Blossom
55+
ring,48,auto,800,1038234,0.963,0.103,12.09,0.096,14.71,0,0.0,0.01,Blossom

‎benchmarks/v0.7.0/summary.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"package": "qector-decoder-v3",
3+
"version": "0.7.0",
4+
"transport": "MCP stdio JSON-RPC 2.0",
5+
"timestamp_utc": "2026-08-02T05:59:13.294039+00:00",
6+
"tools_count": 13,
7+
"correctness_cases": 42,
8+
"correctness_faithful_all": true,
9+
"benchmark_points": 54,
10+
"zero_unfaithful_in_benchmarks": true,
11+
"peak_throughput_shots_per_s": 11540387
12+
}

‎chart_official_batch_scaling.png‎

-116 KB
Binary file not shown.

‎chart_official_ler.png‎

-159 KB
Binary file not shown.

‎chart_official_throughput.png‎

-226 KB
Binary file not shown.

0 commit comments

Comments
 (0)