ci(conformance): gate INDEX_all_formats.json against the packs it summarises - #1578
Open
gHashTag wants to merge 1 commit into
Open
ci(conformance): gate INDEX_all_formats.json against the packs it summarises#1578gHashTag wants to merge 1 commit into
gHashTag wants to merge 1 commit into
Conversation
…marises The index is what consumers read and what Paper B recomputes its coverage from, and nothing checked that it still agreed with the packs underneath it. That gap already cost a regression. gen_all_formats.py hardcoded the tier for the six wide GoldenFloat rungs -- correct when written, but the rungs later acquired independent second witnesses and were promoted in the pack files. Re-running the generator rewrote the index from 75/0/8 back to 69/6/8, silently reverting an honesty-rule-#10 promotion the packs themselves record. #1576 fixes that instance; this closes the class. Checks: entry names a file that exists; recorded sha256 matches disk; kind agrees with the pack's own bitexact + witnesses state; index witness count matches the pack; header totals match the entries; no pack file missing from the index. Two things are deliberately NOT failures. A bitexact pack with an empty witnesses[] is the normal case -- the ~60 uncontested packs were bit-precise from the start, and witnesses[] records the promotions that were CONTESTED; demanding a witness everywhere would fail the whole corpus and misstate rule #10. And five hand-curated packs carry no bitexact key at all while the index labels them bitexact: an absent flag is unknown, not false, so reading silence as denial would make the gate assert something the pack never said. Both are reported as notes. Follows the wp18_selftest_gate.py convention -- --selftest plants a mutant per check and fails if any survives, so the gate proves itself falsifiable before CI trusts it. All six mutants are killed. Replayed against the live corpus by demoting the six rungs exactly as the old generator would, the gate fails with exit 1 and names each rung with its witness count. Clean on current master, so it stands alone and does not depend on #1576. Closes #1577
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INDEX_all_formats.jsonis the corpus's machine-readable summary — what consumers read, and what Paper B recomputes its coverage from. Nothing checked that it still agreed with the packs it summarises.That gap has already cost a real regression.
gen_all_formats.pyhardcoded the tier for the six wide GoldenFloat rungs:That was correct when written. The six rungs then acquired independent second witnesses and were promoted in the pack files on 2026-07-05. The generator was never updated, so re-running it rewrote the index from 75/0/8 back to 69/6/8 — silently reverting an honesty-rule-#10 promotion the packs themselves record. It is fixed in #1576, but nothing would have caught the next one.
What the gate checks
sha256matches the file on diskkindagrees with the pack:bitexact+ populatedwitnesses[]⇒"bitexact"; a pack that is not bit-exact is not labelled otherwiselen(pack["witnesses"])Deliberately not checked: a bit-exact pack with an empty
witnesses[]. That is the normal case, not a violation — the ~60 uncontested packs were bit-precise from the start, their independent reference codec being part of how the vectors were generated.witnesses[]records the packs whose promotion was contested. A check demanding a witness everywhere would fail the whole corpus and would misstate rule #10. The asymmetry is the point: a recorded witness must not be ignored, but its absence is not evidence of anything.Also not treated as failure: five hand-curated packs (
bfloat16,fp8_e4m3,fp8_e5m2,mxfp4,gf16) carry nobitexactkey at all while the index labels them bit-exact. An absent flag is unknown, not false; reading silence as denial would make the gate assert something the pack never said. They are reported as notes.Verification
Follows the
wp18_selftest_gate.pyconvention — the gate proves itself falsifiable before it is trusted:And replayed against the live corpus — demoting the six rungs exactly as the old generator would:
Clean on current master (
failures: 0, 5 notes), so this stands alone and does not depend on #1576.Closes #1577