Skip to content

fix(conformance): make the pack generator runnable from a clean checkout - #1576

Open
gHashTag wants to merge 1 commit into
masterfrom
fix/conformance-generator-reproducible
Open

fix(conformance): make the pack generator runnable from a clean checkout#1576
gHashTag wants to merge 1 commit into
masterfrom
fix/conformance-generator-reproducible

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

conformance/vectors/gen_all_formats.py — the script that produces all 83 conformance packs — reads its catalog from a hardcoded path:

CATALOG_LINES = "/tmp/catalog_lines.txt"

That file is not in the repository. On a clean checkout the generator raises FileNotFoundError in parse_catalog() before producing anything, so the corpus can be read but not regenerated. For an artifact whose stated contribution is reproducible bit-exact conformance vectors, the generator not running from a fresh clone is the weakest link in the chain.

The catalog itself was never missing. specs/numeric/formats_catalog.t27 carries all 83 rows as trailing // CATALOG: id=... name=... bits=... comments — the same lines catalog-count-gate.yml counts to enforce the 83-format invariant, in exactly the format parse_catalog expects. The /tmp file was a second, unversioned copy of data the repository already holds.

Two further problems surfaced while fixing this.

Re-running the generator silently reverted the 2026-07-05 promotions. The SELFCONSISTENT branch hardcoded "kind": "bitexact_selfconsistent" with the comment "NOT promoted to bitexact (no independent second witness)". That was true when written, but the six wide rungs acquired independent second witnesses and were promoted in the pack files. A regeneration would have rewritten the index back to 69/6/8, undoing an honesty-rule-#10 promotion that the packs themselves record. The tier now derives from the pack, which is the artefact of record for its own status.

The index exposed no witness information. Fields were file, id, kind, n_vectors, sha256, source — so a consumer of INDEX_all_formats.json, the corpus's machine-readable summary, could not tell which packs carry a witness record without opening all 83 files. The data was already in the packs.

Verification

conformance/vectors/verify_regeneration.py (new) checks the corpus against the digests recorded at HEAD:

packs compared: 83
  digests unchanged : 83
  digests changed   : 0
  tier changed      : 0
OK: regeneration reproduces the committed corpus.

Index changes are strictly additive — 0 entry fields changed, 0 removed; added witnesses and n_vectors per entry plus a top-level witnessed_packs: 10. Both existing gates pass: wp18_selftest_gate.py all PASS, wp18_conformance_gate.py verdict CLEAN, failures [].

Closes #1575

gen_all_formats.py read its catalog from a hardcoded /tmp/catalog_lines.txt that
was never committed, so on a fresh clone it raised FileNotFoundError before
producing anything: the 83-pack corpus could be read but not regenerated.

The catalog was never actually missing. specs/numeric/formats_catalog.t27 carries
all 83 rows as trailing `// CATALOG: id=... ` comments -- the same lines
catalog-count-gate.yml counts for the 83-format invariant, already in the format
parse_catalog expects. The /tmp file was a second, unversioned copy of data the
repository holds. The generator now reads the SSOT, and parse_catalog accepts the
comment prefix while still taking a bare `id=` line.

Two problems surfaced while fixing that.

Re-running the generator silently reverted the 2026-07-05 promotions. The
SELFCONSISTENT branch hardcoded kind bitexact_selfconsistent with the comment
"NOT promoted to bitexact (no independent second witness)" -- true when written,
but the six wide rungs since acquired independent second witnesses and were
promoted in the pack files. A regeneration would have rewritten the index back to
69/6/8, undoing an honesty-rule-#10 promotion the packs themselves record. The
tier now derives from the pack, which is the artefact of record for its own
status, so a future promotion flows into the index by itself. The hand-written
promotion rationales are kept verbatim in a PROMOTION_SOURCE registry rather than
flattened to a generic string.

The index also exposed no witness information: a consumer of
INDEX_all_formats.json could not tell which packs carry a witness record without
opening all 83 files. Each entry now carries a witnesses count, with a top-level
witnessed_packs.

verify_regeneration.py is added as the falsifiable half: it regenerates and
compares against the digests recorded at HEAD.

  packs compared: 83
    digests unchanged : 83
    digests changed   : 0
    tier changed      : 0

Index changes are strictly additive -- 0 entry fields changed or removed. Both
gates pass: wp18_selftest_gate.py all PASS, wp18_conformance_gate.py CLEAN with
no failures.

Closes #1575
@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-07-31 09:36:40 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 41
PRs with All Checks Green 9
READY 1
FAILING 41
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=d6b51a3bcc20 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gen_all_formats.py cannot run on a clean checkout: catalog read from uncommitted /tmp path

1 participant