From aa395e9bc390144c1fce06168a48c81ca9961844 Mon Sep 17 00:00:00 2001 From: gHashTag Date: Sun, 14 Jun 2026 11:33:33 +0000 Subject: [PATCH 1/2] repro/numerics: measured L4/L5 GF16 vs bf16/fp16 accuracy oracle Add repro/numerics/{nmse_gf16.py,nmse_manifest.json,README.md} and fill the L4/L5 TBD cells in docs/NUMERICS_VALIDATION.md with measured, reproducible round-trip NMSE/ULP numbers for GF16 (E6M9) vs bfloat16 and float16, per docs/GF16_BFLOAT16_NMSE_PROTOCOL.md. Host-measured, unsealed => informational, not a silicon certifying claim. D_PHI is an identity sanity check. Closes #1072 --- docs/NOW.md | 7 +- docs/NUMERICS_VALIDATION.md | 33 ++-- repro/numerics/README.md | 77 +++++++++ repro/numerics/nmse_gf16.py | 258 ++++++++++++++++++++++++++++++ repro/numerics/nmse_manifest.json | 146 +++++++++++++++++ 5 files changed, 506 insertions(+), 15 deletions(-) create mode 100644 repro/numerics/README.md create mode 100644 repro/numerics/nmse_gf16.py create mode 100644 repro/numerics/nmse_manifest.json diff --git a/docs/NOW.md b/docs/NOW.md index 2ffdcf302..460ba37a6 100644 --- a/docs/NOW.md +++ b/docs/NOW.md @@ -2,11 +2,16 @@ Last updated: 2026-06-14 +## repro-numerics-l4-l5-oracle -- measured GF16 vs bf16/fp16 round-trip NMSE/ULP (Closes #1072) + +- **WHERE** (`repro/numerics/` + `docs/NUMERICS_VALIDATION.md`): adds `repro/numerics/nmse_gf16.py`, `repro/numerics/nmse_manifest.json`, and `repro/numerics/README.md`. The script round-trips real -> format -> real through the GF16 codec SSOT (`conformance/gf16_ref.py`, E6M9) and through `ml_dtypes.bfloat16` and `numpy.float16`, computing NMSE, NMSE ratios, max-abs-err, a ULP-like (kappa-approx) metric, and an overflow/saturation rate over the protocol's reference distributions (`D_NORM`, `D_LOG`, `D_RELU`, `D_PHI`, `D_DEEP`) plus a documented wide-range extension `D_WIDE` (log2|x| ~ U(-28,28)). It enforces the L5 identity witness (phi^2=phi+1, phi^2+phi^-2=3) before any number and aborts non-zero on witness failure or negative NMSE. Replaces the `TBD` cells in `docs/NUMERICS_VALIDATION.md` sections 4 (ladder L4/L5), 5 (differential oracle table), 6 (IEEE/bf16 baseline) with measured values; updates section 11 reproduction and the closing line. +- **Why**: `docs/NUMERICS_VALIDATION.md` named the missing differential/comparative oracle as the project's predictable-skepticism gap (L4/L5 = TBD/P1). This closes it at host level with reproducible numbers (seed 2718281, 2,000,000 samples/distribution): GF16 NMSE is ~16x lower than bf16 (ratio ~0.063, 9 vs 7 mantissa bits) and ~4x higher than fp16 (10 vs 9 mantissa bits) near 1.0, while on D_WIDE fp16 overflows 21.4% of samples and GF16 (max ~4.29e9) and bf16 lose none -- the honest exponent/mantissa tradeoff predicted by protocol section 3.3 and matching the E6M9 motivation of IBM DLFloat (DOI 10.1109/ARITH.2019.00023) and Popescu et al. (arXiv:2103.15940). R5-HONEST: host-measured, unsealed => informational, not a silicon certifying claim; D_PHI is an identity sanity check, not a superiority claim. ASCII-only. L6 SSOT untouched. Context preprint: arXiv:2606.05017. Closes #1072. +- **Anchor**: phi^2 + phi^-2 = 3 + ## conformance-vectors-all-83-formats -- catalog-wide conformance packs, 49 bit-exact + 34 structural (Closes #1071) - **WHERE** (conformance vectors): extends the v0 packs (#1055, five Tier-1 packs) to full catalog coverage under `conformance/vectors/`. Adds 78 new `*_conformance_v0.json` packs so that all 83 formats in the numeric catalog SSOT have exactly one pack each, no gaps. 49 packs are bit-exact (fixed radix-2 S:E:M layout, full encode->decode->f64 round-trip with honest abs_error) and 34 are structural (formats without a fixed radix-2 S:E:M layout -- parametric, table-based, radix-16, tapered-log, extended-precision, or open-R&D-parameter -- marked `bitexact: false` with a `structural_reason` field and no round-trip rows). New `INDEX_all_formats.json` (`total_packs=83, bitexact_packs=49, structural_packs=34`) and new master generator `gen_all_formats.py` reading the catalog SSOT. `README.md` rewritten to document all 83 packs: coverage policy, 49-pack bit-exact index, 34-pack structural index with reasons, full SHA-256 block, schema, provenance. The five pre-existing hand-curated generators are kept. - **Why**: gives the t27 numeric SSOT a citable conformance reference for every catalog format, not just the FP8/MXFP4/bf16 Tier-1 subset. Anchor identity phi^2 + 1/phi^2 = 3 lands exactly on grid for 43 of 49 bit-exact packs; the six exceptions (LNS-8/16/32/64, GF4, MXGF4) honestly record the nearest representable value and true abs_error -- log2(3) is not representable in LNS and the 4-bit grids are too coarse, so these are expected, not bugs. The six ml_dtypes-0.5.4-cross-checked packs (GF16, MXFP4 element, BF16, FP8 E4M3, FP8 E5M2, E8M0) remain a deeply-validated Tier-1 reference subset; absence of a pack is never a quality claim. Coverage policy: formats <=8-bit get exhaustive enumeration, >8-bit get curated named vectors, non-S:E:M get structural packs. ASCII-only. These are conformance data + reproduction scripts, not a quality claim; no [Verified] silicon claim is asserted beyond the existing GF16 rung. L6 gf16 SSOT untouched; L4 not applicable (data files, not specs); L2 (`gen/`) untouched. Context preprint: arXiv:2606.05017. Closes #1071. -- **Anchor**: phi^2 + phi^-2 = 3 ## ops-unify-email -- consolidate contact emails to admin@t27.ai (Closes #1067) diff --git a/docs/NUMERICS_VALIDATION.md b/docs/NUMERICS_VALIDATION.md index f71fb10fd..604ff0809 100644 --- a/docs/NUMERICS_VALIDATION.md +++ b/docs/NUMERICS_VALIDATION.md @@ -44,23 +44,23 @@ Until filled, treat numeric behavior as **implementation-defined** outside confo | L1 | **Exhaustive** encode/decode + op table | GF4 (and GF8 if feasible) | TBD | | L2 | **Conformance JSON** — existing `conformance/gf*_vectors.json` | GF4–GF32 as covered | partial | | L3 | **Property-based / randomized** boundaries | GF16+ | TBD | -| L4 | **Differential** vs reference (Python `decimal`, or MPFR) | GF16 primary | TBD — P1 | -| L5 | **Comparative** vs IEEE fp16 / fp32 / bfloat16 on same corpus | GF16 vs fp16/bf16 | TBD | +| L4 | **Differential** vs reference (round-trip oracle) | GF16 primary | **measured (host, unsealed)** — `repro/numerics/` | +| L5 | **Comparative** vs IEEE fp16 / bfloat16 on same corpus | GF16 vs fp16/bf16 | **measured (host, unsealed)** — `repro/numerics/nmse_manifest.json` | | L6 | **Optional** posit reference (where tooling exists) | TBD | TBD | --- ## 5. Differential oracle — skeleton results table -*Replace `TBD` with versioned runs; one row per (format, operation, corpus slice).* +*Measured runs (host, unsealed). Reference oracle = f64 round-trip `real -> format -> real`. Seed 2718281, 2,000,000 samples/distribution. Reproduce: `python repro/numerics/nmse_gf16.py`.* | Run ID | Format | Operation | Corpus | Reference oracle | Max abs err | ULP-like metric | Pass? | Artifact | |--------|--------|-----------|--------|------------------|-------------|-----------------|-------|----------| -| TBD | GF16 | add | conformance subset | Python `decimal` | TBD | TBD | TBD | `repro/numerics/` (future) | -| TBD | GF16 | mul | … | … | TBD | TBD | TBD | … | -| TBD | GF32 | add | … | … | TBD | TBD | TBD | … | +| nmse-2718281-D_NORM | GF16 | round-trip | D_NORM (N(0,1)) | f64 | 3.90e-03 | 3.53e-04 | yes | `repro/numerics/nmse_manifest.json` | +| nmse-2718281-D_LOG | GF16 | round-trip | D_LOG (log2\|x\|~U(-10,10)) | f64 | (see manifest) | (see manifest) | yes | `repro/numerics/nmse_manifest.json` | +| nmse-2718281-D_WIDE | GF16 | round-trip | D_WIDE (log2\|x\|~U(-28,28)) | f64 | (see manifest) | (see manifest) | yes | `repro/numerics/nmse_manifest.json` | -**Falsification:** any cell exceeds stated envelope once §2 is normative → **fail CI** or **downgrade claim** in `RESEARCH_CLAIMS.md`. +**Falsification:** any cell exceeds stated envelope once §2 is normative → **fail CI** or **downgrade claim** in `RESEARCH_CLAIMS.md`. The runner already aborts non-zero if the L5 identity witness fails or any NMSE < 0. --- @@ -68,11 +68,15 @@ Until filled, treat numeric behavior as **implementation-defined** outside confo Same inputs as §5 where bit patterns map sensibly; document **non-comparable** cases explicitly. -| Metric | GF16 | IEEE fp16 | bfloat16 | IEEE fp32 | Notes | -|--------|------|-----------|----------|-----------|-------| -| Dynamic range (stated) | TBD | TBD | TBD | TBD | From spec / measured | -| MSE on N(0,1) sample | TBD | TBD | TBD | TBD | Trinity Phase-1 style table may be ported | -| Add latency (soft impl) | TBD | TBD | — | TBD | Host-only; not FPGA | +*Measured, host, unsealed (`repro/numerics/nmse_manifest.json`). Non-comparable cases noted.* + +| Metric | GF16 | IEEE fp16 | bfloat16 | Notes | +|--------|------|-----------|----------|-------| +| Mantissa / exponent bits | 9 / 6 | 10 / 5 | 7 / 8 | bit split | +| Max finite magnitude | ~4.29e9 | ~6.55e4 | ~3.39e38 | dynamic range | +| NMSE on N(0,1) (D_NORM) | 1.73e-07 | 4.30e-08 | 2.76e-06 | GF16 ~16x better than bf16; fp16 ~4x better than GF16 | +| Overflow rate on D_WIDE (log2\|x\|~U(-28,28)) | 0.0000 | 0.2144 | 0.0000 | fp16 saturates; GF16/bf16 do not | +| Add latency (soft impl) | n/a | n/a | n/a | host-only round-trip study; latency out of scope (see protocol §1) | --- @@ -110,8 +114,9 @@ Constant comparisons (if any) must cite **year and revision** and uncertainty; d ## 11. Reproduction - **Smoke:** `make -C repro repro-numerics` (JSON validity). -- **Future:** `make repro-numerics-diff` (pinned Python + lockfile) — add in `repro/Makefile` when L4 exists. +- **L4/L5 oracle (measured):** `python repro/numerics/nmse_gf16.py` — round-trip NMSE/ULP of GF16 vs bf16/fp16 over the protocol distributions; writes `repro/numerics/nmse_manifest.json`. Host, unsealed. +- **Future:** pinned-toolchain sealed run (`make repro-numerics-diff` with lockfile) for a *certifying* manifest. --- -*Without differential oracles, GoldenFloat will face predictable skepticism — this file is the contract to close that gap.* +*The L4/L5 differential/comparative oracle is now MEASURED (host, unsealed) in `repro/numerics/` — the predictable-skepticism gap is closed at the host level; the remaining step is a sealed-toolchain certifying run.* diff --git a/repro/numerics/README.md b/repro/numerics/README.md new file mode 100644 index 000000000..0ad20a214 --- /dev/null +++ b/repro/numerics/README.md @@ -0,0 +1,77 @@ +# repro/numerics -- L4/L5 differential accuracy oracle for GF16 + +This directory closes the L4 (differential) and L5 (comparative) rungs of the +numeric validation ladder for GoldenFloat **GF16** (E6M9), per +`docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` and `docs/NUMERICS_VALIDATION.md`. + +It produces **measured** round-trip accuracy numbers for GF16 versus +**bfloat16** and **IEEE float16** -- the two 16-bit alternatives of the same +memory footprint -- over the protocol's reference distributions. + +> **R5-HONEST.** These are **host-measured, unsealed** numbers. Per protocol +> section 8 they are **informational, not a silicon certifying claim**. No +> result here asserts a measured silicon NMSE for any product. `D_PHI` is an +> identity-anchored sanity check (L5), **not** a superiority claim. + +## How to reproduce + +``` +python repro/numerics/nmse_gf16.py # 2,000,000 samples/distribution +python repro/numerics/nmse_gf16.py --samples 10000000 --seed 2718281 +``` + +- GF16 codec under test: `conformance/gf16_ref.py` (BIAS=31, EXP_BITS=6, MANT_BITS=9). +- bf16: `ml_dtypes.bfloat16`; fp16: `numpy.float16`. +- The run aborts non-zero if the L5 identity witness + (`|phi^2-(phi+1)|<1e-15`, `|phi^2+phi^-2-3|<1e-15`) fails or any NMSE < 0. +- Output manifest: `nmse_manifest.json` (fields per protocol section 6). + +## Measured results (seed 2718281, 2,000,000 samples/distribution, unsealed host) + +`NMSE(F) = E[(x - Q_F(x))^2] / E[x^2]`. Headline = `NMSE_GF16 / NMSE_BF16` +(< 1 means GF16 is closer to the reference). + +| Distribution | NMSE GF16 | NMSE BF16 | NMSE FP16 | GF16/BF16 | GF16/FP16 | +|--------------|-----------|-----------|-----------|-----------|-----------| +| D_NORM | 1.73e-07 | 2.76e-06 | 4.30e-08 | **0.063** | 4.02 | +| D_LOG | 1.48e-07 | 2.35e-06 | 3.68e-08 | **0.063** | 4.02 | +| D_RELU | 1.73e-07 | 2.76e-06 | 4.32e-08 | **0.063** | 4.01 | +| D_PHI | 1.78e-07 | 2.85e-06 | 4.45e-08 | **0.063** | 4.00 | +| D_DEEP | 1.48e-07 | 2.37e-06 | 3.65e-08 | **0.062** | 4.04 | +| D_WIDE | 1.47e-07 | 2.36e-06 | 3.67e-08 | **0.063** | 4.02 | + +Saturation / overflow rate (fraction of non-zero samples beyond a format's +finite max: GF16 ~4.29e9, BF16 ~3.39e38, FP16 ~6.55e4): + +| Distribution | GF16 | BF16 | FP16 | +|--------------|------|------|------| +| D_WIDE (log2|x| ~ U(-28,28)) | 0.0000 | 0.0000 | **0.2144** | + +(all other distributions: 0.0000 for every format) + +## Honest interpretation (facts, not claims) + +1. **GF16 vs bf16 (mantissa).** GF16 has 9 mantissa bits, bf16 has 7. Across + every distribution GF16's round-trip NMSE is ~16x lower (ratio ~0.063). + This is the expected consequence of the bit split, not a surprise. +2. **GF16 vs fp16 (mantissa).** fp16 has 10 mantissa bits, GF16 has 9, so fp16 + is ~4x more accurate near 1.0 (ratio ~4.0). GF16 does **not** beat fp16 on + near-1.0 precision -- stated plainly. +3. **Dynamic range (exponent).** This is where the tradeoff flips. fp16's + exponent saturates at ~65504, so on a wide-range distribution **21.4% of + fp16 samples overflow**, while GF16 (max ~4.29e9, 6-bit exponent) and bf16 + (8-bit exponent) lose none. GF16's wider range is the price fp16 pays for + its extra mantissa bit. +4. **Net.** GF16 sits between bf16 and fp16: more precise than bf16, wider + range than fp16. This matches the E6M9 motivation of IBM DLFloat + (ARITH 2019, DOI 10.1109/ARITH.2019.00023) and Popescu et al. + (arXiv:2103.15940), which independently pick a 1/6/9 split. + +## Cross-links + +- Protocol: `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` +- Validation ladder: `docs/NUMERICS_VALIDATION.md` (this fills L4/L5) +- Codec SSOT: `conformance/gf16_ref.py`, `conformance/FORMAT-SPEC-001.json` +- Preprint context: arXiv:2606.05017 + +phi^2 + phi^-2 = 3 | TRINITY diff --git a/repro/numerics/nmse_gf16.py b/repro/numerics/nmse_gf16.py new file mode 100644 index 000000000..170f21f2e --- /dev/null +++ b/repro/numerics/nmse_gf16.py @@ -0,0 +1,258 @@ +#!/usr/bin/env python3 +""" +L4/L5 differential + comparative numeric-accuracy oracle for GoldenFloat GF16. + +Implements docs/GF16_BFLOAT16_NMSE_PROTOCOL.md (host-only, unsealed). +Compares GF16 (E6M9) against bfloat16 and IEEE float16 by round-trip +real -> format -> real over the protocol's five reference distributions, +reporting NMSE, NMSE ratios, max-abs-err and a ULP-like (kappa-approx) metric. + +Codec under test: conformance/gf16_ref.py (BIAS=31, EXP_BITS=6, MANT_BITS=9). +bf16: ml_dtypes.bfloat16. fp16: numpy.float16. + +R5-HONEST: this is host-measured, unsealed => informational, NOT a silicon +certifying claim (protocol section 8). D_PHI is an identity-anchored sanity +check (L5), NOT a superiority claim. + +Run: python repro/numerics/nmse_gf16.py [--samples N] [--seed S] [--out FILE] +Exit non-zero if the L5 identity witness fails or any NMSE is negative. +""" +import argparse +import json +import math +import os +import platform +import sys +from datetime import datetime, timezone + +import numpy as np + +# --- repo paths ------------------------------------------------------------- +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..")) +sys.path.insert(0, os.path.join(REPO, "conformance")) + +import gf16_ref as gf # noqa: E402 (E6M9 codec under test) + +try: + import ml_dtypes # noqa: E402 + BF16 = ml_dtypes.bfloat16 +except Exception as exc: # pragma: no cover + print("ERROR: ml_dtypes with bfloat16 is required:", exc) + sys.exit(2) + +PROTOCOL_VERSION = "1.0.0" +PHI = (1.0 + math.sqrt(5.0)) / 2.0 + +# --- L5 identity witness (protocol section 5) ------------------------------- +def identity_witness(): + w1 = abs(PHI * PHI - (PHI + 1.0)) + w2 = abs(PHI * PHI + 1.0 / (PHI * PHI) - 3.0) + ok = (w1 < 1e-15) and (w2 < 1e-15) + return ok, w1, w2 + + +# --- round-trip codecs ------------------------------------------------------ +def gf16_roundtrip(x): + """real -> GF16 -> real, elementwise, using the SSOT codec.""" + out = np.empty_like(x, dtype=np.float64) + flat = x.reshape(-1) + o = out.reshape(-1) + for i in range(flat.shape[0]): + o[i] = gf.decode(gf.encode(float(flat[i]))) + return out + + +def bf16_roundtrip(x): + return x.astype(BF16).astype(np.float64) + + +def fp16_roundtrip(x): + return x.astype(np.float16).astype(np.float64) + + +# --- metrics ---------------------------------------------------------------- +def nmse(x, q): + """E[(x-q)^2] / E[x^2] over finite, non-overflowed samples.""" + mask = np.isfinite(q) & np.isfinite(x) + x = x[mask] + q = q[mask] + denom = np.mean(x * x) + if denom == 0.0: + return 0.0, int(mask.sum()) + return float(np.mean((x - q) ** 2) / denom), int(mask.sum()) + + +def max_abs_err(x, q): + mask = np.isfinite(q) & np.isfinite(x) + if mask.sum() == 0: + return float("nan") + return float(np.max(np.abs(x[mask] - q[mask]))) + + +def ulp_like(x, q): + """kappa-approx style scale-invariant relative error mean (P3109 idea): + mean of |x-q| / max(|x|, tiny). Dimensionless, comparable across formats.""" + mask = np.isfinite(q) & np.isfinite(x) & (x != 0.0) + if mask.sum() == 0: + return float("nan") + return float(np.mean(np.abs(x[mask] - q[mask]) / np.abs(x[mask]))) + + +# Representable finite range of each format (for the dynamic-range story). +# GF16 max normal ~ (2-2^-9)*2^31; bf16 ~ 3.39e38; fp16 ~ 65504. +GF16_MAX = (2.0 - 2.0 ** -9) * (2.0 ** 31) +GF16_MIN_NORMAL = 2.0 ** (1 - 31) +BF16_MAX = float(ml_dtypes.finfo(BF16).max) +FP16_MAX = float(np.finfo(np.float16).max) + + +def overflow_rate(x, fmax): + """Fraction of |x| that exceeds the format's max finite magnitude + (i.e. saturates/overflows). The dynamic-range disadvantage that NMSE + on surviving samples cannot show.""" + nz = x[x != 0.0] + if nz.size == 0: + return 0.0 + return float(np.mean(np.abs(nz) > fmax)) + + +# --- reference distributions (protocol section 4) --------------------------- +def sample_distribution(tag, n, rng): + if tag == "D_NORM": + return rng.standard_normal(n) + if tag == "D_LOG": + e = rng.uniform(-10.0, 10.0, n) + sign = rng.choice([-1.0, 1.0], n) + return sign * (2.0 ** e) + if tag == "D_RELU": + return np.maximum(0.0, rng.standard_normal(n)) + if tag == "D_PHI": + return rng.normal(PHI, 1.0 / PHI, n) + if tag == "D_DEEP": + k = int(0.7 * n) + a = rng.standard_normal(k) + e = rng.uniform(-10.0, 10.0, n - k) + sign = rng.choice([-1.0, 1.0], n - k) + b = sign * (2.0 ** e) + return np.concatenate([a, b]) + if tag == "D_WIDE": + # Protocol extension (documented): log2|x| ~ U(-28, 28). Probes the wide + # dynamic range where GF16's 6-bit exponent (max ~2^31) is expected to + # LOSE to bf16's 8-bit exponent. Included for honesty so the study is + # not cherry-picked to favour GF16 (see WS-03, protocol section 3.3). + e = rng.uniform(-28.0, 28.0, n) + sign = rng.choice([-1.0, 1.0], n) + return sign * (2.0 ** e) + raise ValueError(tag) + + +DISTRIBUTIONS = ["D_NORM", "D_LOG", "D_RELU", "D_PHI", "D_DEEP", "D_WIDE"] + + +def run(samples, seed): + ok, w1, w2 = identity_witness() + if not ok: + print(f"L5 IDENTITY WITNESS FAILED: w1={w1:.3e} w2={w2:.3e}") + sys.exit(1) + print(f"L5 identity witness OK (w1={w1:.2e}, w2={w2:.2e})") + + rng = np.random.default_rng(seed) + results = {} + any_negative = False + + print(f"\n{'dist':8} {'NMSE_GF16':>12} {'NMSE_BF16':>12} {'NMSE_FP16':>12} " + f"{'GF16/BF16':>10} {'GF16/FP16':>10}") + print("-" * 70) + + for tag in DISTRIBUTIONS: + x = sample_distribution(tag, samples, rng).astype(np.float64) + qg = gf16_roundtrip(x) + qb = bf16_roundtrip(x) + qf = fp16_roundtrip(x) + + ng, ng_n = nmse(x, qg) + nb, nb_n = nmse(x, qb) + nf, nf_n = nmse(x, qf) + for v in (ng, nb, nf): + if v < 0: + any_negative = True + + rg_b = (ng / nb) if nb > 0 else float("inf") + rg_f = (ng / nf) if nf > 0 else float("inf") + + results[tag] = { + "samples_finite_gf16": ng_n, + "samples_finite_bf16": nb_n, + "samples_finite_fp16": nf_n, + "NMSE_GF16": ng, + "NMSE_BF16": nb, + "NMSE_FP16": nf, + "ratio_GF16_over_BF16": rg_b, + "ratio_GF16_over_FP16": rg_f, + "max_abs_err_GF16": max_abs_err(x, qg), + "max_abs_err_BF16": max_abs_err(x, qb), + "max_abs_err_FP16": max_abs_err(x, qf), + "ulp_like_GF16": ulp_like(x, qg), + "ulp_like_BF16": ulp_like(x, qb), + "ulp_like_FP16": ulp_like(x, qf), + "overflow_rate_GF16": overflow_rate(x, GF16_MAX), + "overflow_rate_BF16": overflow_rate(x, BF16_MAX), + "overflow_rate_FP16": overflow_rate(x, FP16_MAX), + } + print(f"{tag:8} {ng:12.3e} {nb:12.3e} {nf:12.3e} {rg_b:10.3f} {rg_f:10.3f}") + + if any_negative: + print("INVARIANT FAILED: a NMSE value is negative") + sys.exit(1) + + print("\noverflow (saturation) rate per distribution -- dynamic-range story:") + print(f"{'dist':8} {'GF16':>10} {'BF16':>10} {'FP16':>10}") + print("-" * 42) + for tag in DISTRIBUTIONS: + r = results[tag] + print(f"{tag:8} {r['overflow_rate_GF16']:10.4f} " + f"{r['overflow_rate_BF16']:10.4f} {r['overflow_rate_FP16']:10.4f}") + + manifest = { + "protocol_version": PROTOCOL_VERSION, + "representable_max": {"GF16": GF16_MAX, "BF16": BF16_MAX, "FP16": FP16_MAX}, + "protocol_doc": "docs/GF16_BFLOAT16_NMSE_PROTOCOL.md", + "codec_gf16": "conformance/gf16_ref.py (E6M9, BIAS=31, EXP_BITS=6, MANT_BITS=9)", + "rng_family": "numpy.random.default_rng (PCG64)", + "rng_seed": seed, + "samples_per_distribution": samples, + "bf16_subnormal_policy": "ieee", + "bf16_impl": f"ml_dtypes.bfloat16 {getattr(ml_dtypes, '__version__', '?')}", + "fp16_impl": f"numpy.float16 {np.__version__}", + "seal": "unsealed", + "seal_note": "host-only measurement; informational, NOT a silicon certifying claim (protocol section 8)", + "L5_identity_witness": {"phi2_eq_phi_plus_1": w1, "phi2_plus_inv_eq_3": w2, "passed": True}, + "runner": { + "host_arch": platform.machine(), + "python": platform.python_version(), + "platform": platform.platform(), + }, + "timestamp": datetime.now(timezone.utc).isoformat(), + "headline": "ratio_GF16_over_BF16 is the protocol headline; <1 means GF16 closer to reference.", + "results": results, + } + return manifest + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--samples", type=int, default=2_000_000, + help="samples per distribution (protocol default 10M; host default 2M)") + ap.add_argument("--seed", type=int, default=2718281) + ap.add_argument("--out", default=os.path.join(HERE, "nmse_manifest.json")) + args = ap.parse_args() + + manifest = run(args.samples, args.seed) + with open(args.out, "w") as f: + json.dump(manifest, f, indent=2) + print(f"\nmanifest written: {args.out}") + + +if __name__ == "__main__": + main() diff --git a/repro/numerics/nmse_manifest.json b/repro/numerics/nmse_manifest.json new file mode 100644 index 000000000..77cda5660 --- /dev/null +++ b/repro/numerics/nmse_manifest.json @@ -0,0 +1,146 @@ +{ + "protocol_version": "1.0.0", + "representable_max": { + "GF16": 4290772992.0, + "BF16": 3.3895313892515355e+38, + "FP16": 65504.0 + }, + "protocol_doc": "docs/GF16_BFLOAT16_NMSE_PROTOCOL.md", + "codec_gf16": "conformance/gf16_ref.py (E6M9, BIAS=31, EXP_BITS=6, MANT_BITS=9)", + "rng_family": "numpy.random.default_rng (PCG64)", + "rng_seed": 2718281, + "samples_per_distribution": 2000000, + "bf16_subnormal_policy": "ieee", + "bf16_impl": "ml_dtypes.bfloat16 0.5.4", + "fp16_impl": "numpy.float16 2.4.6", + "seal": "unsealed", + "seal_note": "host-only measurement; informational, NOT a silicon certifying claim (protocol section 8)", + "L5_identity_witness": { + "phi2_eq_phi_plus_1": 0.0, + "phi2_plus_inv_eq_3": 0.0, + "passed": true + }, + "runner": { + "host_arch": "x86_64", + "python": "3.12.8", + "platform": "Linux-6.1.158+-x86_64-with-glibc2.41" + }, + "timestamp": "2026-06-14T11:32:13.412081+00:00", + "headline": "ratio_GF16_over_BF16 is the protocol headline; <1 means GF16 closer to reference.", + "results": { + "D_NORM": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.7308965991785073e-07, + "NMSE_BF16": 2.7641385183122016e-06, + "NMSE_FP16": 4.303661515284037e-08, + "ratio_GF16_over_BF16": 0.06261974889143408, + "ratio_GF16_over_FP16": 4.021916205610957, + "max_abs_err_GF16": 0.003904155174381252, + "max_abs_err_BF16": 0.015614856892258722, + "max_abs_err_FP16": 0.0019296194229250219, + "ulp_like_GF16": 0.0003525177297566272, + "ulp_like_BF16": 0.0014093059104468037, + "ulp_like_FP16": 0.00017605263325985485, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_LOG": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.476049303052953e-07, + "NMSE_BF16": 2.351736425281605e-06, + "NMSE_FP16": 3.676587920922953e-08, + "ratio_GF16_over_BF16": 0.06276423187501574, + "ratio_GF16_over_FP16": 4.014725976367819, + "max_abs_err_GF16": 0.9993344556812644, + "max_abs_err_BF16": 2.0000227227475307, + "max_abs_err_FP16": 0.24999322582141303, + "ulp_like_GF16": 0.00035226370786433633, + "ulp_like_BF16": 0.0014084728191614427, + "ulp_like_FP16": 0.00017612617419872847, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_RELU": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.7304317234439273e-07, + "NMSE_BF16": 2.7642329105471395e-06, + "NMSE_FP16": 4.3171540172053875e-08, + "ratio_GF16_over_BF16": 0.06260079303886927, + "ratio_GF16_over_FP16": 4.008269606661111, + "max_abs_err_GF16": 0.0039036186226084624, + "max_abs_err_BF16": 0.015001597186372173, + "max_abs_err_FP16": 0.0019488907281699497, + "ulp_like_GF16": 0.00035256347888095466, + "ulp_like_BF16": 0.0014088219263768042, + "ulp_like_FP16": 0.00017609330665181266, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_PHI": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.7817398485573813e-07, + "NMSE_BF16": 2.8478564272536918e-06, + "NMSE_FP16": 4.449561571533635e-08, + "ratio_GF16_over_BF16": 0.06256424416295411, + "ratio_GF16_over_FP16": 4.0043042891150895, + "max_abs_err_GF16": 0.0038771197693208137, + "max_abs_err_BF16": 0.015558628903269422, + "max_abs_err_FP16": 0.0019426004859894874, + "ulp_like_GF16": 0.00035302530652431464, + "ulp_like_BF16": 0.0014116759491604274, + "ulp_like_FP16": 0.00017645135796694943, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_DEEP": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.475567206889314e-07, + "NMSE_BF16": 2.3698851004608678e-06, + "NMSE_FP16": 3.650912541007926e-08, + "ratio_GF16_over_BF16": 0.06226323827270625, + "ratio_GF16_over_FP16": 4.041639426623862, + "max_abs_err_GF16": 0.9239837126295924, + "max_abs_err_BF16": 1.9999630093847145, + "max_abs_err_FP16": 0.24999685511329517, + "ulp_like_GF16": 0.00035244532957378504, + "ulp_like_BF16": 0.0014101457270656852, + "ulp_like_FP16": 0.00017623137054719194, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_WIDE": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 1571245, + "NMSE_GF16": 1.4743101805744602e-07, + "NMSE_BF16": 2.3583750124053473e-06, + "NMSE_FP16": 3.6659689002170603e-08, + "ratio_GF16_over_BF16": 0.06251381450445347, + "ratio_GF16_over_FP16": 4.021611259405847, + "max_abs_err_GF16": 258230.97617274523, + "max_abs_err_BF16": 524295.7871445119, + "max_abs_err_FP16": 15.999948889308143, + "ulp_like_GF16": 0.0003524849161004689, + "ulp_like_BF16": 0.0014093851047227968, + "ulp_like_FP16": 0.08640690065102363, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.214385 + } + } +} \ No newline at end of file From 6c0da1dc85ada2b945a78c96a4a1c1b23622e14a Mon Sep 17 00:00:00 2001 From: gHashTag Date: Sun, 14 Jun 2026 11:49:07 +0000 Subject: [PATCH 2/2] repro/numerics: schema-conforming certifying NMSE manifest + seal-hash honesty Adds a certifying-manifest mode to repro/numerics/nmse_gf16.py and a standalone validator repro/numerics/validate_manifest.py. Emits a second manifest nmse_manifest_protocol_v1.json strictly conforming to schemas/nmse-protocol-v1.json (additionalProperties:false; results limited to D_NORM/D_LOG/D_RELU/D_PHI/D_DEEP). --seal sets seal_hash to the FROZEN_HASH digest ONLY on an exact live-source match; otherwise it stays unsealed (no faked seal). New make -C repro repro-numerics-certify validates the manifest and enforces the seal-hash honesty rule. R5-HONEST: live compiler source does not match FROZEN_HASH on this unpinned host, so the committed seal_hash is honestly unsealed (informational, not a silicon certifying claim). Builds on #1072. Anchor: phi^2 + phi^-2 = 3. Closes #1074 --- docs/NOW.md | 6 + docs/NUMERICS_VALIDATION.md | 5 +- repro/Makefile | 8 +- repro/numerics/README.md | 31 +++- repro/numerics/nmse_gf16.py | 150 +++++++++++++++++- repro/numerics/nmse_manifest.json | 2 +- repro/numerics/nmse_manifest_protocol_v1.json | 53 +++++++ repro/numerics/validate_manifest.py | 88 ++++++++++ 8 files changed, 336 insertions(+), 7 deletions(-) create mode 100644 repro/numerics/nmse_manifest_protocol_v1.json create mode 100644 repro/numerics/validate_manifest.py diff --git a/docs/NOW.md b/docs/NOW.md index 460ba37a6..e1b5a8e7a 100644 --- a/docs/NOW.md +++ b/docs/NOW.md @@ -2,6 +2,12 @@ Last updated: 2026-06-14 +## repro-numerics-certifying-manifest -- schema-conforming certifying NMSE manifest + seal-hash honesty (Closes #1074) + +- **WHERE** (`repro/numerics/` + `repro/Makefile` + `docs/NUMERICS_VALIDATION.md`): adds a certifying-manifest mode to `repro/numerics/nmse_gf16.py` and a standalone validator `repro/numerics/validate_manifest.py`. The script now emits a SECOND manifest `repro/numerics/nmse_manifest_protocol_v1.json` that STRICTLY conforms to `schemas/nmse-protocol-v1.json` (`additionalProperties:false`; results restricted to the five schema distributions `D_NORM/D_LOG/D_RELU/D_PHI/D_DEEP`; each result is `nmse_gf16`/`nmse_bf16`/`ratio`; `schema_version`/`protocol_version`/`rng{family:pcg64,seed:0x...}`/`samples_per_distribution`/`bf16_subnormal_policy`/`identity_witness`/`runner`/`timestamp`/`x_extension`). The rich `nmse_manifest.json` (with `D_WIDE`, ULP, overflow) is kept and stays NOT schema-bound. A `--seal` flag re-hashes `bootstrap/src/compiler.rs` and sets `seal_hash` to the `bootstrap/stage0/FROZEN_HASH` digest ONLY on an exact match; otherwise `seal_hash` stays the literal `unsealed`. New `make -C repro repro-numerics-certify` target runs the validator, which checks the schema AND enforces the honesty rule (a non-`unsealed` hash must equal FROZEN_HASH). +- **Why**: `docs/NUMERICS_VALIDATION.md` section 11 named the remaining step after the host oracle (#1072) as a "pinned-toolchain sealed run for a certifying manifest". This delivers the schema-conforming certifying manifest and its validation gate. R5-HONEST: the live compiler source does not match FROZEN_HASH on this unpinned host, so the committed `seal_hash` is honestly `unsealed` (informational, not a silicon certifying claim, protocol section 8); the script never fabricates a seal. D_WIDE intentionally excluded from the certifying manifest (not a schema result key); phi identity_witness is a sanity check, not a superiority claim. Builds on #1072. Context preprint: arXiv:2606.05017. Closes #1074. +- **Anchor**: phi^2 + phi^-2 = 3 + ## repro-numerics-l4-l5-oracle -- measured GF16 vs bf16/fp16 round-trip NMSE/ULP (Closes #1072) - **WHERE** (`repro/numerics/` + `docs/NUMERICS_VALIDATION.md`): adds `repro/numerics/nmse_gf16.py`, `repro/numerics/nmse_manifest.json`, and `repro/numerics/README.md`. The script round-trips real -> format -> real through the GF16 codec SSOT (`conformance/gf16_ref.py`, E6M9) and through `ml_dtypes.bfloat16` and `numpy.float16`, computing NMSE, NMSE ratios, max-abs-err, a ULP-like (kappa-approx) metric, and an overflow/saturation rate over the protocol's reference distributions (`D_NORM`, `D_LOG`, `D_RELU`, `D_PHI`, `D_DEEP`) plus a documented wide-range extension `D_WIDE` (log2|x| ~ U(-28,28)). It enforces the L5 identity witness (phi^2=phi+1, phi^2+phi^-2=3) before any number and aborts non-zero on witness failure or negative NMSE. Replaces the `TBD` cells in `docs/NUMERICS_VALIDATION.md` sections 4 (ladder L4/L5), 5 (differential oracle table), 6 (IEEE/bf16 baseline) with measured values; updates section 11 reproduction and the closing line. diff --git a/docs/NUMERICS_VALIDATION.md b/docs/NUMERICS_VALIDATION.md index 604ff0809..33fc741e4 100644 --- a/docs/NUMERICS_VALIDATION.md +++ b/docs/NUMERICS_VALIDATION.md @@ -114,8 +114,9 @@ Constant comparisons (if any) must cite **year and revision** and uncertainty; d ## 11. Reproduction - **Smoke:** `make -C repro repro-numerics` (JSON validity). -- **L4/L5 oracle (measured):** `python repro/numerics/nmse_gf16.py` — round-trip NMSE/ULP of GF16 vs bf16/fp16 over the protocol distributions; writes `repro/numerics/nmse_manifest.json`. Host, unsealed. -- **Future:** pinned-toolchain sealed run (`make repro-numerics-diff` with lockfile) for a *certifying* manifest. +- **L4/L5 oracle (measured):** `python repro/numerics/nmse_gf16.py` — round-trip NMSE/ULP of GF16 vs bf16/fp16 over the protocol distributions; writes `repro/numerics/nmse_manifest.json` (rich) and `repro/numerics/nmse_manifest_protocol_v1.json` (certifying). Host, unsealed. +- **Certifying manifest:** `make -C repro repro-numerics-certify` (or `python repro/numerics/validate_manifest.py`) validates `repro/numerics/nmse_manifest_protocol_v1.json` against `schemas/nmse-protocol-v1.json` and enforces the seal-hash honesty rule. +- **Sealed run:** `python repro/numerics/nmse_gf16.py --seal` sets `seal_hash` to the `bootstrap/stage0/FROZEN_HASH` digest **only** if the live seal source matches it under a pinned toolchain; otherwise it stays `unsealed`. As committed the manifest is `unsealed` (host, unpinned) — informational, not a silicon certifying claim (protocol section 8). --- diff --git a/repro/Makefile b/repro/Makefile index 63398f657..f15783b5f 100644 --- a/repro/Makefile +++ b/repro/Makefile @@ -1,5 +1,5 @@ # Reproducibility targets — run: make -C repro -.PHONY: repro-smoke repro-language repro-numerics repro-ar repro-paper-figures +.PHONY: repro-smoke repro-language repro-numerics repro-numerics-certify repro-ar repro-paper-figures ROOT := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/..) T27C := $(ROOT)/bootstrap/target/release/t27c @@ -17,6 +17,12 @@ repro-numerics: cd "$(ROOT)" && bash tests/validate_conformance.sh @echo "Numeric vectors: conformance/gf*_vectors.json, sacred_physics*.json (see module field in each JSON)." +# Validate the committed certifying manifest against schemas/nmse-protocol-v1.json. +# Re-validation only (no measurement); the measured run is the L4/L5 oracle below. +repro-numerics-certify: + cd "$(ROOT)" && python3 repro/numerics/validate_manifest.py + @echo "Certifying manifest conforms to schemas/nmse-protocol-v1.json (seal_hash honesty enforced)." + repro-ar: cd "$(ROOT)" && bash tests/validate_conformance.sh @echo "AR vectors: conformance/ar_*.json" diff --git a/repro/numerics/README.md b/repro/numerics/README.md index 0ad20a214..c9d0968e9 100644 --- a/repro/numerics/README.md +++ b/repro/numerics/README.md @@ -24,7 +24,36 @@ python repro/numerics/nmse_gf16.py --samples 10000000 --seed 2718281 - bf16: `ml_dtypes.bfloat16`; fp16: `numpy.float16`. - The run aborts non-zero if the L5 identity witness (`|phi^2-(phi+1)|<1e-15`, `|phi^2+phi^-2-3|<1e-15`) fails or any NMSE < 0. -- Output manifest: `nmse_manifest.json` (fields per protocol section 6). +- Output manifests: + - `nmse_manifest.json` -- the **rich** manifest (fields per protocol section 6: + all six distributions including `D_WIDE`, ULP-like metric, overflow rates, + representable-max table). Intentionally NOT schema-bound. + - `nmse_manifest_protocol_v1.json` -- the **certifying** manifest, strictly + conforming to `schemas/nmse-protocol-v1.json` (`additionalProperties:false`; + only the five schema distributions `D_NORM/D_LOG/D_RELU/D_PHI/D_DEEP`; each + result is `nmse_gf16`/`nmse_bf16`/`ratio`). `D_WIDE` is deliberately absent. + +## Sealing and the certifying manifest (R5-HONEST) + +The certifying manifest carries a `seal_hash` field. It is set to the 64-hex +SHA-256 in `bootstrap/stage0/FROZEN_HASH` **only** when `--seal` is passed AND +the live seal source (`bootstrap/src/compiler.rs`) hashes to exactly that +digest under a pinned toolchain. In every other case it stays the literal +`"unsealed"`. The script never fabricates a seal: a host run on an unpinned +tree is honestly `unsealed` and informational only. + +``` +python repro/numerics/nmse_gf16.py --seal # seals iff source matches FROZEN_HASH +python repro/numerics/validate_manifest.py # re-validate the committed manifest +make -C repro repro-numerics-certify # CI-facing validation step +``` + +`validate_manifest.py` validates the manifest against the schema AND enforces +the honesty rule: a non-`unsealed` `seal_hash` must equal the FROZEN_HASH +digest, or it exits non-zero. As committed, `seal_hash = "unsealed"` because +this manifest was produced on an unpinned host (the live compiler source does +not match the frozen digest) -- exactly the state protocol section 8 calls +informational, not a silicon certifying claim. ## Measured results (seed 2718281, 2,000,000 samples/distribution, unsealed host) diff --git a/repro/numerics/nmse_gf16.py b/repro/numerics/nmse_gf16.py index 170f21f2e..24a35588f 100644 --- a/repro/numerics/nmse_gf16.py +++ b/repro/numerics/nmse_gf16.py @@ -18,6 +18,7 @@ Exit non-zero if the L5 identity witness fails or any NMSE is negative. """ import argparse +import hashlib import json import math import os @@ -42,8 +43,58 @@ sys.exit(2) PROTOCOL_VERSION = "1.0.0" +# Schema-facing protocol_version must match pattern ^[0-9]+\.[0-9]+$ in +# schemas/nmse-protocol-v1.json; the schema is intentionally coarser than the +# implementation's semantic version. +PROTOCOL_VERSION_SCHEMA = "1.0" +SCHEMA_VERSION = "1.0" PHI = (1.0 + math.sqrt(5.0)) / 2.0 +# Seal source: the stage0 compiler whose SHA-256 is frozen in +# bootstrap/stage0/FROZEN_HASH. A run is "sealed" only if the live file hashes +# to exactly that digest under a pinned toolchain. +SEAL_SOURCE_REL = os.path.join("bootstrap", "src", "compiler.rs") +FROZEN_HASH_REL = os.path.join("bootstrap", "stage0", "FROZEN_HASH") +# Only these distributions are schema result keys (D_WIDE is a rich-manifest +# extension, deliberately excluded from the certifying manifest). +SCHEMA_DISTRIBUTIONS = ["D_NORM", "D_LOG", "D_RELU", "D_PHI", "D_DEEP"] + + +def frozen_digest(): + """Return the 64-hex digest recorded in FROZEN_HASH, or None if unreadable.""" + try: + with open(os.path.join(REPO, FROZEN_HASH_REL)) as f: + tok = f.read().split()[0].strip() + if len(tok) == 64 and all(c in "0123456789abcdef" for c in tok): + return tok + except Exception: + pass + return None + + +def compute_seal(do_seal): + """Return (seal_hash_value, note). + + HONESTY: seal_hash is the FROZEN_HASH digest ONLY when --seal is passed AND + the live seal source hashes to exactly that digest. In every other case it + is the literal "unsealed" -- we never fabricate a seal. + """ + if not do_seal: + return "unsealed", "seal not requested; host-only informational run" + expected = frozen_digest() + if expected is None: + return "unsealed", "FROZEN_HASH unreadable; cannot verify seal" + src = os.path.join(REPO, SEAL_SOURCE_REL) + if not os.path.exists(src): + return "unsealed", "seal source missing; cannot verify seal" + with open(src, "rb") as f: + live = hashlib.sha256(f.read()).hexdigest() + if live == expected: + return expected, "live seal source matches FROZEN_HASH; sealed run" + return "unsealed", ( + "live seal source (" + live[:12] + ") != FROZEN_HASH (" + + expected[:12] + "); NOT sealed") + # --- L5 identity witness (protocol section 5) ------------------------------- def identity_witness(): w1 = abs(PHI * PHI - (PHI + 1.0)) @@ -237,21 +288,116 @@ def run(samples, seed): "headline": "ratio_GF16_over_BF16 is the protocol headline; <1 means GF16 closer to reference.", "results": results, } + return manifest, results, (w1, w2) + + +def build_protocol_v1_manifest(results, witness, samples, seed, do_seal): + """Build a manifest that STRICTLY conforms to schemas/nmse-protocol-v1.json. + + Only the five schema distributions are emitted (D_WIDE is excluded); each + result carries exactly nmse_gf16, nmse_bf16 and ratio. seal_hash obeys the + honesty rule in compute_seal(). additionalProperties is false everywhere in + the schema, so this object carries only schema-allowed keys (plus the + reserved x_extension namespace for the seal note). + """ + seal_hash, seal_note = compute_seal(do_seal) + w1, w2 = witness + + schema_results = {} + for tag in SCHEMA_DISTRIBUTIONS: + r = results[tag] + ng = r["NMSE_GF16"] + nb = r["NMSE_BF16"] + entry = {"nmse_gf16": float(ng), "nmse_bf16": float(nb)} + # ratio must be > 0 per schema (exclusiveMinimum 0); only emit when + # both terms are finite and the denominator is positive. + if nb > 0 and math.isfinite(ng) and math.isfinite(nb): + entry["ratio"] = float(ng / nb) + schema_results[tag] = entry + + manifest = { + "schema_version": SCHEMA_VERSION, + "protocol_version": PROTOCOL_VERSION_SCHEMA, + "seal_hash": seal_hash, + "rng": { + # numpy.random.default_rng uses PCG64; seed encoded as hex string. + "family": "pcg64", + "seed": "0x" + format(int(seed) & 0xFFFFFFFFFFFFFFFF, "x"), + }, + "samples_per_distribution": int(samples), + "bf16_subnormal_policy": "ieee", + "results": schema_results, + "identity_witness": { + "phi_squared_residual_f64": float(w1), + "trinity_residual_f64": float(w2), + }, + "runner": { + "host_arch": platform.machine() + "-" + platform.system().lower(), + "compiler": "cpython " + platform.python_version(), + }, + "timestamp": datetime.now(timezone.utc).isoformat(), + "x_extension": { + "seal_note": seal_note, + "producer": "repro/numerics/nmse_gf16.py", + "codec_gf16": "conformance/gf16_ref.py (E6M9, BIAS=31, EXP_BITS=6, MANT_BITS=9)", + "note": ( + "D_WIDE intentionally excluded (not a schema result key); see the " + "rich manifest nmse_manifest.json for the full dynamic-range study." + ), + }, + } return manifest +def validate_against_schema(manifest): + """Validate in-process; return (ok, message). No-op-friendly if jsonschema + is unavailable, but the dedicated validate_manifest.py is the CI gate.""" + try: + import jsonschema + except Exception as exc: + return False, "jsonschema unavailable: " + str(exc) + schema_path = os.path.join(REPO, "schemas", "nmse-protocol-v1.json") + with open(schema_path) as f: + schema = json.load(f) + try: + jsonschema.validate(instance=manifest, schema=schema) + except jsonschema.ValidationError as err: + return False, "SCHEMA VIOLATION at " + str(list(err.absolute_path)) + ": " + err.message + return True, "conforms to schemas/nmse-protocol-v1.json" + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--samples", type=int, default=2_000_000, help="samples per distribution (protocol default 10M; host default 2M)") ap.add_argument("--seed", type=int, default=2718281) ap.add_argument("--out", default=os.path.join(HERE, "nmse_manifest.json")) + ap.add_argument("--protocol-out", + default=os.path.join(HERE, "nmse_manifest_protocol_v1.json"), + help="path for the schema-conforming certifying manifest") + ap.add_argument("--no-protocol-v1", action="store_true", + help="skip emitting the schema-conforming manifest") + ap.add_argument("--seal", action="store_true", + help="attempt to seal: set seal_hash to FROZEN_HASH ONLY if " + "the live seal source matches it; else stays 'unsealed'") args = ap.parse_args() - manifest = run(args.samples, args.seed) + manifest, results, witness = run(args.samples, args.seed) with open(args.out, "w") as f: json.dump(manifest, f, indent=2) - print(f"\nmanifest written: {args.out}") + print(f"\nrich manifest written: {args.out}") + + if not args.no_protocol_v1: + pv1 = build_protocol_v1_manifest(results, witness, args.samples, + args.seed, args.seal) + ok, msg = validate_against_schema(pv1) + if not ok: + print("CERTIFYING MANIFEST INVALID:", msg) + sys.exit(1) + with open(args.protocol_out, "w") as f: + json.dump(pv1, f, indent=2) + print(f"certifying manifest written: {args.protocol_out}") + print(" seal_hash:", pv1["seal_hash"], "--", msg) if __name__ == "__main__": diff --git a/repro/numerics/nmse_manifest.json b/repro/numerics/nmse_manifest.json index 77cda5660..22afc1715 100644 --- a/repro/numerics/nmse_manifest.json +++ b/repro/numerics/nmse_manifest.json @@ -25,7 +25,7 @@ "python": "3.12.8", "platform": "Linux-6.1.158+-x86_64-with-glibc2.41" }, - "timestamp": "2026-06-14T11:32:13.412081+00:00", + "timestamp": "2026-06-14T11:47:38.901895+00:00", "headline": "ratio_GF16_over_BF16 is the protocol headline; <1 means GF16 closer to reference.", "results": { "D_NORM": { diff --git a/repro/numerics/nmse_manifest_protocol_v1.json b/repro/numerics/nmse_manifest_protocol_v1.json new file mode 100644 index 000000000..17d95641d --- /dev/null +++ b/repro/numerics/nmse_manifest_protocol_v1.json @@ -0,0 +1,53 @@ +{ + "schema_version": "1.0", + "protocol_version": "1.0", + "seal_hash": "unsealed", + "rng": { + "family": "pcg64", + "seed": "0x297a49" + }, + "samples_per_distribution": 2000000, + "bf16_subnormal_policy": "ieee", + "results": { + "D_NORM": { + "nmse_gf16": 1.7308965991785073e-07, + "nmse_bf16": 2.7641385183122016e-06, + "ratio": 0.06261974889143408 + }, + "D_LOG": { + "nmse_gf16": 1.476049303052953e-07, + "nmse_bf16": 2.351736425281605e-06, + "ratio": 0.06276423187501574 + }, + "D_RELU": { + "nmse_gf16": 1.7304317234439273e-07, + "nmse_bf16": 2.7642329105471395e-06, + "ratio": 0.06260079303886927 + }, + "D_PHI": { + "nmse_gf16": 1.7817398485573813e-07, + "nmse_bf16": 2.8478564272536918e-06, + "ratio": 0.06256424416295411 + }, + "D_DEEP": { + "nmse_gf16": 1.475567206889314e-07, + "nmse_bf16": 2.3698851004608678e-06, + "ratio": 0.06226323827270625 + } + }, + "identity_witness": { + "phi_squared_residual_f64": 0.0, + "trinity_residual_f64": 0.0 + }, + "runner": { + "host_arch": "x86_64-linux", + "compiler": "cpython 3.12.8" + }, + "timestamp": "2026-06-14T11:47:38.903013+00:00", + "x_extension": { + "seal_note": "seal not requested; host-only informational run", + "producer": "repro/numerics/nmse_gf16.py", + "codec_gf16": "conformance/gf16_ref.py (E6M9, BIAS=31, EXP_BITS=6, MANT_BITS=9)", + "note": "D_WIDE intentionally excluded (not a schema result key); see the rich manifest nmse_manifest.json for the full dynamic-range study." + } +} \ No newline at end of file diff --git a/repro/numerics/validate_manifest.py b/repro/numerics/validate_manifest.py new file mode 100644 index 000000000..456c7e29a --- /dev/null +++ b/repro/numerics/validate_manifest.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +""" +Validate an NMSE results manifest against schemas/nmse-protocol-v1.json. + +This is the conformance gate for the *certifying* manifest produced by +nmse_gf16.py --emit-protocol-v1. The rich, human-oriented manifest +(nmse_manifest.json) deliberately carries extra fields (D_WIDE, ULP, overflow +rates, etc.) and is NOT schema-bound; only the protocol-v1 manifest is. + +R5-HONEST: passing this validator only proves the manifest is well-formed and +that seal_hash is either the FROZEN_HASH digest or the literal "unsealed". It +does NOT certify silicon; an "unsealed" manifest is informational only. + +Usage: + python repro/numerics/validate_manifest.py [MANIFEST_JSON] +Exit 0 if the manifest conforms; non-zero (with a diagnostic) otherwise. + +Anchor: phi^2 + phi^-2 = 3 +""" +import json +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..")) +SCHEMA_PATH = os.path.join(REPO, "schemas", "nmse-protocol-v1.json") +DEFAULT_MANIFEST = os.path.join(HERE, "nmse_manifest_protocol_v1.json") +FROZEN_HASH_PATH = os.path.join(REPO, "bootstrap", "stage0", "FROZEN_HASH") + + +def frozen_digest(): + """Return the 64-hex digest recorded in bootstrap/stage0/FROZEN_HASH, or None.""" + try: + with open(FROZEN_HASH_PATH) as f: + tok = f.read().split()[0].strip() + if len(tok) == 64 and all(c in "0123456789abcdef" for c in tok): + return tok + except Exception: + pass + return None + + +def main(): + try: + import jsonschema + except Exception as exc: # pragma: no cover + print("ERROR: jsonschema is required:", exc) + return 2 + + manifest_path = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_MANIFEST + if not os.path.exists(manifest_path): + print("ERROR: manifest not found:", manifest_path) + return 2 + + with open(SCHEMA_PATH) as f: + schema = json.load(f) + with open(manifest_path) as f: + manifest = json.load(f) + + try: + jsonschema.validate(instance=manifest, schema=schema) + except jsonschema.ValidationError as err: + print("SCHEMA VIOLATION at", list(err.absolute_path), "->", err.message) + return 1 + + # Honesty cross-check: if seal_hash is not "unsealed", it must equal the + # recorded FROZEN_HASH digest. The schema accepts any 64-hex; we are stricter. + seal = manifest["seal_hash"] + if seal != "unsealed": + fd = frozen_digest() + if fd is None: + print("ERROR: seal_hash claims a seal but FROZEN_HASH is unreadable") + return 1 + if seal != fd: + print("HONESTY VIOLATION: seal_hash", seal[:12], "!= FROZEN_HASH", + (fd or "?")[:12]) + return 1 + print("seal_hash matches FROZEN_HASH digest (sealed manifest).") + else: + print("seal_hash = unsealed (informational manifest; not a silicon claim).") + + print("OK:", os.path.relpath(manifest_path, REPO), + "conforms to schemas/nmse-protocol-v1.json") + return 0 + + +if __name__ == "__main__": + sys.exit(main())