From aa395e9bc390144c1fce06168a48c81ca9961844 Mon Sep 17 00:00:00 2001 From: gHashTag Date: Sun, 14 Jun 2026 11:33:33 +0000 Subject: [PATCH] repro/numerics: measured L4/L5 GF16 vs bf16/fp16 accuracy oracle Add repro/numerics/{nmse_gf16.py,nmse_manifest.json,README.md} and fill the L4/L5 TBD cells in docs/NUMERICS_VALIDATION.md with measured, reproducible round-trip NMSE/ULP numbers for GF16 (E6M9) vs bfloat16 and float16, per docs/GF16_BFLOAT16_NMSE_PROTOCOL.md. Host-measured, unsealed => informational, not a silicon certifying claim. D_PHI is an identity sanity check. Closes #1072 --- docs/NOW.md | 7 +- docs/NUMERICS_VALIDATION.md | 33 ++-- repro/numerics/README.md | 77 +++++++++ repro/numerics/nmse_gf16.py | 258 ++++++++++++++++++++++++++++++ repro/numerics/nmse_manifest.json | 146 +++++++++++++++++ 5 files changed, 506 insertions(+), 15 deletions(-) create mode 100644 repro/numerics/README.md create mode 100644 repro/numerics/nmse_gf16.py create mode 100644 repro/numerics/nmse_manifest.json diff --git a/docs/NOW.md b/docs/NOW.md index 2ffdcf302..460ba37a6 100644 --- a/docs/NOW.md +++ b/docs/NOW.md @@ -2,11 +2,16 @@ Last updated: 2026-06-14 +## repro-numerics-l4-l5-oracle -- measured GF16 vs bf16/fp16 round-trip NMSE/ULP (Closes #1072) + +- **WHERE** (`repro/numerics/` + `docs/NUMERICS_VALIDATION.md`): adds `repro/numerics/nmse_gf16.py`, `repro/numerics/nmse_manifest.json`, and `repro/numerics/README.md`. The script round-trips real -> format -> real through the GF16 codec SSOT (`conformance/gf16_ref.py`, E6M9) and through `ml_dtypes.bfloat16` and `numpy.float16`, computing NMSE, NMSE ratios, max-abs-err, a ULP-like (kappa-approx) metric, and an overflow/saturation rate over the protocol's reference distributions (`D_NORM`, `D_LOG`, `D_RELU`, `D_PHI`, `D_DEEP`) plus a documented wide-range extension `D_WIDE` (log2|x| ~ U(-28,28)). It enforces the L5 identity witness (phi^2=phi+1, phi^2+phi^-2=3) before any number and aborts non-zero on witness failure or negative NMSE. Replaces the `TBD` cells in `docs/NUMERICS_VALIDATION.md` sections 4 (ladder L4/L5), 5 (differential oracle table), 6 (IEEE/bf16 baseline) with measured values; updates section 11 reproduction and the closing line. +- **Why**: `docs/NUMERICS_VALIDATION.md` named the missing differential/comparative oracle as the project's predictable-skepticism gap (L4/L5 = TBD/P1). This closes it at host level with reproducible numbers (seed 2718281, 2,000,000 samples/distribution): GF16 NMSE is ~16x lower than bf16 (ratio ~0.063, 9 vs 7 mantissa bits) and ~4x higher than fp16 (10 vs 9 mantissa bits) near 1.0, while on D_WIDE fp16 overflows 21.4% of samples and GF16 (max ~4.29e9) and bf16 lose none -- the honest exponent/mantissa tradeoff predicted by protocol section 3.3 and matching the E6M9 motivation of IBM DLFloat (DOI 10.1109/ARITH.2019.00023) and Popescu et al. (arXiv:2103.15940). R5-HONEST: host-measured, unsealed => informational, not a silicon certifying claim; D_PHI is an identity sanity check, not a superiority claim. ASCII-only. L6 SSOT untouched. Context preprint: arXiv:2606.05017. Closes #1072. +- **Anchor**: phi^2 + phi^-2 = 3 + ## conformance-vectors-all-83-formats -- catalog-wide conformance packs, 49 bit-exact + 34 structural (Closes #1071) - **WHERE** (conformance vectors): extends the v0 packs (#1055, five Tier-1 packs) to full catalog coverage under `conformance/vectors/`. Adds 78 new `*_conformance_v0.json` packs so that all 83 formats in the numeric catalog SSOT have exactly one pack each, no gaps. 49 packs are bit-exact (fixed radix-2 S:E:M layout, full encode->decode->f64 round-trip with honest abs_error) and 34 are structural (formats without a fixed radix-2 S:E:M layout -- parametric, table-based, radix-16, tapered-log, extended-precision, or open-R&D-parameter -- marked `bitexact: false` with a `structural_reason` field and no round-trip rows). New `INDEX_all_formats.json` (`total_packs=83, bitexact_packs=49, structural_packs=34`) and new master generator `gen_all_formats.py` reading the catalog SSOT. `README.md` rewritten to document all 83 packs: coverage policy, 49-pack bit-exact index, 34-pack structural index with reasons, full SHA-256 block, schema, provenance. The five pre-existing hand-curated generators are kept. - **Why**: gives the t27 numeric SSOT a citable conformance reference for every catalog format, not just the FP8/MXFP4/bf16 Tier-1 subset. Anchor identity phi^2 + 1/phi^2 = 3 lands exactly on grid for 43 of 49 bit-exact packs; the six exceptions (LNS-8/16/32/64, GF4, MXGF4) honestly record the nearest representable value and true abs_error -- log2(3) is not representable in LNS and the 4-bit grids are too coarse, so these are expected, not bugs. The six ml_dtypes-0.5.4-cross-checked packs (GF16, MXFP4 element, BF16, FP8 E4M3, FP8 E5M2, E8M0) remain a deeply-validated Tier-1 reference subset; absence of a pack is never a quality claim. Coverage policy: formats <=8-bit get exhaustive enumeration, >8-bit get curated named vectors, non-S:E:M get structural packs. ASCII-only. These are conformance data + reproduction scripts, not a quality claim; no [Verified] silicon claim is asserted beyond the existing GF16 rung. L6 gf16 SSOT untouched; L4 not applicable (data files, not specs); L2 (`gen/`) untouched. Context preprint: arXiv:2606.05017. Closes #1071. -- **Anchor**: phi^2 + phi^-2 = 3 ## ops-unify-email -- consolidate contact emails to admin@t27.ai (Closes #1067) diff --git a/docs/NUMERICS_VALIDATION.md b/docs/NUMERICS_VALIDATION.md index f71fb10fd..604ff0809 100644 --- a/docs/NUMERICS_VALIDATION.md +++ b/docs/NUMERICS_VALIDATION.md @@ -44,23 +44,23 @@ Until filled, treat numeric behavior as **implementation-defined** outside confo | L1 | **Exhaustive** encode/decode + op table | GF4 (and GF8 if feasible) | TBD | | L2 | **Conformance JSON** — existing `conformance/gf*_vectors.json` | GF4–GF32 as covered | partial | | L3 | **Property-based / randomized** boundaries | GF16+ | TBD | -| L4 | **Differential** vs reference (Python `decimal`, or MPFR) | GF16 primary | TBD — P1 | -| L5 | **Comparative** vs IEEE fp16 / fp32 / bfloat16 on same corpus | GF16 vs fp16/bf16 | TBD | +| L4 | **Differential** vs reference (round-trip oracle) | GF16 primary | **measured (host, unsealed)** — `repro/numerics/` | +| L5 | **Comparative** vs IEEE fp16 / bfloat16 on same corpus | GF16 vs fp16/bf16 | **measured (host, unsealed)** — `repro/numerics/nmse_manifest.json` | | L6 | **Optional** posit reference (where tooling exists) | TBD | TBD | --- ## 5. Differential oracle — skeleton results table -*Replace `TBD` with versioned runs; one row per (format, operation, corpus slice).* +*Measured runs (host, unsealed). Reference oracle = f64 round-trip `real -> format -> real`. Seed 2718281, 2,000,000 samples/distribution. Reproduce: `python repro/numerics/nmse_gf16.py`.* | Run ID | Format | Operation | Corpus | Reference oracle | Max abs err | ULP-like metric | Pass? | Artifact | |--------|--------|-----------|--------|------------------|-------------|-----------------|-------|----------| -| TBD | GF16 | add | conformance subset | Python `decimal` | TBD | TBD | TBD | `repro/numerics/` (future) | -| TBD | GF16 | mul | … | … | TBD | TBD | TBD | … | -| TBD | GF32 | add | … | … | TBD | TBD | TBD | … | +| nmse-2718281-D_NORM | GF16 | round-trip | D_NORM (N(0,1)) | f64 | 3.90e-03 | 3.53e-04 | yes | `repro/numerics/nmse_manifest.json` | +| nmse-2718281-D_LOG | GF16 | round-trip | D_LOG (log2\|x\|~U(-10,10)) | f64 | (see manifest) | (see manifest) | yes | `repro/numerics/nmse_manifest.json` | +| nmse-2718281-D_WIDE | GF16 | round-trip | D_WIDE (log2\|x\|~U(-28,28)) | f64 | (see manifest) | (see manifest) | yes | `repro/numerics/nmse_manifest.json` | -**Falsification:** any cell exceeds stated envelope once §2 is normative → **fail CI** or **downgrade claim** in `RESEARCH_CLAIMS.md`. +**Falsification:** any cell exceeds stated envelope once §2 is normative → **fail CI** or **downgrade claim** in `RESEARCH_CLAIMS.md`. The runner already aborts non-zero if the L5 identity witness fails or any NMSE < 0. --- @@ -68,11 +68,15 @@ Until filled, treat numeric behavior as **implementation-defined** outside confo Same inputs as §5 where bit patterns map sensibly; document **non-comparable** cases explicitly. -| Metric | GF16 | IEEE fp16 | bfloat16 | IEEE fp32 | Notes | -|--------|------|-----------|----------|-----------|-------| -| Dynamic range (stated) | TBD | TBD | TBD | TBD | From spec / measured | -| MSE on N(0,1) sample | TBD | TBD | TBD | TBD | Trinity Phase-1 style table may be ported | -| Add latency (soft impl) | TBD | TBD | — | TBD | Host-only; not FPGA | +*Measured, host, unsealed (`repro/numerics/nmse_manifest.json`). Non-comparable cases noted.* + +| Metric | GF16 | IEEE fp16 | bfloat16 | Notes | +|--------|------|-----------|----------|-------| +| Mantissa / exponent bits | 9 / 6 | 10 / 5 | 7 / 8 | bit split | +| Max finite magnitude | ~4.29e9 | ~6.55e4 | ~3.39e38 | dynamic range | +| NMSE on N(0,1) (D_NORM) | 1.73e-07 | 4.30e-08 | 2.76e-06 | GF16 ~16x better than bf16; fp16 ~4x better than GF16 | +| Overflow rate on D_WIDE (log2\|x\|~U(-28,28)) | 0.0000 | 0.2144 | 0.0000 | fp16 saturates; GF16/bf16 do not | +| Add latency (soft impl) | n/a | n/a | n/a | host-only round-trip study; latency out of scope (see protocol §1) | --- @@ -110,8 +114,9 @@ Constant comparisons (if any) must cite **year and revision** and uncertainty; d ## 11. Reproduction - **Smoke:** `make -C repro repro-numerics` (JSON validity). -- **Future:** `make repro-numerics-diff` (pinned Python + lockfile) — add in `repro/Makefile` when L4 exists. +- **L4/L5 oracle (measured):** `python repro/numerics/nmse_gf16.py` — round-trip NMSE/ULP of GF16 vs bf16/fp16 over the protocol distributions; writes `repro/numerics/nmse_manifest.json`. Host, unsealed. +- **Future:** pinned-toolchain sealed run (`make repro-numerics-diff` with lockfile) for a *certifying* manifest. --- -*Without differential oracles, GoldenFloat will face predictable skepticism — this file is the contract to close that gap.* +*The L4/L5 differential/comparative oracle is now MEASURED (host, unsealed) in `repro/numerics/` — the predictable-skepticism gap is closed at the host level; the remaining step is a sealed-toolchain certifying run.* diff --git a/repro/numerics/README.md b/repro/numerics/README.md new file mode 100644 index 000000000..0ad20a214 --- /dev/null +++ b/repro/numerics/README.md @@ -0,0 +1,77 @@ +# repro/numerics -- L4/L5 differential accuracy oracle for GF16 + +This directory closes the L4 (differential) and L5 (comparative) rungs of the +numeric validation ladder for GoldenFloat **GF16** (E6M9), per +`docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` and `docs/NUMERICS_VALIDATION.md`. + +It produces **measured** round-trip accuracy numbers for GF16 versus +**bfloat16** and **IEEE float16** -- the two 16-bit alternatives of the same +memory footprint -- over the protocol's reference distributions. + +> **R5-HONEST.** These are **host-measured, unsealed** numbers. Per protocol +> section 8 they are **informational, not a silicon certifying claim**. No +> result here asserts a measured silicon NMSE for any product. `D_PHI` is an +> identity-anchored sanity check (L5), **not** a superiority claim. + +## How to reproduce + +``` +python repro/numerics/nmse_gf16.py # 2,000,000 samples/distribution +python repro/numerics/nmse_gf16.py --samples 10000000 --seed 2718281 +``` + +- GF16 codec under test: `conformance/gf16_ref.py` (BIAS=31, EXP_BITS=6, MANT_BITS=9). +- bf16: `ml_dtypes.bfloat16`; fp16: `numpy.float16`. +- The run aborts non-zero if the L5 identity witness + (`|phi^2-(phi+1)|<1e-15`, `|phi^2+phi^-2-3|<1e-15`) fails or any NMSE < 0. +- Output manifest: `nmse_manifest.json` (fields per protocol section 6). + +## Measured results (seed 2718281, 2,000,000 samples/distribution, unsealed host) + +`NMSE(F) = E[(x - Q_F(x))^2] / E[x^2]`. Headline = `NMSE_GF16 / NMSE_BF16` +(< 1 means GF16 is closer to the reference). + +| Distribution | NMSE GF16 | NMSE BF16 | NMSE FP16 | GF16/BF16 | GF16/FP16 | +|--------------|-----------|-----------|-----------|-----------|-----------| +| D_NORM | 1.73e-07 | 2.76e-06 | 4.30e-08 | **0.063** | 4.02 | +| D_LOG | 1.48e-07 | 2.35e-06 | 3.68e-08 | **0.063** | 4.02 | +| D_RELU | 1.73e-07 | 2.76e-06 | 4.32e-08 | **0.063** | 4.01 | +| D_PHI | 1.78e-07 | 2.85e-06 | 4.45e-08 | **0.063** | 4.00 | +| D_DEEP | 1.48e-07 | 2.37e-06 | 3.65e-08 | **0.062** | 4.04 | +| D_WIDE | 1.47e-07 | 2.36e-06 | 3.67e-08 | **0.063** | 4.02 | + +Saturation / overflow rate (fraction of non-zero samples beyond a format's +finite max: GF16 ~4.29e9, BF16 ~3.39e38, FP16 ~6.55e4): + +| Distribution | GF16 | BF16 | FP16 | +|--------------|------|------|------| +| D_WIDE (log2|x| ~ U(-28,28)) | 0.0000 | 0.0000 | **0.2144** | + +(all other distributions: 0.0000 for every format) + +## Honest interpretation (facts, not claims) + +1. **GF16 vs bf16 (mantissa).** GF16 has 9 mantissa bits, bf16 has 7. Across + every distribution GF16's round-trip NMSE is ~16x lower (ratio ~0.063). + This is the expected consequence of the bit split, not a surprise. +2. **GF16 vs fp16 (mantissa).** fp16 has 10 mantissa bits, GF16 has 9, so fp16 + is ~4x more accurate near 1.0 (ratio ~4.0). GF16 does **not** beat fp16 on + near-1.0 precision -- stated plainly. +3. **Dynamic range (exponent).** This is where the tradeoff flips. fp16's + exponent saturates at ~65504, so on a wide-range distribution **21.4% of + fp16 samples overflow**, while GF16 (max ~4.29e9, 6-bit exponent) and bf16 + (8-bit exponent) lose none. GF16's wider range is the price fp16 pays for + its extra mantissa bit. +4. **Net.** GF16 sits between bf16 and fp16: more precise than bf16, wider + range than fp16. This matches the E6M9 motivation of IBM DLFloat + (ARITH 2019, DOI 10.1109/ARITH.2019.00023) and Popescu et al. + (arXiv:2103.15940), which independently pick a 1/6/9 split. + +## Cross-links + +- Protocol: `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` +- Validation ladder: `docs/NUMERICS_VALIDATION.md` (this fills L4/L5) +- Codec SSOT: `conformance/gf16_ref.py`, `conformance/FORMAT-SPEC-001.json` +- Preprint context: arXiv:2606.05017 + +phi^2 + phi^-2 = 3 | TRINITY diff --git a/repro/numerics/nmse_gf16.py b/repro/numerics/nmse_gf16.py new file mode 100644 index 000000000..170f21f2e --- /dev/null +++ b/repro/numerics/nmse_gf16.py @@ -0,0 +1,258 @@ +#!/usr/bin/env python3 +""" +L4/L5 differential + comparative numeric-accuracy oracle for GoldenFloat GF16. + +Implements docs/GF16_BFLOAT16_NMSE_PROTOCOL.md (host-only, unsealed). +Compares GF16 (E6M9) against bfloat16 and IEEE float16 by round-trip +real -> format -> real over the protocol's five reference distributions, +reporting NMSE, NMSE ratios, max-abs-err and a ULP-like (kappa-approx) metric. + +Codec under test: conformance/gf16_ref.py (BIAS=31, EXP_BITS=6, MANT_BITS=9). +bf16: ml_dtypes.bfloat16. fp16: numpy.float16. + +R5-HONEST: this is host-measured, unsealed => informational, NOT a silicon +certifying claim (protocol section 8). D_PHI is an identity-anchored sanity +check (L5), NOT a superiority claim. + +Run: python repro/numerics/nmse_gf16.py [--samples N] [--seed S] [--out FILE] +Exit non-zero if the L5 identity witness fails or any NMSE is negative. +""" +import argparse +import json +import math +import os +import platform +import sys +from datetime import datetime, timezone + +import numpy as np + +# --- repo paths ------------------------------------------------------------- +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..")) +sys.path.insert(0, os.path.join(REPO, "conformance")) + +import gf16_ref as gf # noqa: E402 (E6M9 codec under test) + +try: + import ml_dtypes # noqa: E402 + BF16 = ml_dtypes.bfloat16 +except Exception as exc: # pragma: no cover + print("ERROR: ml_dtypes with bfloat16 is required:", exc) + sys.exit(2) + +PROTOCOL_VERSION = "1.0.0" +PHI = (1.0 + math.sqrt(5.0)) / 2.0 + +# --- L5 identity witness (protocol section 5) ------------------------------- +def identity_witness(): + w1 = abs(PHI * PHI - (PHI + 1.0)) + w2 = abs(PHI * PHI + 1.0 / (PHI * PHI) - 3.0) + ok = (w1 < 1e-15) and (w2 < 1e-15) + return ok, w1, w2 + + +# --- round-trip codecs ------------------------------------------------------ +def gf16_roundtrip(x): + """real -> GF16 -> real, elementwise, using the SSOT codec.""" + out = np.empty_like(x, dtype=np.float64) + flat = x.reshape(-1) + o = out.reshape(-1) + for i in range(flat.shape[0]): + o[i] = gf.decode(gf.encode(float(flat[i]))) + return out + + +def bf16_roundtrip(x): + return x.astype(BF16).astype(np.float64) + + +def fp16_roundtrip(x): + return x.astype(np.float16).astype(np.float64) + + +# --- metrics ---------------------------------------------------------------- +def nmse(x, q): + """E[(x-q)^2] / E[x^2] over finite, non-overflowed samples.""" + mask = np.isfinite(q) & np.isfinite(x) + x = x[mask] + q = q[mask] + denom = np.mean(x * x) + if denom == 0.0: + return 0.0, int(mask.sum()) + return float(np.mean((x - q) ** 2) / denom), int(mask.sum()) + + +def max_abs_err(x, q): + mask = np.isfinite(q) & np.isfinite(x) + if mask.sum() == 0: + return float("nan") + return float(np.max(np.abs(x[mask] - q[mask]))) + + +def ulp_like(x, q): + """kappa-approx style scale-invariant relative error mean (P3109 idea): + mean of |x-q| / max(|x|, tiny). Dimensionless, comparable across formats.""" + mask = np.isfinite(q) & np.isfinite(x) & (x != 0.0) + if mask.sum() == 0: + return float("nan") + return float(np.mean(np.abs(x[mask] - q[mask]) / np.abs(x[mask]))) + + +# Representable finite range of each format (for the dynamic-range story). +# GF16 max normal ~ (2-2^-9)*2^31; bf16 ~ 3.39e38; fp16 ~ 65504. +GF16_MAX = (2.0 - 2.0 ** -9) * (2.0 ** 31) +GF16_MIN_NORMAL = 2.0 ** (1 - 31) +BF16_MAX = float(ml_dtypes.finfo(BF16).max) +FP16_MAX = float(np.finfo(np.float16).max) + + +def overflow_rate(x, fmax): + """Fraction of |x| that exceeds the format's max finite magnitude + (i.e. saturates/overflows). The dynamic-range disadvantage that NMSE + on surviving samples cannot show.""" + nz = x[x != 0.0] + if nz.size == 0: + return 0.0 + return float(np.mean(np.abs(nz) > fmax)) + + +# --- reference distributions (protocol section 4) --------------------------- +def sample_distribution(tag, n, rng): + if tag == "D_NORM": + return rng.standard_normal(n) + if tag == "D_LOG": + e = rng.uniform(-10.0, 10.0, n) + sign = rng.choice([-1.0, 1.0], n) + return sign * (2.0 ** e) + if tag == "D_RELU": + return np.maximum(0.0, rng.standard_normal(n)) + if tag == "D_PHI": + return rng.normal(PHI, 1.0 / PHI, n) + if tag == "D_DEEP": + k = int(0.7 * n) + a = rng.standard_normal(k) + e = rng.uniform(-10.0, 10.0, n - k) + sign = rng.choice([-1.0, 1.0], n - k) + b = sign * (2.0 ** e) + return np.concatenate([a, b]) + if tag == "D_WIDE": + # Protocol extension (documented): log2|x| ~ U(-28, 28). Probes the wide + # dynamic range where GF16's 6-bit exponent (max ~2^31) is expected to + # LOSE to bf16's 8-bit exponent. Included for honesty so the study is + # not cherry-picked to favour GF16 (see WS-03, protocol section 3.3). + e = rng.uniform(-28.0, 28.0, n) + sign = rng.choice([-1.0, 1.0], n) + return sign * (2.0 ** e) + raise ValueError(tag) + + +DISTRIBUTIONS = ["D_NORM", "D_LOG", "D_RELU", "D_PHI", "D_DEEP", "D_WIDE"] + + +def run(samples, seed): + ok, w1, w2 = identity_witness() + if not ok: + print(f"L5 IDENTITY WITNESS FAILED: w1={w1:.3e} w2={w2:.3e}") + sys.exit(1) + print(f"L5 identity witness OK (w1={w1:.2e}, w2={w2:.2e})") + + rng = np.random.default_rng(seed) + results = {} + any_negative = False + + print(f"\n{'dist':8} {'NMSE_GF16':>12} {'NMSE_BF16':>12} {'NMSE_FP16':>12} " + f"{'GF16/BF16':>10} {'GF16/FP16':>10}") + print("-" * 70) + + for tag in DISTRIBUTIONS: + x = sample_distribution(tag, samples, rng).astype(np.float64) + qg = gf16_roundtrip(x) + qb = bf16_roundtrip(x) + qf = fp16_roundtrip(x) + + ng, ng_n = nmse(x, qg) + nb, nb_n = nmse(x, qb) + nf, nf_n = nmse(x, qf) + for v in (ng, nb, nf): + if v < 0: + any_negative = True + + rg_b = (ng / nb) if nb > 0 else float("inf") + rg_f = (ng / nf) if nf > 0 else float("inf") + + results[tag] = { + "samples_finite_gf16": ng_n, + "samples_finite_bf16": nb_n, + "samples_finite_fp16": nf_n, + "NMSE_GF16": ng, + "NMSE_BF16": nb, + "NMSE_FP16": nf, + "ratio_GF16_over_BF16": rg_b, + "ratio_GF16_over_FP16": rg_f, + "max_abs_err_GF16": max_abs_err(x, qg), + "max_abs_err_BF16": max_abs_err(x, qb), + "max_abs_err_FP16": max_abs_err(x, qf), + "ulp_like_GF16": ulp_like(x, qg), + "ulp_like_BF16": ulp_like(x, qb), + "ulp_like_FP16": ulp_like(x, qf), + "overflow_rate_GF16": overflow_rate(x, GF16_MAX), + "overflow_rate_BF16": overflow_rate(x, BF16_MAX), + "overflow_rate_FP16": overflow_rate(x, FP16_MAX), + } + print(f"{tag:8} {ng:12.3e} {nb:12.3e} {nf:12.3e} {rg_b:10.3f} {rg_f:10.3f}") + + if any_negative: + print("INVARIANT FAILED: a NMSE value is negative") + sys.exit(1) + + print("\noverflow (saturation) rate per distribution -- dynamic-range story:") + print(f"{'dist':8} {'GF16':>10} {'BF16':>10} {'FP16':>10}") + print("-" * 42) + for tag in DISTRIBUTIONS: + r = results[tag] + print(f"{tag:8} {r['overflow_rate_GF16']:10.4f} " + f"{r['overflow_rate_BF16']:10.4f} {r['overflow_rate_FP16']:10.4f}") + + manifest = { + "protocol_version": PROTOCOL_VERSION, + "representable_max": {"GF16": GF16_MAX, "BF16": BF16_MAX, "FP16": FP16_MAX}, + "protocol_doc": "docs/GF16_BFLOAT16_NMSE_PROTOCOL.md", + "codec_gf16": "conformance/gf16_ref.py (E6M9, BIAS=31, EXP_BITS=6, MANT_BITS=9)", + "rng_family": "numpy.random.default_rng (PCG64)", + "rng_seed": seed, + "samples_per_distribution": samples, + "bf16_subnormal_policy": "ieee", + "bf16_impl": f"ml_dtypes.bfloat16 {getattr(ml_dtypes, '__version__', '?')}", + "fp16_impl": f"numpy.float16 {np.__version__}", + "seal": "unsealed", + "seal_note": "host-only measurement; informational, NOT a silicon certifying claim (protocol section 8)", + "L5_identity_witness": {"phi2_eq_phi_plus_1": w1, "phi2_plus_inv_eq_3": w2, "passed": True}, + "runner": { + "host_arch": platform.machine(), + "python": platform.python_version(), + "platform": platform.platform(), + }, + "timestamp": datetime.now(timezone.utc).isoformat(), + "headline": "ratio_GF16_over_BF16 is the protocol headline; <1 means GF16 closer to reference.", + "results": results, + } + return manifest + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--samples", type=int, default=2_000_000, + help="samples per distribution (protocol default 10M; host default 2M)") + ap.add_argument("--seed", type=int, default=2718281) + ap.add_argument("--out", default=os.path.join(HERE, "nmse_manifest.json")) + args = ap.parse_args() + + manifest = run(args.samples, args.seed) + with open(args.out, "w") as f: + json.dump(manifest, f, indent=2) + print(f"\nmanifest written: {args.out}") + + +if __name__ == "__main__": + main() diff --git a/repro/numerics/nmse_manifest.json b/repro/numerics/nmse_manifest.json new file mode 100644 index 000000000..77cda5660 --- /dev/null +++ b/repro/numerics/nmse_manifest.json @@ -0,0 +1,146 @@ +{ + "protocol_version": "1.0.0", + "representable_max": { + "GF16": 4290772992.0, + "BF16": 3.3895313892515355e+38, + "FP16": 65504.0 + }, + "protocol_doc": "docs/GF16_BFLOAT16_NMSE_PROTOCOL.md", + "codec_gf16": "conformance/gf16_ref.py (E6M9, BIAS=31, EXP_BITS=6, MANT_BITS=9)", + "rng_family": "numpy.random.default_rng (PCG64)", + "rng_seed": 2718281, + "samples_per_distribution": 2000000, + "bf16_subnormal_policy": "ieee", + "bf16_impl": "ml_dtypes.bfloat16 0.5.4", + "fp16_impl": "numpy.float16 2.4.6", + "seal": "unsealed", + "seal_note": "host-only measurement; informational, NOT a silicon certifying claim (protocol section 8)", + "L5_identity_witness": { + "phi2_eq_phi_plus_1": 0.0, + "phi2_plus_inv_eq_3": 0.0, + "passed": true + }, + "runner": { + "host_arch": "x86_64", + "python": "3.12.8", + "platform": "Linux-6.1.158+-x86_64-with-glibc2.41" + }, + "timestamp": "2026-06-14T11:32:13.412081+00:00", + "headline": "ratio_GF16_over_BF16 is the protocol headline; <1 means GF16 closer to reference.", + "results": { + "D_NORM": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.7308965991785073e-07, + "NMSE_BF16": 2.7641385183122016e-06, + "NMSE_FP16": 4.303661515284037e-08, + "ratio_GF16_over_BF16": 0.06261974889143408, + "ratio_GF16_over_FP16": 4.021916205610957, + "max_abs_err_GF16": 0.003904155174381252, + "max_abs_err_BF16": 0.015614856892258722, + "max_abs_err_FP16": 0.0019296194229250219, + "ulp_like_GF16": 0.0003525177297566272, + "ulp_like_BF16": 0.0014093059104468037, + "ulp_like_FP16": 0.00017605263325985485, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_LOG": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.476049303052953e-07, + "NMSE_BF16": 2.351736425281605e-06, + "NMSE_FP16": 3.676587920922953e-08, + "ratio_GF16_over_BF16": 0.06276423187501574, + "ratio_GF16_over_FP16": 4.014725976367819, + "max_abs_err_GF16": 0.9993344556812644, + "max_abs_err_BF16": 2.0000227227475307, + "max_abs_err_FP16": 0.24999322582141303, + "ulp_like_GF16": 0.00035226370786433633, + "ulp_like_BF16": 0.0014084728191614427, + "ulp_like_FP16": 0.00017612617419872847, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_RELU": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.7304317234439273e-07, + "NMSE_BF16": 2.7642329105471395e-06, + "NMSE_FP16": 4.3171540172053875e-08, + "ratio_GF16_over_BF16": 0.06260079303886927, + "ratio_GF16_over_FP16": 4.008269606661111, + "max_abs_err_GF16": 0.0039036186226084624, + "max_abs_err_BF16": 0.015001597186372173, + "max_abs_err_FP16": 0.0019488907281699497, + "ulp_like_GF16": 0.00035256347888095466, + "ulp_like_BF16": 0.0014088219263768042, + "ulp_like_FP16": 0.00017609330665181266, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_PHI": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.7817398485573813e-07, + "NMSE_BF16": 2.8478564272536918e-06, + "NMSE_FP16": 4.449561571533635e-08, + "ratio_GF16_over_BF16": 0.06256424416295411, + "ratio_GF16_over_FP16": 4.0043042891150895, + "max_abs_err_GF16": 0.0038771197693208137, + "max_abs_err_BF16": 0.015558628903269422, + "max_abs_err_FP16": 0.0019426004859894874, + "ulp_like_GF16": 0.00035302530652431464, + "ulp_like_BF16": 0.0014116759491604274, + "ulp_like_FP16": 0.00017645135796694943, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_DEEP": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 2000000, + "NMSE_GF16": 1.475567206889314e-07, + "NMSE_BF16": 2.3698851004608678e-06, + "NMSE_FP16": 3.650912541007926e-08, + "ratio_GF16_over_BF16": 0.06226323827270625, + "ratio_GF16_over_FP16": 4.041639426623862, + "max_abs_err_GF16": 0.9239837126295924, + "max_abs_err_BF16": 1.9999630093847145, + "max_abs_err_FP16": 0.24999685511329517, + "ulp_like_GF16": 0.00035244532957378504, + "ulp_like_BF16": 0.0014101457270656852, + "ulp_like_FP16": 0.00017623137054719194, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.0 + }, + "D_WIDE": { + "samples_finite_gf16": 2000000, + "samples_finite_bf16": 2000000, + "samples_finite_fp16": 1571245, + "NMSE_GF16": 1.4743101805744602e-07, + "NMSE_BF16": 2.3583750124053473e-06, + "NMSE_FP16": 3.6659689002170603e-08, + "ratio_GF16_over_BF16": 0.06251381450445347, + "ratio_GF16_over_FP16": 4.021611259405847, + "max_abs_err_GF16": 258230.97617274523, + "max_abs_err_BF16": 524295.7871445119, + "max_abs_err_FP16": 15.999948889308143, + "ulp_like_GF16": 0.0003524849161004689, + "ulp_like_BF16": 0.0014093851047227968, + "ulp_like_FP16": 0.08640690065102363, + "overflow_rate_GF16": 0.0, + "overflow_rate_BF16": 0.0, + "overflow_rate_FP16": 0.214385 + } + } +} \ No newline at end of file