|
| 1 | +# repro/numerics -- L4/L5 differential accuracy oracle for GF16 |
| 2 | + |
| 3 | +This directory closes the L4 (differential) and L5 (comparative) rungs of the |
| 4 | +numeric validation ladder for GoldenFloat **GF16** (E6M9), per |
| 5 | +`docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` and `docs/NUMERICS_VALIDATION.md`. |
| 6 | + |
| 7 | +It produces **measured** round-trip accuracy numbers for GF16 versus |
| 8 | +**bfloat16** and **IEEE float16** -- the two 16-bit alternatives of the same |
| 9 | +memory footprint -- over the protocol's reference distributions. |
| 10 | + |
| 11 | +> **R5-HONEST.** These are **host-measured, unsealed** numbers. Per protocol |
| 12 | +> section 8 they are **informational, not a silicon certifying claim**. No |
| 13 | +> result here asserts a measured silicon NMSE for any product. `D_PHI` is an |
| 14 | +> identity-anchored sanity check (L5), **not** a superiority claim. |
| 15 | +
|
| 16 | +## How to reproduce |
| 17 | + |
| 18 | +``` |
| 19 | +python repro/numerics/nmse_gf16.py # 2,000,000 samples/distribution |
| 20 | +python repro/numerics/nmse_gf16.py --samples 10000000 --seed 2718281 |
| 21 | +``` |
| 22 | + |
| 23 | +- GF16 codec under test: `conformance/gf16_ref.py` (BIAS=31, EXP_BITS=6, MANT_BITS=9). |
| 24 | +- bf16: `ml_dtypes.bfloat16`; fp16: `numpy.float16`. |
| 25 | +- The run aborts non-zero if the L5 identity witness |
| 26 | + (`|phi^2-(phi+1)|<1e-15`, `|phi^2+phi^-2-3|<1e-15`) fails or any NMSE < 0. |
| 27 | +- Output manifests: |
| 28 | + - `nmse_manifest.json` -- the **rich** manifest (fields per protocol section 6: |
| 29 | + all six distributions including `D_WIDE`, ULP-like metric, overflow rates, |
| 30 | + representable-max table). Intentionally NOT schema-bound. |
| 31 | + - `nmse_manifest_protocol_v1.json` -- the **certifying** manifest, strictly |
| 32 | + conforming to `schemas/nmse-protocol-v1.json` (`additionalProperties:false`; |
| 33 | + only the five schema distributions `D_NORM/D_LOG/D_RELU/D_PHI/D_DEEP`; each |
| 34 | + result is `nmse_gf16`/`nmse_bf16`/`ratio`). `D_WIDE` is deliberately absent. |
| 35 | + |
| 36 | +## Sealing and the certifying manifest (R5-HONEST) |
| 37 | + |
| 38 | +The certifying manifest carries a `seal_hash` field. It is set to the 64-hex |
| 39 | +SHA-256 in `bootstrap/stage0/FROZEN_HASH` **only** when `--seal` is passed AND |
| 40 | +the live seal source (`bootstrap/src/compiler.rs`) hashes to exactly that |
| 41 | +digest under a pinned toolchain. In every other case it stays the literal |
| 42 | +`"unsealed"`. The script never fabricates a seal: a host run on an unpinned |
| 43 | +tree is honestly `unsealed` and informational only. |
| 44 | + |
| 45 | +``` |
| 46 | +python repro/numerics/nmse_gf16.py --seal # seals iff source matches FROZEN_HASH |
| 47 | +python repro/numerics/validate_manifest.py # re-validate the committed manifest |
| 48 | +make -C repro repro-numerics-certify # CI-facing validation step |
| 49 | +``` |
| 50 | + |
| 51 | +`validate_manifest.py` validates the manifest against the schema AND enforces |
| 52 | +the honesty rule: a non-`unsealed` `seal_hash` must equal the FROZEN_HASH |
| 53 | +digest, or it exits non-zero. As committed, `seal_hash = "unsealed"` because |
| 54 | +this manifest was produced on an unpinned host (the live compiler source does |
| 55 | +not match the frozen digest) -- exactly the state protocol section 8 calls |
| 56 | +informational, not a silicon certifying claim. |
| 57 | + |
| 58 | +## Measured results (seed 2718281, 2,000,000 samples/distribution, unsealed host) |
| 59 | + |
| 60 | +`NMSE(F) = E[(x - Q_F(x))^2] / E[x^2]`. Headline = `NMSE_GF16 / NMSE_BF16` |
| 61 | +(< 1 means GF16 is closer to the reference). |
| 62 | + |
| 63 | +| Distribution | NMSE GF16 | NMSE BF16 | NMSE FP16 | GF16/BF16 | GF16/FP16 | |
| 64 | +|--------------|-----------|-----------|-----------|-----------|-----------| |
| 65 | +| D_NORM | 1.73e-07 | 2.76e-06 | 4.30e-08 | **0.063** | 4.02 | |
| 66 | +| D_LOG | 1.48e-07 | 2.35e-06 | 3.68e-08 | **0.063** | 4.02 | |
| 67 | +| D_RELU | 1.73e-07 | 2.76e-06 | 4.32e-08 | **0.063** | 4.01 | |
| 68 | +| D_PHI | 1.78e-07 | 2.85e-06 | 4.45e-08 | **0.063** | 4.00 | |
| 69 | +| D_DEEP | 1.48e-07 | 2.37e-06 | 3.65e-08 | **0.062** | 4.04 | |
| 70 | +| D_WIDE | 1.47e-07 | 2.36e-06 | 3.67e-08 | **0.063** | 4.02 | |
| 71 | + |
| 72 | +Saturation / overflow rate (fraction of non-zero samples beyond a format's |
| 73 | +finite max: GF16 ~4.29e9, BF16 ~3.39e38, FP16 ~6.55e4): |
| 74 | + |
| 75 | +| Distribution | GF16 | BF16 | FP16 | |
| 76 | +|--------------|------|------|------| |
| 77 | +| D_WIDE (log2|x| ~ U(-28,28)) | 0.0000 | 0.0000 | **0.2144** | |
| 78 | + |
| 79 | +(all other distributions: 0.0000 for every format) |
| 80 | + |
| 81 | +## Honest interpretation (facts, not claims) |
| 82 | + |
| 83 | +1. **GF16 vs bf16 (mantissa).** GF16 has 9 mantissa bits, bf16 has 7. Across |
| 84 | + every distribution GF16's round-trip NMSE is ~16x lower (ratio ~0.063). |
| 85 | + This is the expected consequence of the bit split, not a surprise. |
| 86 | +2. **GF16 vs fp16 (mantissa).** fp16 has 10 mantissa bits, GF16 has 9, so fp16 |
| 87 | + is ~4x more accurate near 1.0 (ratio ~4.0). GF16 does **not** beat fp16 on |
| 88 | + near-1.0 precision -- stated plainly. |
| 89 | +3. **Dynamic range (exponent).** This is where the tradeoff flips. fp16's |
| 90 | + exponent saturates at ~65504, so on a wide-range distribution **21.4% of |
| 91 | + fp16 samples overflow**, while GF16 (max ~4.29e9, 6-bit exponent) and bf16 |
| 92 | + (8-bit exponent) lose none. GF16's wider range is the price fp16 pays for |
| 93 | + its extra mantissa bit. |
| 94 | +4. **Net.** GF16 sits between bf16 and fp16: more precise than bf16, wider |
| 95 | + range than fp16. This matches the E6M9 motivation of IBM DLFloat |
| 96 | + (ARITH 2019, DOI 10.1109/ARITH.2019.00023) and Popescu et al. |
| 97 | + (arXiv:2103.15940), which independently pick a 1/6/9 split. |
| 98 | + |
| 99 | +## Cross-links |
| 100 | + |
| 101 | +- Protocol: `docs/GF16_BFLOAT16_NMSE_PROTOCOL.md` |
| 102 | +- Validation ladder: `docs/NUMERICS_VALIDATION.md` (this fills L4/L5) |
| 103 | +- Codec SSOT: `conformance/gf16_ref.py`, `conformance/FORMAT-SPEC-001.json` |
| 104 | +- Preprint context: arXiv:2606.05017 |
| 105 | + |
| 106 | +phi^2 + phi^-2 = 3 | TRINITY |
0 commit comments