Skip to content

Commit 00e92fe

Browse files
gHashTaggHashTag
gHashTag
and
gHashTag
authored
conformance: add v0 vector packs (GF16, FP8 E4M3FN/E5M2, MXFP4 E2M1, bf16 golden) + index (#1054)
Bit-precise conformance vectors in a shared row schema, one differ across all formats. Anchor identity phi^2 + 1/phi^2 = 3 exact in every pack. ASCII-only. abs_error honest (nonzero only for non-representable inputs). Generators included for byte-identical reproduction. Context: arXiv:2606.05017. Closes #1055 Co-authored-by: gHashTag <admin@t27.ai>
1 parent 0f7424b commit 00e92fe

11 files changed

Lines changed: 1618 additions & 0 deletions

‎conformance/vectors/README.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# t27 numeric-format conformance vector packs (v0)
2+
3+
Bit-precise conformance vectors for a family of low-precision numeric formats,
4+
in a single shared row schema so one differ runs across all packs.
5+
6+
- SSOT: https://github.com/gHashTag/t27/blob/master/conformance/FORMAT-SPEC-001.json
7+
- Anchor identity (ASCII): `phi^2 + 1/phi^2 = 3`
8+
- Context preprint: https://arxiv.org/abs/2606.05017
9+
- Schema tag: `t27-conformance/v0.1`
10+
11+
Every pack decodes its native bits to f64 exactly (`abs_error = 0` by
12+
construction for representable values). Values that are NOT exactly
13+
representable in a format report a nonzero `abs_error` honestly (for example
14+
0.1 in bf16) -- nothing is hidden.
15+
16+
## Index
17+
18+
| Pack | Format | Bits | Vectors | 3.0 anchor bits | Max finite | Inf | NaN | Has golden | tt-mlir issue |
19+
|---|---|---|---|---|---|---|---|---|---|
20+
| `gf16_conformance_v0.json` | GF16 | 16 | 21 | `0x4100` | n/a | yes | no | no | sibling (SSOT anchor pack) |
21+
| `fp8_e4m3fn_conformance_v0.json` | FP8 E4M3FN | 8 | 14 | `0x44` | 448.0 (`0x7E`) | no | single (`0x7F`/`0xFF`) | no | [#8549](https://github.com/tenstorrent/tt-mlir/issues/8549) |
22+
| `fp8_e5m2_conformance_v0.json` | FP8 E5M2 | 8 | 16 | `0x42` | 57344.0 (`0x7B`) | yes (`0x7C`/`0xFC`) | yes (E=11111,M!=0) | no | [#8549](https://github.com/tenstorrent/tt-mlir/issues/8549) |
23+
| `mxfp4_e2m1_conformance_v0.json` | MXFP4 E2M1 | 4 | 16 | `0x5` | 6.0 (`0x7`) | no | no | no | [#7085](https://github.com/tenstorrent/tt-mlir/issues/7085) |
24+
| `bf16_golden_conformance_v0.json` | bfloat16 | 16 | 8 + golden | `0x4040` | n/a | yes | yes | yes | [#6252](https://github.com/tenstorrent/tt-mlir/issues/6252) |
25+
26+
## SHA-256
27+
28+
```
29+
7aea5b9e86ea71a54ae0c1601cea13e2d90d95fecaf2ae969eac1349cf7a2b42 gf16_conformance_v0.json
30+
7193ccd0d330d3e05154432abcec5da4a4c170e11004d4ffa44ff5cbbff9cba9 fp8_e4m3fn_conformance_v0.json
31+
9c31fbd03923bd6555304848a092504dfbc02f72d2be82d2b80f49243e925a18 fp8_e5m2_conformance_v0.json
32+
b5795fed0c0f2b580174b443d2c54519c4953916525237bb7ea7d6831f14fde7 mxfp4_e2m1_conformance_v0.json
33+
98bbddcbb8a520dc45a6dfed7209c50a0acc0fabc4d3b430359969467eee4e13 bf16_golden_conformance_v0.json
34+
```
35+
36+
## Shared row schema
37+
38+
Each vector row carries:
39+
40+
```
41+
name short label
42+
input_f64 the f64 value (or "NaN" / "Inf" / "-Inf" for specials)
43+
input_f64_hex big-endian IEEE-754 double hex of input
44+
<fmt>_bits_hex the native format bit pattern, hex
45+
<fmt>_bits_int the native format bit pattern, integer
46+
decoded_f64 value after decode back to f64
47+
decoded_f64_hex big-endian double hex of decoded
48+
abs_error |input_f64 - decoded_f64| (0 for representable values)
49+
category one of: zero, subnormal, normal, inf, nan, phi_anchor
50+
```
51+
52+
The bits field is named per format: `gf16_bits_*`, `fp8_bits_*`,
53+
`mxfp4_bits_*`, `bf16_bits_*`. All other keys are identical across packs.
54+
55+
## Per-format notes (the round-trip decision in each)
56+
57+
### GF16 (16-bit)
58+
The original anchor pack. Carries `phi`, `inv_phi`, `phi^2`, `inv_phi^2`, and
59+
the identity sum `phi^2 + 1/phi^2 = 3` (vector `identity_sum`, bits `0x4100`).
60+
Categories include `phi_anchor`. Sibling to the FP8/MXFP4/bf16 packs via the
61+
shared row schema.
62+
63+
### FP8 E4M3FN (`torch.float8_e4m3fn`)
64+
1s4e3m, bias 7. No inf. Single NaN at `S.1111.111`. Max finite 448.0 at `0x7E`
65+
(mantissa 110, since all-ones mantissa at max exponent is reserved for NaN).
66+
Min subnormal 2^-9, min normal 2^-6.
67+
Encode policy (reference): round-nearest-ties-even; overflow SATURATES to
68+
448.0, not NaN. That saturate-vs-NaN choice is the round-trip decision in
69+
tt-mlir #8549.
70+
71+
### FP8 E5M2 (`torch.float8_e5m2`)
72+
1s5e2m, bias 15. HAS inf at `0x7C`/`0xFC` (E=11111, M=00); NaN at E=11111,
73+
M!=00. Max finite 57344.0 at `0x7B`. Min subnormal 2^-16, min normal 2^-14.
74+
Encode policy (reference): round-nearest-ties-even; overflow goes to +/-Inf
75+
(true IEEE), NOT saturate. The saturate (E4M3FN) vs overflow-to-Inf (E5M2)
76+
split is the practical round-trip gotcha across the two FP8 variants.
77+
78+
### MXFP4 E2M1 (OCP Microscaling element format)
79+
1s2e1m, bias 1. Only 16 codes; no inf, no NaN. Full positive grid
80+
`{0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0}`. Max finite 6.0 at `0x7`.
81+
tt-mlir #7085 clamps to +/-7, which is one step ABOVE the format max:
82+
values in (6, 7] still round to the 6.0 grid point. The clamp bound and the
83+
representable max are different numbers; the pack enumerates all 16 codes so
84+
that relationship is explicit. Anchor 3.0 is an exact grid point at `0x5`.
85+
86+
### bfloat16 (golden accumulation)
87+
1s8e7m, bias 127 (top 16 bits of fp32). HAS inf and NaN. Decode exact into f64.
88+
Element table plus a `golden_accumulation` section targeting tt-mlir #6252:
89+
the all-close gap in depth-31 fused reduction trees comes from accumulation
90+
ORDER, not single-op rounding. The section gives an f64 golden reference and
91+
compares sequential vs pairwise bf16 reductions; pairwise stays near the golden
92+
while sequential drifts. A reproducible all-close test needs a clamped
93+
deterministic input set plus an f64 golden -- both are supplied.
94+
95+
## Provenance
96+
97+
Generators (re-run to reproduce byte-identical packs):
98+
99+
```
100+
gen_fp8_e4m3.py -> fp8_e4m3fn_conformance_v0.json
101+
gen_fp8_e5m2.py -> fp8_e5m2_conformance_v0.json
102+
gen_mxfp4_e2m1.py -> mxfp4_e2m1_conformance_v0.json
103+
gen_bf16_golden.py -> bf16_golden_conformance_v0.json
104+
```
105+
106+
All packs are ASCII-only. Apache-2.0, consistent with the t27 repository.

‎conformance/vectors/bf16_golden_conformance_v0.json‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
{
2+
"schema": "t27-conformance/v0.1",
3+
"format": "BFLOAT16",
4+
"format_notes": "1s8e7m, bias 127, same exponent range as fp32 (top 16 bits of fp32). HAS inf and NaN. Decode exact into f64.",
5+
"ssot": "https://github.com/gHashTag/t27/blob/master/conformance/FORMAT-SPEC-001.json",
6+
"preprint": "https://arxiv.org/abs/2606.05017",
7+
"anchor_identity": "phi^2 + 1/phi^2 = 3",
8+
"anchor_check": {
9+
"value": 3.0,
10+
"expected": 3.0,
11+
"ieee754_exact": true,
12+
"bf16_bits_hex": "0x4040"
13+
},
14+
"issue_6252_note": "Rounding-error accumulation in depth-31 fused trees. The all-close gap comes from accumulation ORDER, not single-op rounding. The golden_accumulation section gives an f64 golden reference plus sequential vs pairwise bf16 reductions; pairwise reduces the accumulated error. A reproducible all-close test needs (a) a clamped deterministic input set and (b) an f64 golden, both supplied here.",
15+
"round_trip_policy": "decode: bf16 bits -> f64 exact. encode: fp32->bf16 round-nearest-ties-even on the 16-bit truncation boundary. abs_error is nonzero only for inputs not representable in bf16 (e.g. 0.1) -- stated honestly, not hidden.",
16+
"n_vectors": 8,
17+
"vectors": [
18+
{
19+
"name": "pos_zero",
20+
"input_f64": 0.0,
21+
"input_f64_hex": "0x0000000000000000",
22+
"bf16_bits_hex": "0x0000",
23+
"bf16_bits_int": 0,
24+
"decoded_f64": 0.0,
25+
"decoded_f64_hex": "0x0000000000000000",
26+
"abs_error": 0.0,
27+
"category": "zero"
28+
},
29+
{
30+
"name": "pos_one",
31+
"input_f64": 1.0,
32+
"input_f64_hex": "0x3FF0000000000000",
33+
"bf16_bits_hex": "0x3F80",
34+
"bf16_bits_int": 16256,
35+
"decoded_f64": 1.0,
36+
"decoded_f64_hex": "0x3FF0000000000000",
37+
"abs_error": 0.0,
38+
"category": "normal"
39+
},
40+
{
41+
"name": "neg_one",
42+
"input_f64": -1.0,
43+
"input_f64_hex": "0xBFF0000000000000",
44+
"bf16_bits_hex": "0xBF80",
45+
"bf16_bits_int": 49024,
46+
"decoded_f64": -1.0,
47+
"decoded_f64_hex": "0xBFF0000000000000",
48+
"abs_error": 0.0,
49+
"category": "normal"
50+
},
51+
{
52+
"name": "pos_two",
53+
"input_f64": 2.0,
54+
"input_f64_hex": "0x4000000000000000",
55+
"bf16_bits_hex": "0x4000",
56+
"bf16_bits_int": 16384,
57+
"decoded_f64": 2.0,
58+
"decoded_f64_hex": "0x4000000000000000",
59+
"abs_error": 0.0,
60+
"category": "normal"
61+
},
62+
{
63+
"name": "pos_three",
64+
"input_f64": 3.0,
65+
"input_f64_hex": "0x4008000000000000",
66+
"bf16_bits_hex": "0x4040",
67+
"bf16_bits_int": 16448,
68+
"decoded_f64": 3.0,
69+
"decoded_f64_hex": "0x4008000000000000",
70+
"abs_error": 0.0,
71+
"category": "normal"
72+
},
73+
{
74+
"name": "pos_half",
75+
"input_f64": 0.5,
76+
"input_f64_hex": "0x3FE0000000000000",
77+
"bf16_bits_hex": "0x3F00",
78+
"bf16_bits_int": 16128,
79+
"decoded_f64": 0.5,
80+
"decoded_f64_hex": "0x3FE0000000000000",
81+
"abs_error": 0.0,
82+
"category": "normal"
83+
},
84+
{
85+
"name": "pos_quarter",
86+
"input_f64": 0.25,
87+
"input_f64_hex": "0x3FD0000000000000",
88+
"bf16_bits_hex": "0x3E80",
89+
"bf16_bits_int": 16000,
90+
"decoded_f64": 0.25,
91+
"decoded_f64_hex": "0x3FD0000000000000",
92+
"abs_error": 0.0,
93+
"category": "normal"
94+
},
95+
{
96+
"name": "pos_0p1",
97+
"input_f64": 0.1,
98+
"input_f64_hex": "0x3FB999999999999A",
99+
"bf16_bits_hex": "0x3DCD",
100+
"bf16_bits_int": 15821,
101+
"decoded_f64": 0.10009765625,
102+
"decoded_f64_hex": "0x3FB9A00000000000",
103+
"abs_error": 9.765624999999445e-05,
104+
"category": "normal"
105+
}
106+
],
107+
"golden_accumulation": [
108+
{
109+
"element_f64": 0.1,
110+
"element_f64_hex": "0x3FB999999999999A",
111+
"element_bf16_hex": "0x3DCD",
112+
"n_terms": 1024,
113+
"golden_f64": 102.4,
114+
"golden_f64_hex": "0x405999999999999A",
115+
"sequential_bf16": 32.0,
116+
"pairwise_bf16": 102.5,
117+
"abs_err_sequential": 70.4,
118+
"abs_err_pairwise": 0.09999999999999432,
119+
"rel_err_sequential": 0.6875,
120+
"rel_err_pairwise": 0.0009765624999999445,
121+
"pairwise_better": true
122+
},
123+
{
124+
"element_f64": 3.0,
125+
"element_f64_hex": "0x4008000000000000",
126+
"element_bf16_hex": "0x4040",
127+
"n_terms": 1024,
128+
"golden_f64": 3072.0,
129+
"golden_f64_hex": "0x40A8000000000000",
130+
"sequential_bf16": 1024.0,
131+
"pairwise_bf16": 3072.0,
132+
"abs_err_sequential": 2048.0,
133+
"abs_err_pairwise": 0.0,
134+
"rel_err_sequential": 0.6666666666666666,
135+
"rel_err_pairwise": 0.0,
136+
"pairwise_better": true
137+
},
138+
{
139+
"element_f64": 0.1,
140+
"element_f64_hex": "0x3FB999999999999A",
141+
"element_bf16_hex": "0x3DCD",
142+
"n_terms": 4096,
143+
"golden_f64": 409.6,
144+
"golden_f64_hex": "0x407999999999999A",
145+
"sequential_bf16": 32.0,
146+
"pairwise_bf16": 410.0,
147+
"abs_err_sequential": 377.6,
148+
"abs_err_pairwise": 0.39999999999997726,
149+
"rel_err_sequential": 0.921875,
150+
"rel_err_pairwise": 0.0009765624999999445,
151+
"pairwise_better": true
152+
},
153+
{
154+
"element_f64": 0.3333333333333333,
155+
"element_f64_hex": "0x3FD5555555555555",
156+
"element_bf16_hex": "0x3EAB",
157+
"n_terms": 2048,
158+
"golden_f64": 682.6666666666666,
159+
"golden_f64_hex": "0x4085555555555555",
160+
"sequential_bf16": 128.0,
161+
"pairwise_bf16": 684.0,
162+
"abs_err_sequential": 554.6666666666666,
163+
"abs_err_pairwise": 1.3333333333333712,
164+
"rel_err_sequential": 0.8125,
165+
"rel_err_pairwise": 0.0019531250000000555,
166+
"pairwise_better": true
167+
}
168+
]
169+
}

0 commit comments

Comments
 (0)