Skip to content

Commit ce73320

Browse files
author
Dmitrii Vasilev
committed
docs: Add benchmark results README with analysis
1 parent bc132d6 commit ce73320

1 file changed

Lines changed: 88 additions & 0 deletions

File tree

data/README.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# PhD Comparative Benchmark Suite Results
2+
3+
## Overview
4+
This directory contains CSV files with benchmark results comparing 12 floating-point formats:
5+
- **IEEE formats**: fp32, fp16, bf16
6+
- **φ-based GoldenFloat formats**: gf16, gf24, gf20, gf12, gf8, gf6a, gf4a, gf32, gf64
7+
8+
## Benchmark Files
9+
10+
### 1. `bench1_roundtrip_mse.csv`
11+
**Roundtrip MSE** (encode → decode error)
12+
- Columns: format, mse, mae, max_abs_error, timestamp
13+
- 12 data points
14+
15+
**Key findings:**
16+
- gf64: Near-perfect (MSE ≈ 0)
17+
- fp32: Perfect (MSE = 0)
18+
- gf32: Excellent (MSE = 0.007)
19+
- gf24: Good (MSE = 1.13)
20+
- bf16: Moderate (MSE = 126.7)
21+
- gf16: Moderate (MSE = 290)
22+
- gf20: Poor (MSE = 4.77)
23+
- gf12/gf8/gf6a/gf4a: High error (MSE > 22M)
24+
25+
### 2. `bench2_phi_distance.csv`
26+
**φ-distance** (deviation from ideal φ-based quantization)
27+
- Columns: format, phi_distance, timestamp
28+
- 12 data points
29+
30+
**Key findings:**
31+
- All φ-based formats: 0.0 (by definition, they are φ-optimal)
32+
- fp32: Minimal (0.000003)
33+
- bf16: Low (0.008)
34+
- fp16: High (0.997) - IEEE fp16 has high φ-distance
35+
36+
### 3. `bench3_sacred_constants.csv`
37+
**Sacred constants preservation** (π, e, φ, √2)
38+
- Columns: format, constant, abs_error, rel_error, timestamp
39+
- 48 data points (12 formats × 4 constants)
40+
41+
**Key findings:**
42+
- fp32/gf64: Perfect preservation
43+
- gf32: Excellent (avg rel error ≈ 1.45e-5)
44+
- gf24: Good (avg rel error ≈ 1.12e-4)
45+
- bf16: Moderate (avg rel error ≈ 0.0016)
46+
- gf16/gf20: Fair (avg rel error ≈ 0.002-0.003)
47+
- gf12/gf8: Poor (avg rel error ≈ 0.006-0.026)
48+
- gf6a/gf4a: Very poor (avg rel error ≈ 0.08-0.26)
49+
50+
### 4. `bench4_gradient_norm.csv`
51+
**Gradient norm distribution** (He/Kaiming initialization, 10K gradients)
52+
- Columns: format, l1_norm, l2_norm, linf_norm, sparsity, l1_ratio, l2_ratio, linf_ratio, timestamp
53+
- 12 data points
54+
55+
**Key findings:**
56+
- fp16: 100% sparsity (gradient collapse!)
57+
- All other formats: 0% sparsity (gradients preserved)
58+
- L2 ratios: Most formats within 1-3% of fp32 baseline
59+
- gf6a shows +3.15% L2 deviation
60+
- gf4a shows +1.70% L2 deviation
61+
62+
## Running the Benchmarks
63+
64+
```bash
65+
cargo run --bin phd-benchmarks
66+
```
67+
68+
## Format Specifications
69+
70+
| Format | Layout | Bias | Range | Notes |
71+
|--------|--------|------|-------|-------|
72+
| fp32 | IEEE 754 | 127 | ±3.4e38 | Baseline |
73+
| fp16 | IEEE 754 | 15 | ±65504 | Full subnormals |
74+
| bf16 | IEEE 754 | 127 | ~±3.4e38 | 7-bit mantissa |
75+
| gf16 | φ-based | 31 | ~±65504 | φ-optimal |
76+
| gf24 | φ-based | 127 | Large | 15-bit mantissa |
77+
| gf20 | φ-based | 31 | Medium | 13-bit mantissa |
78+
| gf12 | φ-based | 7 | Small | 7-bit mantissa |
79+
| gf8 | φ-based | 3 | ~±15 | Very small range |
80+
| gf6a | φ-based | 3 | ~±15 | 2-bit mantissa |
81+
| gf4a | φ-based | 1 | ~±φ | Ternary-like |
82+
| gf32 | φ-based | 1023 | Very large | 20-bit mantissa |
83+
| gf64 | φ-based | 16383 | Double-like | 48-bit mantissa |
84+
85+
## PhD Deliverable
86+
Total: **48 data points** (4 benchmarks × 12 formats)
87+
88+
All data is publishable in CSV format for gradient distribution analysis.

0 commit comments

Comments
 (0)