|
| 1 | +# PhD Comparative Benchmark Suite Results |
| 2 | + |
| 3 | +## Overview |
| 4 | +This directory contains CSV files with benchmark results comparing 12 floating-point formats: |
| 5 | +- **IEEE formats**: fp32, fp16, bf16 |
| 6 | +- **φ-based GoldenFloat formats**: gf16, gf24, gf20, gf12, gf8, gf6a, gf4a, gf32, gf64 |
| 7 | + |
| 8 | +## Benchmark Files |
| 9 | + |
| 10 | +### 1. `bench1_roundtrip_mse.csv` |
| 11 | +**Roundtrip MSE** (encode → decode error) |
| 12 | +- Columns: format, mse, mae, max_abs_error, timestamp |
| 13 | +- 12 data points |
| 14 | + |
| 15 | +**Key findings:** |
| 16 | +- gf64: Near-perfect (MSE ≈ 0) |
| 17 | +- fp32: Perfect (MSE = 0) |
| 18 | +- gf32: Excellent (MSE = 0.007) |
| 19 | +- gf24: Good (MSE = 1.13) |
| 20 | +- bf16: Moderate (MSE = 126.7) |
| 21 | +- gf16: Moderate (MSE = 290) |
| 22 | +- gf20: Poor (MSE = 4.77) |
| 23 | +- gf12/gf8/gf6a/gf4a: High error (MSE > 22M) |
| 24 | + |
| 25 | +### 2. `bench2_phi_distance.csv` |
| 26 | +**φ-distance** (deviation from ideal φ-based quantization) |
| 27 | +- Columns: format, phi_distance, timestamp |
| 28 | +- 12 data points |
| 29 | + |
| 30 | +**Key findings:** |
| 31 | +- All φ-based formats: 0.0 (by definition, they are φ-optimal) |
| 32 | +- fp32: Minimal (0.000003) |
| 33 | +- bf16: Low (0.008) |
| 34 | +- fp16: High (0.997) - IEEE fp16 has high φ-distance |
| 35 | + |
| 36 | +### 3. `bench3_sacred_constants.csv` |
| 37 | +**Sacred constants preservation** (π, e, φ, √2) |
| 38 | +- Columns: format, constant, abs_error, rel_error, timestamp |
| 39 | +- 48 data points (12 formats × 4 constants) |
| 40 | + |
| 41 | +**Key findings:** |
| 42 | +- fp32/gf64: Perfect preservation |
| 43 | +- gf32: Excellent (avg rel error ≈ 1.45e-5) |
| 44 | +- gf24: Good (avg rel error ≈ 1.12e-4) |
| 45 | +- bf16: Moderate (avg rel error ≈ 0.0016) |
| 46 | +- gf16/gf20: Fair (avg rel error ≈ 0.002-0.003) |
| 47 | +- gf12/gf8: Poor (avg rel error ≈ 0.006-0.026) |
| 48 | +- gf6a/gf4a: Very poor (avg rel error ≈ 0.08-0.26) |
| 49 | + |
| 50 | +### 4. `bench4_gradient_norm.csv` |
| 51 | +**Gradient norm distribution** (He/Kaiming initialization, 10K gradients) |
| 52 | +- Columns: format, l1_norm, l2_norm, linf_norm, sparsity, l1_ratio, l2_ratio, linf_ratio, timestamp |
| 53 | +- 12 data points |
| 54 | + |
| 55 | +**Key findings:** |
| 56 | +- fp16: 100% sparsity (gradient collapse!) |
| 57 | +- All other formats: 0% sparsity (gradients preserved) |
| 58 | +- L2 ratios: Most formats within 1-3% of fp32 baseline |
| 59 | +- gf6a shows +3.15% L2 deviation |
| 60 | +- gf4a shows +1.70% L2 deviation |
| 61 | + |
| 62 | +## Running the Benchmarks |
| 63 | + |
| 64 | +```bash |
| 65 | +cargo run --bin phd-benchmarks |
| 66 | +``` |
| 67 | + |
| 68 | +## Format Specifications |
| 69 | + |
| 70 | +| Format | Layout | Bias | Range | Notes | |
| 71 | +|--------|--------|------|-------|-------| |
| 72 | +| fp32 | IEEE 754 | 127 | ±3.4e38 | Baseline | |
| 73 | +| fp16 | IEEE 754 | 15 | ±65504 | Full subnormals | |
| 74 | +| bf16 | IEEE 754 | 127 | ~±3.4e38 | 7-bit mantissa | |
| 75 | +| gf16 | φ-based | 31 | ~±65504 | φ-optimal | |
| 76 | +| gf24 | φ-based | 127 | Large | 15-bit mantissa | |
| 77 | +| gf20 | φ-based | 31 | Medium | 13-bit mantissa | |
| 78 | +| gf12 | φ-based | 7 | Small | 7-bit mantissa | |
| 79 | +| gf8 | φ-based | 3 | ~±15 | Very small range | |
| 80 | +| gf6a | φ-based | 3 | ~±15 | 2-bit mantissa | |
| 81 | +| gf4a | φ-based | 1 | ~±φ | Ternary-like | |
| 82 | +| gf32 | φ-based | 1023 | Very large | 20-bit mantissa | |
| 83 | +| gf64 | φ-based | 16383 | Double-like | 48-bit mantissa | |
| 84 | + |
| 85 | +## PhD Deliverable |
| 86 | +Total: **48 data points** (4 benchmarks × 12 formats) |
| 87 | + |
| 88 | +All data is publishable in CSV format for gradient distribution analysis. |
0 commit comments