Skip to content

Commit a8b5a0b

Browse files
committed
docs: CRITICAL — calibrate against ONNX f32 (RAW), not GGUF BF16 (TIFF)
BF16 7-bit mantissa flips rank order for cosines within ±0.008. Spearman ρ drops ~5% from BF16 truncation alone, not encoding. Calibrate against ONNX f32 to isolate pure encoding error. ICC profile then corrects BOTH encoding AND BF16 truncation. Camera analogy: calibrate lens against RAW, never against JPEG. https://claude.ai/code/session_01ChLvBfpJS8dQhHxRD4pYNp
1 parent 6619cb1 commit a8b5a0b

1 file changed

Lines changed: 26 additions & 0 deletions

File tree

.claude/HANDOVER_MAVERICK_SESSION.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -646,3 +646,29 @@ Compute: rten inference on 1000 texts ≈ 5 min
646646
Repeat for remaining 5 models: ~2 hours total
647647
Full calibration matrix: ~2.5 hours
648648
```
649+
650+
---
651+
652+
## CRITICAL: Calibrate against ONNX f32, NOT GGUF BF16
653+
654+
```
655+
ONNX f32 (2.4 GB) = RAW file (full sensor, 24-bit mantissa)
656+
GGUF BF16 (1.2 GB) = TIFF (7-bit mantissa, legs chopped)
657+
Our table = JPEG (8-bit u8/i8, compressed for distribution)
658+
659+
Camera profile calibrates against RAW, never against JPEG.
660+
Lens ICC calibrates against ONNX f32, never against GGUF BF16.
661+
662+
BF16 truncation flips rank order for cosines within ±0.008:
663+
f32: cos(A,B)=0.7234, cos(A,C)=0.7229 → B closer
664+
BF16: both round to 0.7226 → TIE or FLIP
665+
Spearman drops 5% from BF16 alone, not from our encoding.
666+
667+
Calibration pipeline:
668+
ONNX f32 → CLAM → table → compare with ONNX f32 inference → pure encoding error
669+
NOT: GGUF BF16 → CLAM → compare with ONNX f32 → mixed error (encoding + truncation)
670+
671+
Production pipeline:
672+
GGUF BF16 → CLAM → table + ICC correction (from ONNX calibration) → corrected table
673+
The ICC absorbs the BF16 truncation error because it was calibrated against f32.
674+
```

0 commit comments

Comments
 (0)