ml8: deterministic calibration (default-on) — fixes ~0.6 PPL run-to-run nondeterminism

The GPU Hessian forward had unforced reduction order; GPTQ's sequential error-feedback
amplified the low-bit noise into different discrete weight assignments run-to-run (~0.6 PPL
spread — swamping the ±0.05 levers). Same recipe + same data gave 151/188 differing blobs;
only the deterministic .fp8 tier matched. Y_SNR stayed flat (~20.87) so it never surfaced.

Fix (calibrate_ml8_paged.py, default-on, ML8_NONDETERMINISTIC=1 to opt out):
- env before torch import: CUBLAS_WORKSPACE_CONFIG=:4096:8, HIPBLASLT_DETERMINISTIC=1
- torch.manual_seed/cuda.manual_seed_all, use_deterministic_algorithms(True),
  cudnn.deterministic, tf32 off, fp16/bf16 reduced-precision reduction off

Verified gfx1201: warn_only diag flagged zero ops without deterministic impls; two 1-layer
runs -> 6/6 blobs bit-identical; full-scale gate (--max-layers 12, incl self_attn SDPA +
fla fp32 SSM + hipBLAS) -> 21/21 bit-identical. Adds diag_determinism.py (warn_only enumerator).

Unlocks: (1) zero run-to-run noise -> faithful-acts/weights deltas now cleanly measurable;
(2) --rotation-seed becomes a reproducible best-of-N lever (select on held-out, report on wiki).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: kmbandy

scripts/calibration/calibrate_ml8_paged.py

@@ -50,11 +50,41 @@
 
 print = functools.partial(print, flush=True)
 
+# ── Deterministic calibration — env half (default-on; MUST precede `import torch`) ───
+# ml8 GPTQ calibration was nondeterministic: the GPU Hessian forward used unforced reduction
+# order and GPTQ's sequential error-feedback amplified the low-bit noise into DIFFERENT weight
+# assignments run-to-run (~0.6 PPL spread — swamps the ±0.05 levers). The fix is pinning the GEMM
+# workspace (here) + enabling deterministic algorithms (after the torch import). Verified
+# bit-identical across back-to-back runs on gfx1201. Opt out with ML8_NONDETERMINISTIC=1.
+_DETERMINISTIC = os.environ.get("ML8_NONDETERMINISTIC") != "1"
+if _DETERMINISTIC:
+    os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")  # required for deterministic cuBLAS/hipBLAS GEMM
+    os.environ.setdefault("HIPBLASLT_DETERMINISTIC", "1")
+
 import numpy as np
 import torch
 from torch import nn
 from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
 
+# ── Deterministic calibration — torch half (see env half above) ───
+if _DETERMINISTIC:
+    torch.manual_seed(0)
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed_all(0)
+    torch.use_deterministic_algorithms(True)
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.benchmark = False
+    for _attr in ("allow_tf32",):
+        try: setattr(torch.backends.cudnn, _attr, False)
+        except AttributeError: pass
+    for _attr, _val in (("allow_tf32", False),
+                        ("allow_fp16_reduced_precision_reduction", False),
+                        ("allow_bf16_reduced_precision_reduction", False)):
+        try: setattr(torch.backends.cuda.matmul, _attr, _val)
+        except AttributeError: pass
+    print("[determinism] use_deterministic_algorithms(True) + seeded + pinned GEMM "
+          "(set ML8_NONDETERMINISTIC=1 to disable)")
+
 # Re-use existing calibration helpers (compute_hessian / gptq_quantize_linear / eval_ppl_wikitext / etc.)
 sys.path.insert(0, str(Path(__file__).parent))
 from calibrate_ml8 import (  # noqa: E402

scripts/calibration/diag_determinism.py

@@ -0,0 +1,45 @@
+#!/usr/bin/env python3
+"""Determinism diagnostic for the ml8 GPTQ calibration path.
+
+Enables torch.use_deterministic_algorithms(True, warn_only=True) + seeding + GEMM-determinism
+env BEFORE running the real calibrate_ml8_paged driver, so PyTorch logs every op in our path
+that has no deterministic implementation. Pass-through args go to the driver (use a tiny
+--max-layers / --n-samples for a fast run). Run with PYTHONWARNINGS=always to see every warning.
+
+Read the result by grepping stderr for:  "does not have a deterministic implementation"
+"""
+import os
+import sys
+import warnings
+
+# GEMM / reduction determinism env (must be set before the first CUDA/HIP context).
+os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
+os.environ.setdefault("HIPBLASLT_DETERMINISTIC", "1")   # hipBLASLt analog; harmless if ignored
+os.environ.setdefault("MIOPEN_DEBUG_CONV_IMPLICIT_GEMM", "0")
+
+import torch  # noqa: E402
+
+torch.manual_seed(0)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(0)
+try:
+    torch.backends.cuda.matmul.allow_tf32 = False
+    torch.backends.cudnn.allow_tf32 = False
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.benchmark = False
+    torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction = False
+    torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = False
+except Exception as e:
+    print(f"[diag] backend flag note: {e}", file=sys.stderr)
+
+# warn_only=True ⇒ nondeterministic ops WARN instead of raising, so we enumerate them all.
+torch.use_deterministic_algorithms(True, warn_only=True)
+warnings.simplefilter("always")
+print("[diag] determinism enabled (warn_only=True); seed=0; running calibrate driver...",
+      file=sys.stderr, flush=True)
+
+# Hand off to the real driver in THIS process (the flags above persist on the torch singleton).
+_here = os.path.dirname(os.path.abspath(__file__))
+sys.argv = [os.path.join(_here, "calibrate_ml8_paged.py")] + sys.argv[1:]
+import runpy  # noqa: E402
+runpy.run_path(sys.argv[0], run_name="__main__")