ml8: make determinism opt-in (ML8_DETERMINISTIC=1) — full-calib Cholesky regression deferred

The default-on determinism from 14bccfa breaks full-model calibration: from the first
full-self-attention layer onward (layer 3 on 0.8B), torch.use_deterministic_algorithms(True)
makes the SDPA forward yield a degenerate (NaN-suspected) Hessian for that layer's MLP → GPTQ
Cholesky throws "not positive-definite" → the tensor is silently skipped → ~130/151 tensors
fall back to bf16 in convert (1144 MB "model" with a misleading 18.25 PPL that's mostly bf16).

Diagnosis (durable in KG): determinism is the trigger (det-off=151 tensors / det-on=21, same
data); the failing tensors never reach batched_gptq (no diag line), and every Hessian the probe
DID see is solidly PD (eig_min 0.05-0.15) — so it's not a marginal-conditioning issue, it's the
self-attn forward corrupting the downstream Hessian. The "best-of-N + clean ladder" payoff still
stands once this is fixed; reverting to opt-in restores the known-good nondeterministic calibrator
as the default. Adds diag_soft_determinism.py (proved soft determinism = full coverage but NOT
reproducible, so the hard hammer is load-bearing → must fix the SDPA/Cholesky path, not drop it).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: kmbandy

scripts/calibration/calibrate_ml8_paged.py

@@ -50,13 +50,16 @@
 
 print = functools.partial(print, flush=True)
 
-# ── Deterministic calibration — env half (default-on; MUST precede `import torch`) ───
-# ml8 GPTQ calibration was nondeterministic: the GPU Hessian forward used unforced reduction
-# order and GPTQ's sequential error-feedback amplified the low-bit noise into DIFFERENT weight
-# assignments run-to-run (~0.6 PPL spread — swamps the ±0.05 levers). The fix is pinning the GEMM
-# workspace (here) + enabling deterministic algorithms (after the torch import). Verified
-# bit-identical across back-to-back runs on gfx1201. Opt out with ML8_NONDETERMINISTIC=1.
-_DETERMINISTIC = os.environ.get("ML8_NONDETERMINISTIC") != "1"
+# ── Deterministic calibration — env half (OPT-IN via ML8_DETERMINISTIC=1; MUST precede `import torch`) ───
+# ml8 GPTQ calibration was nondeterministic: the GPU Hessian forward used unforced reduction order
+# and GPTQ's sequential error-feedback amplified the low-bit noise into DIFFERENT weight assignments
+# run-to-run (~0.6 PPL spread — swamps the ±0.05 levers). These flags pin it bit-identically on the
+# linear-attn/MLP path (verified). BUT torch.use_deterministic_algorithms(True) currently breaks
+# full-model calibration: from the FIRST full-self-attention layer onward, the SDPA forward under
+# the hammer yields a degenerate (NaN-suspected) Hessian → GPTQ Cholesky fails → tensors silently
+# skipped → partial-coverage model. Root-cause + fix is deferred (see KG / handoff), so determinism
+# is OPT-IN until then; the default path is the known-good nondeterministic calibrator.
+_DETERMINISTIC = os.environ.get("ML8_DETERMINISTIC") == "1"
 if _DETERMINISTIC:
     os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")  # required for deterministic cuBLAS/hipBLAS GEMM
     os.environ.setdefault("HIPBLASLT_DETERMINISTIC", "1")
@@ -82,8 +85,8 @@
                         ("allow_bf16_reduced_precision_reduction", False)):
         try: setattr(torch.backends.cuda.matmul, _attr, _val)
         except AttributeError: pass
-    print("[determinism] use_deterministic_algorithms(True) + seeded + pinned GEMM "
-          "(set ML8_NONDETERMINISTIC=1 to disable)")
+    print("[determinism] OPT-IN ENABLED: use_deterministic_algorithms(True) + seeded + pinned GEMM "
+          "— WARNING: breaks full-model coverage at the first self-attn layer (Cholesky/SDPA bug, WIP)")
 
 # Re-use existing calibration helpers (compute_hessian / gptq_quantize_linear / eval_ppl_wikitext / etc.)
 sys.path.insert(0, str(Path(__file__).parent))

scripts/calibration/diag_soft_determinism.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+"""Test 'soft' determinism: seed + pinned GEMM workspace + tf32 off, but WITHOUT the strict
+torch.use_deterministic_algorithms(True) hammer. Hypothesis: the hammer is what perturbs
+cholesky_inverse into a non-PD H^-1 (breaking GPTQ's second Cholesky on 130/151 tensors), while
+softer determinism still gives bit-reproducible calibration. Disables the driver's built-in
+(hard) determinism block via ML8_NONDETERMINISTIC=1, then sets the soft subset here."""
+import os
+import sys
+
+os.environ["ML8_NONDETERMINISTIC"] = "1"                  # skip the driver's hard-determinism block
+os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
+os.environ.setdefault("HIPBLASLT_DETERMINISTIC", "1")
+
+import torch  # noqa: E402
+
+torch.manual_seed(0)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(0)
+for _obj, _attr, _val in (
+    (torch.backends.cudnn, "deterministic", True),
+    (torch.backends.cudnn, "benchmark", False),
+    (torch.backends.cudnn, "allow_tf32", False),
+    (torch.backends.cuda.matmul, "allow_tf32", False),
+    (torch.backends.cuda.matmul, "allow_fp16_reduced_precision_reduction", False),
+    (torch.backends.cuda.matmul, "allow_bf16_reduced_precision_reduction", False),
+):
+    try:
+        setattr(_obj, _attr, _val)
+    except AttributeError:
+        pass
+print("[soft-determinism] seed + pinned GEMM + tf32 off, NO use_deterministic_algorithms",
+      file=sys.stderr, flush=True)
+
+_here = os.path.dirname(os.path.abspath(__file__))
+sys.argv = [os.path.join(_here, "calibrate_ml8_paged.py")] + sys.argv[1:]
+import runpy  # noqa: E402
+runpy.run_path(sys.argv[0], run_name="__main__")