Drift test: tolerate both GEMM reduction orders on the staircase rows

The staircase rows measure fp32 reduction order, not the op: lane
contributions reach ~6.4e5 and cancel to ~4.0, so the surviving error is
ULP-quantized at the partial-sum magnitude and lands one to two binades
apart between sequential (CPU BLAS) and blocked (A100 cuBLAS) orders.
A targeted factor-of-8 ratio guard on piecewise_linear's staircase_*
distributions covers both orders while still catching a genuine
decomposition regression; the global 40% tolerance is unchanged for
every other row. Verified: passes locally and on an A100 (previously
the suite's only failure). Findings entry updated; restructuring the
lane decomposition remains the open principled fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: physicsrob

docs/numerical_noise_findings.md

@@ -34,11 +34,18 @@ a property of the summation order — which differs between local CPU BLAS
 and A100 cuBLAS — and the op's real defect is catastrophic cancellation in
 its lane decomposition.
 
-**Status: decision pending (spans_plan.md decision 2).** Either pin the
-probe's device and reduction so the committed number means something, or
-restructure the lane decomposition so contributions don't blow up.  Do
-**not** re-run `make measure-noise` on whichever machine is to hand — that
-re-pins one device's reduction order and papers over the defect again.
+**Status: decided 2026-07-22 (Rob) — the drift gate tolerates both
+reduction orders.** `test_numerical_noise_drift.py` now applies a targeted
+factor-of-8 ratio guard to the `staircase_*` rows of `piecewise_linear`
+(both machines) instead of the global 40% tolerance: the observed
+CPU-sequential vs A100-blocked swing is 2–4x and ULP-quantized, and a
+genuine decomposition regression moves these numbers by orders of
+magnitude, so the wider guard still catches it.  The committed numbers
+remain the sequential-order row.  Restructuring the lane decomposition so
+contributions don't blow up remains open as the principled fix.  The
+warning still stands: do **not** re-run `make measure-noise` on whichever
+machine is to hand to "fix" a staircase diff — that just re-pins the other
+reduction order.
 
 ### MLP-side cancel on the swish machine: worst residue 1.5e-8, gate passes (2026-07-07)
 

tests/docs/test_numerical_noise_drift.py

@@ -60,6 +60,21 @@
 # exact on every kernel. Absolute moves above 5e-4 are genuine drift; tight
 # per-op budgets live in the ops' unit tests, not here.
 _ATOL = 5e-4
+# The staircase rows measure fp32 GEMM reduction order, not the op: the
+# decomposition sums lanes whose contributions reach ~6.4e5 and cancel to
+# ~4.0, so the surviving error is a small multiple of the fp32 ulp at the
+# partial-sum magnitude — quantized in powers of two, and one to two binades
+# apart between sequential (local CPU BLAS) and blocked (A100 cuBLAS)
+# reduction orders (largest observed swing: p99_abs_error 4x, 0.03125 ->
+# 0.125). A factor-of-8 ratio guard covers both orders with one binade of
+# headroom while still catching a genuine decomposition regression, which
+# moves these numbers by orders of magnitude. See
+# docs/numerical_noise_findings.md § staircase.
+_STAIRCASE_RATIO = 8.0
+
+
+def _is_staircase(op_key: tuple, dist_name: str) -> bool:
+    return op_key[1] == "piecewise_linear" and dist_name.startswith("staircase_")
 
 
 def _close_enough(a: float, b: float) -> bool:
@@ -72,6 +87,21 @@ def _close_enough(a: float, b: float) -> bool:
     return abs(a - b) <= _ATOL + _RTOL * max(abs(a), abs(b))
 
 
+def _ratio_close(a: float, b: float, factor: float) -> bool:
+    """True when the two magnitudes are within ``factor`` of each other —
+    the machine-dependent-row guard (see _STAIRCASE_RATIO)."""
+    if a is None and b is None:
+        return True
+    if a is None or b is None:
+        return False
+    if a == b:
+        return True
+    lo, hi = sorted((abs(a), abs(b)))
+    if lo == 0.0:
+        return hi <= _ATOL
+    return hi / lo <= factor
+
+
 def _compare_ops(committed: dict, regenerated: dict) -> List[str]:
     """Compare two stripped JSON dicts and return a list of failure messages.
 
@@ -116,7 +146,11 @@ def _compare_ops(committed: dict, regenerated: dict) -> List[str]:
 
             for key in _ERROR_METRIC_KEYS:
                 cv, rv = cd[key], rd[key]
-                if not _close_enough(cv, rv):
+                if _is_staircase(op_key, dist_name):
+                    ok = _ratio_close(cv, rv, _STAIRCASE_RATIO)
+                else:
+                    ok = _close_enough(cv, rv)
+                if not ok:
                     failures.append(f"{op_name}/{dist_name}: {key} " f"{cv} -> {rv}")
 
     return failures