table_lookup: widen output-range guard slack for GPU cross-test variation

The row-vector / staircase output-range *guards* (assert_matches_value_type,
not the correctness path) used a per-element slack of
max_abs*sharpness*n_steps*1e-6, calibrated for fp32 single-run noise. At the
16x128x128 target (rows = A*B = 2048) this leaves no headroom for GPU
cross-test FP variation (cuBLAS algorithm selection / TF32), so the guard
tripped intermittently in the full sharded suite while passing in isolation
and under fp32 (op math overshoots the table range by only ~1.4e-5 in every
reproducible config). Bump the unit to 1e-5 — the GPU-variation scale noted
in the numerical-noise docs — giving the guard margin without loosening the
caller's value-match correctness test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: physicsrob

torchwright/ops/map_select.py

@@ -120,7 +120,18 @@ def _constant_vector(values: torch.Tensor, name: str) -> Node:
 
 
 def _lookup_numeric_slack(max_abs: float, sharpness: float, n_steps: int) -> float:
-    return max(1e-3, max_abs * sharpness * max(n_steps, 1) * 1e-6)
+    # Per-element slack for the row-vector / staircase output-range *guards*
+    # (assert_matches_value_type), not the correctness path. The guard must
+    # have margin above accumulated fp32 noise in the wide PWL AND above GPU
+    # cross-test FP variation (cuBLAS algorithm selection / TF32), which the
+    # noise notes peg at ~1e-5..1e-6 — an order of magnitude above the fp32
+    # single-run unit. At the 16x128x128 target (rows = A*B = 2048) the hidden
+    # PWL activations reach ~20k*sharpness, so reduced-precision matmul on A100
+    # can push the guarded value to ~max_abs*sharpness*rows*1e-5; a 1e-6 budget
+    # tripped intermittently in the full sharded suite (passes in isolation and
+    # under fp32). Using 1e-5 gives the guard headroom for GPU variation without
+    # loosening real correctness (the caller's value-match test stays tight).
+    return max(1e-3, max_abs * sharpness * max(n_steps, 1) * 1e-5)
 
 
 def _table_lookup_row_vector(