transformerless_lm: weight-substrate bench results — A close, B saturated

Principle A (tied QKV via Fibonacci channel rotation):
  dense_crt fp32:       2.4396
  tied_substrate fp32:  2.5927   (+6.3%, 2.02x attention param reduction)

Within 1.3 percentage points of the 5% validation threshold. The
substrate constraint (W learns a representation whose Q, K, V views
are interderivable by channel rotation) DID train. The 6.3% gap is
small enough to plausibly close with longer training, multi-head
tied attention (different F_K/F_V per head), or attenuable tying
(learnable mix between identity and rotation). Not formally validated
at threshold, but very close.

Principle B (Fibonacci-tier weight quantization) FALSIFIED as
implemented. Quantization saturates at ~+0.96 nats loss regardless
of bit depth:
  n_tiers=4  (~3 bit):  +5.41
  n_tiers=8  (~4 bit):  +0.98
  n_tiers=16 (~5 bit):  +0.96
  n_tiers=32 (~6 bit):  +0.96

The plateau diagnoses the cause: with a global per-tensor scale set
by the max-magnitude weight, the Fibonacci tier values {1, 2, 3,
5, 8, 13, ...} all land FAR from where most weights live (near 0,
Gaussian-distributed). Avg unique tier values used:
  n_tiers=32: 24.2 of 63 levels (38%) — many tiers unused.

The tail Fibonacci values (89, 144, 233, ...) are never used because
no weight has magnitude that large. The bottleneck is the lack of
fine-grained tier values BETWEEN 0 and 1.

Two fixes worth testing in a follow-up:
  1. Include reciprocal Fibonacci values 1/F(k) so the tier set has
     fine resolution near 0: {..., 1/13, 1/8, 1/5, 1/3, 1/2, 1, 2,
     3, 5, 8, 13, ...} — geometric spacing by phi^pi crossings.
  2. Per-output-channel scale (instead of global per-tensor scale) so
     each row of W gets its own tier resolution matched to its
     magnitude range.

Either fix may rescue Principle B. The underlying claim (language is
low-Fibonacci-rank) is not falsified; what's falsified is the naive
"snap each weight to the nearest large Fibonacci value" implementation.

Author: claude

experiments/transformerless_lm/results_weight_substrate.json

@@ -0,0 +1,66 @@
+{
+  "archs": {
+    "dense_crt": {
+      "n_params": 801664,
+      "n_attn_params": 264192,
+      "val_fp32": 2.439574755728245,
+      "quantized": {
+        "4": {
+          "val": 7.848854899406433,
+          "delta": 5.409280143678188,
+          "params_quantized": 801664,
+          "avg_unique_tier_values": 5.784313725490196
+        },
+        "8": {
+          "val": 3.4171799793839455,
+          "delta": 0.9776052236557007,
+          "params_quantized": 801664,
+          "avg_unique_tier_values": 12.254901960784315
+        },
+        "16": {
+          "val": 3.404290087521076,
+          "delta": 0.9647153317928314,
+          "params_quantized": 801664,
+          "avg_unique_tier_values": 19.96078431372549
+        },
+        "32": {
+          "val": 3.4040319994091988,
+          "delta": 0.964457243680954,
+          "params_quantized": 801664,
+          "avg_unique_tier_values": 24.235294117647058
+        }
+      }
+    },
+    "tied_substrate": {
+      "n_params": 668536,
+      "n_attn_params": 131072,
+      "val_fp32": 2.592747889459133,
+      "quantized": {
+        "4": {
+          "val": 6.7923648953437805,
+          "delta": 4.199617005884647,
+          "params_quantized": 668536,
+          "avg_unique_tier_values": 6.373831775700935
+        },
+        "8": {
+          "val": 3.584361217916012,
+          "delta": 0.9916133284568787,
+          "params_quantized": 668536,
+          "avg_unique_tier_values": 13.598130841121495
+        },
+        "16": {
+          "val": 3.6512366607785225,
+          "delta": 1.0584887713193893,
+          "params_quantized": 668536,
+          "avg_unique_tier_values": 22.289719626168225
+        },
+        "32": {
+          "val": 3.6507833153009415,
+          "delta": 1.0580354258418083,
+          "params_quantized": 668536,
+          "avg_unique_tier_values": 26.14018691588785
+        }
+      }
+    }
+  }
+}