Experiment 7: phi_pi_fib search wired in as OMC builtins, sublinear

Four new builtins exposing omnimcode-core/src/phi_pi_fib.rs to OMC:

  phi_pi_fib_search(arr, target)  -> int       exact match or -(insert+1)
  phi_pi_fib_nearest(arr, target) -> int       always-valid nearest index
  phi_pi_fib_stats()              -> [s, cmps] global counters
  phi_pi_fib_reset()              -> null      zero the counters

phi_pi_fib_nearest is the gate primitive: missing-key lookups route to
the nearest surviving library entry. Returns -1 only for an empty
array; otherwise always a valid index. This is what gives the
compression-gate architecture its "die gracefully" property.

Experiment 7 reruns experiment 6's compression gate on top of the
real Fibonacci-step search and measures the work:

  N=8     ->  3.79 compares/search
  N=16    ->  5.25
  N=32    ->  6.18
  N=64    ->  8.10
  N=128   ->  8.67
  N=256   -> 10.33
  N=512   -> 11.29
  N=1024  -> 12.57

128x library growth -> ~3.3x compare growth. Tracks log_2(N) closely
(slightly better than log_phi(N) = 1.44 * log_2(N)). A linear scan
would show avg_compares ~ N/2, growing 64x over the same range.

Sanity: chain [3, 8, 13, 5, 21] on state 7 -> final state 9, matching
experiment 6. Death tolerance: same per-deletion shifts as exp 6.
All 12 deletions complete without crashing.

148/148 existing tests still pass. Both engines audit byte-identical
on the experiment file (2627 bytes).

Stops here for review before experiment 8 (learnable routing policy).

Author: claude

experiments/hybrid_llm/README.md

@@ -91,6 +91,7 @@ OMC_VM=1 ./target/release/omnimcode-standalone experiments/hybrid_llm/experiment
 | 4B | Same gates, **magnitude-matched** structural OOD (inverted attractor weights: 10%/30%/60% small/med/large vs in-dist's 60%/30%/10%). | **Harmonic edges past L2 in AUROC: 0.956 vs 0.946.** At low FPR L2 still wins (TPR@FPR=1%: L2 0.60 vs harmonic 0.48), but on overall ranking the structural rarity signal beats the L2 metric once magnitude is no longer a giveaway. |
 | 5 | HBit cross-cutting tension (no reference) + combined gate (sum of z-normalised HBit, marginal rarity, L2) on both scenarios. | **Scenario A: HBit tension AUROC = 1.0** (perfect — mean tension 0.0 in-dist vs 20.1 OOD). Combined: 0.999. **Scenario B: HBit AUROC = 0.5** (random — both sides on-manifold, tension = 0 everywhere). Combined: 0.967, beating every single gate. Each gate owns a different OOD axis: HBit→off-manifold, marginal→distribution-shift, L2→magnitude. |
 | 6 | Phi-Pi-Fib compression gate: model as `(library + chain of keys)` instead of dense weights. 12-primitive library keyed by Fibonacci attractors, gate = nearest-key lookup, chains = "parameters". | Composition: trace `[3, 8, 13, 5, 21]` on state 7 → 9. Compression: 29 ints (library+chain) vs ~1001 ints dense table over [0,1000] = ~34× smaller (extrapolates to 9 orders of magnitude at LLM scale). **Death tolerance: all 12 library deletions complete without crashing — biggest deltas: kill key=13 → +12, kill key=5 → +5, kill key=21 → +3. 8 of 12 deletions invisible to output (unused capabilities or path coincidence).** Interchangeability: 6 different chains over the same library yield 6 different outputs (9, 22, 9, 5, 5, 52). |
+| 7 | Wire `phi_pi_fib::fibonacci_search` in as four OMC builtins (`phi_pi_fib_search`, `phi_pi_fib_nearest`, `phi_pi_fib_stats`, `phi_pi_fib_reset`). Rerun exp 6's gate using the real Fibonacci-step search; measure comparison counts vs library size. | **Sublinear scaling confirmed.** N=8 → 3.8 compares/search, N=1024 → 12.6. Going 128× wider in library size grows the per-lookup work only ~3.3×, vs ~64× for a linear scan. Empirically tracks `~log₂(N)`, slightly better than `log_φ(N) ≈ 1.44·log₂(N)`. Sanity check passes (same final state as exp 6). Death tolerance preserved across all 12 library deletions. 148/148 existing tests still pass. |
 
 ### Cumulative read across experiments 0–5
 
@@ -170,9 +171,8 @@ result.
 - **5** HBit cross-cutting tension + 3-gate combined detector. ✓ done
 - **6** Phi-Pi-Fib compression gate: model = library + chain. ✓ done
 - **7** Wire `omnimcode-core/src/phi_pi_fib.rs::fibonacci_search` in
-  as an OMC builtin so the gate uses sublinear search instead of the
-  linear scan in experiment 6. Semantics identical at small library
-  size; necessary at 10^6+ entries.
+  as four OMC builtins; rerun exp 6's gate on top; measure compare
+  counts. ✓ done
 - **8** Learnable routing policy: a function `state -> chain` that
   picks WHICH chain to run from input state. Start with a simple
   hand-authored policy (if state on small attractor use chain A,

experiments/hybrid_llm/experiment_7_phi_fib_gate.omc

@@ -0,0 +1,305 @@
+# =============================================================================
+# Experiment 7 — Compression gate backed by the real phi-pi-fib search.
+#
+# Experiment 6 used a linear scan inside the gate. That stands in for the
+# Fibonacci-step search at small library sizes but makes the "sublinear
+# lookup" claim implied rather than proven. This experiment wires the
+# Rust phi_pi_fib::fibonacci_search in as four new OMC builtins and
+# reruns the compression-gate machinery on top:
+#
+#   phi_pi_fib_search(sorted_arr, target) -> int
+#       Fibonacci-step binary search. Returns the exact-match index when
+#       found, or -(insert_pos + 1) when not found (Rust binary_search
+#       sign convention).
+#
+#   phi_pi_fib_nearest(sorted_arr, target) -> int
+#       Same search, returns the index of the nearest entry by absolute
+#       integer distance. -1 only when the array is empty. This is the
+#       gate primitive — missing-key lookups route to nearest surviving
+#       entry.
+#
+#   phi_pi_fib_stats() -> [total_searches, total_comparisons]
+#       Global counters since last reset. Used here to measure that
+#       the gate's cost grows as O(log_phi n), not O(n).
+#
+#   phi_pi_fib_reset() -> null
+#       Zero the counters.
+#
+# What this experiment shows:
+#
+#   (i)   The new builtins audit byte-identical across tree-walk and
+#         bytecode VM. The phi-pi-fib search lifts cleanly into OMC.
+#   (ii)  Sublinear behaviour: average comparisons per search across
+#         libraries of increasing size scales sub-linearly with N.
+#   (iii) The exp-6 chain produces the SAME final state under the new
+#         gate, because phi_pi_fib_nearest agrees with the linear-scan
+#         nearest-search on this library (sanity check).
+#   (iv)  The dead-key fallback still works: kill one library entry,
+#         rerun, observe re-routing through the nearest survivor.
+#
+# Run:
+#   ./target/release/omnimcode-standalone experiments/hybrid_llm/experiment_7_phi_fib_gate.omc
+# =============================================================================
+
+# ---------------------------------------------------------------------------
+# Library — same 12 primitives as experiment 6, but keys live in an
+# actual sorted array so the search can index it directly.
+# ---------------------------------------------------------------------------
+
+h LIB_KEYS = [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233];
+h LIB_OPS  = [0, 1, 2, 3, 4,  5,  6,  7,  8,  9,  10,  11];
+h LIB_SIZE = arr_len(LIB_KEYS);
+
+fn apply_op(op, x) -> int {
+    if op == 0  { return x; }
+    if op == 1  { return x + 1; }
+    if op == 2  { return x * 2; }
+    if op == 3  { return phi.fold(x); }
+    if op == 4  { return x + phi.fold(x); }
+    if op == 5  { return to_int(harmonic_interfere(x, 5)); }
+    if op == 6  { return to_int(harmonic_interfere(x, 13)); }
+    if op == 7  { return x * x; }
+    if op == 8  { return phi.fold(x * 2); }
+    if op == 9  { return x - phi.fold(x); }
+    if op == 10 { return to_int(harmonic_interfere(x, 21)); }
+    if op == 11 { return phi.fold(x + 7); }
+    return x;
+}
+
+fn op_name(op) -> string {
+    if op == 0  { return "identity"; }
+    if op == 1  { return "increment"; }
+    if op == 2  { return "double"; }
+    if op == 3  { return "fold"; }
+    if op == 4  { return "add_fold"; }
+    if op == 5  { return "interfere_5"; }
+    if op == 6  { return "interfere_13"; }
+    if op == 7  { return "square"; }
+    if op == 8  { return "double_fold"; }
+    if op == 9  { return "residual"; }
+    if op == 10 { return "interfere_21"; }
+    if op == 11 { return "phi_shift_fold"; }
+    return "?";
+}
+
+# ---------------------------------------------------------------------------
+# The new gate: keys and ops live in two parallel arrays. The gate masks
+# dead entries by COPYING the alive keys into a smaller sorted array, then
+# running phi_pi_fib_nearest on that. The op_index returned is into the
+# ALIVE arrays, so we project back to the underlying op_id.
+# ---------------------------------------------------------------------------
+
+fn alive_keys_and_ops(alive_mask) -> array {
+    h alive_keys = [];
+    h alive_ops = [];
+    h i = 0;
+    while i < LIB_SIZE {
+        if arr_get(alive_mask, i) == 1 {
+            arr_push(alive_keys, arr_get(LIB_KEYS, i));
+            arr_push(alive_ops,  arr_get(LIB_OPS,  i));
+        }
+        i = i + 1;
+    }
+    h pack = arr_new(2, 0);
+    arr_set(pack, 0, alive_keys);
+    arr_set(pack, 1, alive_ops);
+    return pack;
+}
+
+# Apply the chain using phi-pi-fib nearest-key search at each step.
+# Returns final state; the comparison-count counter records total
+# work inside the gate via phi_pi_fib_stats().
+fn run_chain_with_phi_fib(initial_state, keys, alive_mask) -> int {
+    h packed = alive_keys_and_ops(alive_mask);
+    h alive_keys = arr_get(packed, 0);
+    h alive_ops  = arr_get(packed, 1);
+    h n_alive = arr_len(alive_keys);
+
+    h state = initial_state;
+    h n = arr_len(keys);
+    h i = 0;
+    while i < n {
+        h target = arr_get(keys, i);
+        h idx = phi_pi_fib_nearest(alive_keys, target);
+        if idx >= 0 {
+            h op = arr_get(alive_ops, idx);
+            state = apply_op(op, state);
+        }
+        i = i + 1;
+    }
+    return state;
+}
+
+fn run_chain_traced_with_phi_fib(initial_state, keys, alive_mask) -> int {
+    h packed = alive_keys_and_ops(alive_mask);
+    h alive_keys = arr_get(packed, 0);
+    h alive_ops  = arr_get(packed, 1);
+
+    h state = initial_state;
+    h n = arr_len(keys);
+    h i = 0;
+    while i < n {
+        h target = arr_get(keys, i);
+        h idx = phi_pi_fib_nearest(alive_keys, target);
+        if idx >= 0 {
+            h actual_key = arr_get(alive_keys, idx);
+            h op = arr_get(alive_ops, idx);
+            h prev = state;
+            state = apply_op(op, state);
+            print(concat_many(
+                "  step ", i, ": target=", target,
+                " -> alive_idx=", idx,
+                " (key=", actual_key, ", op=", op_name(op), ")",
+                "  state ", prev, " -> ", state
+            ));
+        }
+        i = i + 1;
+    }
+    return state;
+}
+
+fn alive_all() -> array {
+    h m = arr_new(LIB_SIZE, 0);
+    h i = 0;
+    while i < LIB_SIZE { arr_set(m, i, 1); i = i + 1; }
+    return m;
+}
+
+fn alive_kill(base, kill_idx) -> array {
+    h m = arr_new(LIB_SIZE, 0);
+    h i = 0;
+    while i < LIB_SIZE { arr_set(m, i, arr_get(base, i)); i = i + 1; }
+    arr_set(m, kill_idx, 0);
+    return m;
+}
+
+# ===========================================================================
+# (i) Sanity — chain trace under the new gate.
+# ===========================================================================
+
+print("== Experiment 7: compression gate backed by phi_pi_fib_nearest ==");
+print("");
+print(concat_many("Library keys = ", LIB_KEYS));
+print("");
+
+h chain_main = [3, 8, 13, 5, 21];
+h initial = 7;
+h alive = alive_all();
+
+phi_pi_fib_reset();
+print(concat_many("Chain: ", chain_main, "   initial state = ", initial));
+print("Trace:");
+h final_main = run_chain_traced_with_phi_fib(initial, chain_main, alive);
+print(concat_many("Final state: ", final_main));
+h stats_main = phi_pi_fib_stats();
+print(concat_many(
+    "  gate work: searches=", arr_get(stats_main, 0),
+    "  comparisons=", arr_get(stats_main, 1)
+));
+print("");
+
+# ===========================================================================
+# (ii) Sublinear scaling — comparisons per search across growing libraries.
+#
+# We synthesise sorted arrays of length N over consecutive integers, then
+# run M random nearest-key lookups against each. For phi_pi_fib_search
+# this should scale roughly as M * log_phi(N) comparisons.
+# ===========================================================================
+
+fn build_sorted_n(n) -> array {
+    h arr = arr_new(n, 0);
+    h i = 0;
+    while i < n {
+        arr_set(arr, i, i * 2 + 1);   # 1, 3, 5, ..., 2n-1
+        i = i + 1;
+    }
+    return arr;
+}
+
+fn measure_avg_compares(arr, n_queries) -> float {
+    h n = arr_len(arr);
+    h max_target = arr_get(arr, n - 1) + 5;
+    phi_pi_fib_reset();
+    h k = 0;
+    while k < n_queries {
+        h target = random_int(0, max_target);
+        h _idx = phi_pi_fib_nearest(arr, target);
+        k = k + 1;
+    }
+    h s = phi_pi_fib_stats();
+    return to_float(arr_get(s, 1)) / to_float(arr_get(s, 0));
+}
+
+random_seed(42);
+print("== (ii) Sublinear scaling of phi_pi_fib_nearest ==");
+print(concat_many("(random queries against sorted [1,3,5,...,2N-1]; ",
+                  "200 queries per N)"));
+print("");
+print("    N    avg_compares  log_phi(N)");
+print("  -----  ------------  ----------");
+
+h sizes = [8, 16, 32, 64, 128, 256, 512, 1024];
+h ns = arr_len(sizes);
+h si = 0;
+while si < ns {
+    h N = arr_get(sizes, si);
+    h arr_n = build_sorted_n(N);
+    h avg = measure_avg_compares(arr_n, 200);
+    # log_phi(N) = ln(N) / ln(phi)
+    h lphi = log(to_float(N)) / log(1.6180339887498948);
+    print(concat_many("  ", N, "       ", avg, "       ", lphi));
+    si = si + 1;
+}
+print("");
+print("If avg_compares grows like log_phi(N) (about 1.44 * log2(N)), the");
+print("search is doing sublinear work — the gate's cost stays bounded as");
+print("the library grows. A linear scan would show avg_compares ~ N/2.");
+print("");
+
+# ===========================================================================
+# (iii) Sanity vs experiment 6 — same chain, same final state.
+# ===========================================================================
+
+print("== (iii) Sanity: same chain, same final state as experiment 6 ==");
+print(concat_many("phi-fib gate final state:  ", final_main));
+print("experiment 6 final state:  9");
+print("(should match exactly — both gates pick the same nearest entry");
+print("when keys match exactly.)");
+print("");
+
+# ===========================================================================
+# (iv) Death tolerance, re-measured under phi-fib search.
+# ===========================================================================
+
+print("== (iv) Death tolerance under phi_pi_fib_nearest ==");
+print("");
+print("Kill | final state | shift from baseline");
+print("-----+-------------+--------------------");
+
+h kill_idx = 0;
+while kill_idx < LIB_SIZE {
+    h km = alive_kill(alive, kill_idx);
+    h killed = run_chain_with_phi_fib(initial, chain_main, km);
+    h shift = killed - final_main;
+    if shift < 0 { shift = 0 - shift; }
+    h k_key = arr_get(LIB_KEYS, kill_idx);
+    h k_op  = arr_get(LIB_OPS,  kill_idx);
+    print(concat_many(
+        "  key=", k_key, " (", op_name(k_op), ")",
+        "  -> ", killed,
+        "    delta=", shift
+    ));
+    kill_idx = kill_idx + 1;
+}
+print("");
+
+print("== Summary ==");
+print("- Four new builtins: phi_pi_fib_search, phi_pi_fib_nearest,");
+print("  phi_pi_fib_stats, phi_pi_fib_reset. Audit clean against the VM.");
+print("- Comparison counters confirm the search is sublinear in N.");
+print("- The compression-gate prototype now uses the real Fibonacci-step");
+print("  search instead of a linear scan. Semantics match experiment 6 on");
+print("  this library (sanity), death tolerance preserved.");
+print("- Ready for experiment 8: learnable routing policy that emits");
+print("  chains from inputs (state -> chain_of_keys).");
+print("== End ==");