Close exp 9 measurement gap: traced search variants, real coherence

Two new OMC builtins return [result, probe_indices_array]:

  phi_pi_fib_search_traced(arr, target)   -> [int, array]
  phi_pi_fib_nearest_traced(arr, target)  -> [int, array]

Backed by a new fibonacci_search_with_trace in phi_pi_fib.rs that
collects probed indices alongside the normal search and shares the
global compare counters.

Re-running experiment 9 with traced phi_pi_fib gives REAL coherence
numbers instead of the prior 0/0 measurement gap:

  linear scan:    step coherence  2800/2800 = 1.000  (degenerate)
  binary search:  step coherence   392/592  = 0.662  (~34% NOT Fibonacci)
  phi_pi_fib:     step coherence   821/916  = 0.896  (~90% ARE Fibonacci)

The phi_pi_fib trace for phi.fold(50) is now visible:
  probes [14, 13, 8, 12, 11, 10, 10]
  step sizes [1, 5, 4, 1, 1, 0] — 5 of 6 are Fibonacci, the lone
  outlier (4) comes from an offset-reset after a "go right" branch.

So phi_pi_fib is empirically ~90% substrate-coherent, not 100% as I
analytically claimed. Binary's apparent 66% is partly chance —
small-attractor tables happen to have Fibonacci-valued midpoint
step sizes (1, 2, 3, 8); at larger tables binary's halving step
sizes (16, 32, 64, 128) are all non-Fibonacci, so its coherence
should DROP while phi_pi_fib's stays high.

Path is now: phi_pi_fib is the substrate-coherent choice for the
foundational phi.fold and HInt::compute_resonance operations. Next
commit does the refactor — promote phi_pi_fib::FIBONACCI to canonical,
delete the 17 duplicate fibs tables, rewrite fold_to_fibonacci_const
and HInt::compute_resonance to use phi_pi_fib_nearest.

Both engines audit byte-identical (3114 bytes). 148/148 tests pass.

Author: claude

experiments/hybrid_llm/experiment_9_phi_fold_alignment.omc

@@ -129,19 +129,19 @@ fn phi_fold_binary(x) -> array {
 }
 
 # Variant C: Fibonacci-step search via the substrate's own algorithm.
-# We use phi_pi_fib_search but ALSO track compares via the shared counter.
+# Uses the traced builtin so we can measure step-size coherence
+# externally — this closes the measurement gap from the v1 of this
+# experiment.
 fn phi_fold_fibstep(x) -> array {
     h abs_x = x;
     if abs_x < 0 { abs_x = 0 - abs_x; }
     phi_pi_fib_reset();
-    h idx = phi_pi_fib_nearest(FIBS, abs_x);
+    h traced = phi_pi_fib_nearest_traced(FIBS, abs_x);
+    h idx = arr_get(traced, 0);
+    h probes = arr_get(traced, 1);
     h s = phi_pi_fib_stats();
     h compares = arr_get(s, 1);
     h nearest = arr_get(FIBS, idx);
-    # We can't recover the probe trace from the builtin directly.
-    # Instead, emit the index returned + a synthetic trace marker.
-    h probes = [];
-    arr_push(probes, idx);
     h out = arr_new(3, 0);
     arr_set(out, 0, nearest);
     arr_set(out, 1, compares);
@@ -286,18 +286,10 @@ report("linear scan: ", agg_lin);
 report("binary search:", agg_bin);
 report("phi_pi_fib:  ", agg_fib);
 print("");
-print("NOTE on phi_pi_fib coherence: 0/0 above is a MEASUREMENT GAP,");
-print("not a real zero. The builtin phi_pi_fib_nearest doesn't expose");
-print("its internal probe trace, so we can't externally tally its step");
-print("sizes. Analytically, the Rust implementation in phi_pi_fib.rs");
-print("decrements fib_idx by 1 or 2 each iteration and probes at");
-print("offset + F(fib_idx); step sizes between consecutive probes are");
-print("F(k) values, which ARE Fibonacci by construction. So the");
-print("coherence is ~1.0 by design even though this experiment can't");
-print("measure it directly. Fixing the gap requires either adding a");
-print("traced variant to the builtin surface, or porting the search");
-print("to OMC for tracing. Worthwhile follow-up; the compare-count");
-print("savings (and the architectural argument) hold either way.");
+print("Coherence is now measured directly for phi_pi_fib using the");
+print("phi_pi_fib_nearest_traced builtin (new in this commit). Each");
+print("call returns [nearest_index, probes_array]; step coherence is");
+print("computed on consecutive elements of probes_array.");
 print("");
 
 # ---------------------------------------------------------------------------

omnimcode-core/src/interpreter.rs

@@ -1666,6 +1666,8 @@ impl Interpreter {
             // Phi-Pi-Fib search v2 + binary baseline + theoretical bound
             | "phi_pi_fib_search_v2" | "phi_pi_fib_nearest_v2"
             | "phi_pi_bin_search" | "log_phi_pi_fibonacci"
+            // Traced variants — return [result, probe_indices_array]
+            | "phi_pi_fib_search_traced" | "phi_pi_fib_nearest_traced"
             // Self-healing
             | "safe_divide" | "safe_arr_get" | "safe_arr_set"
             | "safe_add" | "safe_sub" | "safe_mul" | "resolve_singularity"
@@ -4133,6 +4135,96 @@ impl Interpreter {
                 let n = self.eval_expr(&args[0])?.to_float();
                 Ok(Value::HFloat(crate::phi_pi_fib::log_phi_pi_fibonacci(n)))
             }
+            // phi_pi_fib_search_traced(sorted_arr, target)
+            //   Returns [result_int, probe_indices_array]. `result_int`
+            //   is the exact-match index when found, or -(insert_pos+1)
+            //   when not. `probe_indices_array` is the sequence of
+            //   indices the Fibonacci-step search visited, in order.
+            //   Used by experiments that need to measure step-size
+            //   coherence externally.
+            "phi_pi_fib_search_traced" => {
+                if args.len() < 2 {
+                    return Err("phi_pi_fib_search_traced requires (sorted_array, target)".to_string());
+                }
+                let arr_v = self.eval_expr(&args[0])?;
+                let target = self.eval_expr(&args[1])?.to_int();
+                if let Value::Array(arr) = arr_v {
+                    let items_b = arr.items.borrow();
+                    let ints: Vec<i64> = items_b.iter().map(|v| v.to_int()).collect();
+                    let (r, probes) = crate::phi_pi_fib::fibonacci_search_with_trace(
+                        &ints,
+                        &target,
+                        |a, b| if a < b { -1 } else if a > b { 1 } else { 0 },
+                    );
+                    let result_int = match r {
+                        Ok(i) => i as i64,
+                        Err(insert_pos) => -(insert_pos as i64 + 1),
+                    };
+                    let probe_vals: Vec<Value> = probes
+                        .into_iter()
+                        .map(|p| Value::HInt(HInt::new(p as i64)))
+                        .collect();
+                    let out = vec![
+                        Value::HInt(HInt::new(result_int)),
+                        Value::Array(HArray::from_vec(probe_vals)),
+                    ];
+                    Ok(Value::Array(HArray::from_vec(out)))
+                } else {
+                    Err("phi_pi_fib_search_traced: first argument must be an array".to_string())
+                }
+            }
+            // phi_pi_fib_nearest_traced(sorted_arr, target)
+            //   Returns [nearest_index, probe_indices_array]. Always
+            //   resolves to a valid nearest index (or -1 for empty arrays).
+            "phi_pi_fib_nearest_traced" => {
+                if args.len() < 2 {
+                    return Err("phi_pi_fib_nearest_traced requires (sorted_array, target)".to_string());
+                }
+                let arr_v = self.eval_expr(&args[0])?;
+                let target = self.eval_expr(&args[1])?.to_int();
+                if let Value::Array(arr) = arr_v {
+                    let items_b = arr.items.borrow();
+                    let ints: Vec<i64> = items_b.iter().map(|v| v.to_int()).collect();
+                    if ints.is_empty() {
+                        let out = vec![
+                            Value::HInt(HInt::new(-1)),
+                            Value::Array(HArray::from_vec(vec![])),
+                        ];
+                        return Ok(Value::Array(HArray::from_vec(out)));
+                    }
+                    let (r, probes) = crate::phi_pi_fib::fibonacci_search_with_trace(
+                        &ints,
+                        &target,
+                        |a, b| if a < b { -1 } else if a > b { 1 } else { 0 },
+                    );
+                    let idx: usize = match r {
+                        Ok(i) => i,
+                        Err(insert_pos) => {
+                            let n = ints.len();
+                            if insert_pos == 0 {
+                                0
+                            } else if insert_pos >= n {
+                                n - 1
+                            } else {
+                                let left = (target - ints[insert_pos - 1]).abs();
+                                let right = (ints[insert_pos] - target).abs();
+                                if right < left { insert_pos } else { insert_pos - 1 }
+                            }
+                        }
+                    };
+                    let probe_vals: Vec<Value> = probes
+                        .into_iter()
+                        .map(|p| Value::HInt(HInt::new(p as i64)))
+                        .collect();
+                    let out = vec![
+                        Value::HInt(HInt::new(idx as i64)),
+                        Value::Array(HArray::from_vec(probe_vals)),
+                    ];
+                    Ok(Value::Array(HArray::from_vec(out)))
+                } else {
+                    Err("phi_pi_fib_nearest_traced: first argument must be an array".to_string())
+                }
+            }
             "arr_slice" => {
                 if args.len() < 3 {
                     return Err("arr_slice requires (array, start, end)".to_string());
@@ -5102,6 +5194,7 @@ pub(crate) const HEAL_BUILTIN_NAMES: &[&str] = &[
     "phi_pi_fib_stats", "phi_pi_fib_reset",
     "phi_pi_fib_search_v2", "phi_pi_fib_nearest_v2",
     "phi_pi_bin_search", "log_phi_pi_fibonacci",
+    "phi_pi_fib_search_traced", "phi_pi_fib_nearest_traced",
     // Self-healing
     "safe_divide", "safe_arr_get", "safe_arr_set",
     "safe_add", "safe_sub", "safe_mul", "resolve_singularity",

omnimcode-core/src/phi_pi_fib.rs

@@ -225,6 +225,56 @@ pub fn phi_pi_fib_search_v2<T>(
     Err(low)
 }
 
+/// fibonacci_search_with_trace — same as fibonacci_search but also
+/// returns the sequence of probed indices, in order. Used by
+/// experiments that need to measure step-size coherence externally.
+/// Counters are updated identically to fibonacci_search so combined
+/// runs still report meaningful totals.
+pub fn fibonacci_search_with_trace<T>(
+    arr: &[T],
+    target: &T,
+    cmp: impl Fn(&T, &T) -> i32,
+) -> (Result<usize, usize>, Vec<usize>) {
+    let mut probes: Vec<usize> = Vec::new();
+    if arr.is_empty() {
+        return (Err(0), probes);
+    }
+    TOTAL_SEARCHES.fetch_add(1, Ordering::Relaxed);
+
+    let mut fib_idx = find_fib_index(arr.len());
+    let mut offset = 0usize;
+    let mut comparisons = 0u64;
+
+    while fib_idx > 0 {
+        comparisons += 1;
+        let fib_val = get_fib(fib_idx) as usize;
+        let mid = (offset + fib_val.min(arr.len() - offset - 1)).min(arr.len() - 1);
+        probes.push(mid);
+
+        let cmp_result = cmp(&arr[mid], target);
+        match cmp_result {
+            0 => {
+                TOTAL_COMPARISONS.fetch_add(comparisons, Ordering::Relaxed);
+                return (Ok(mid), probes);
+            }
+            n if n < 0 => {
+                offset = mid + 1;
+                fib_idx = fib_idx.saturating_sub(2);
+            }
+            _ => {
+                fib_idx = fib_idx.saturating_sub(1);
+            }
+        }
+
+        if offset >= arr.len() {
+            break;
+        }
+    }
+
+    TOTAL_COMPARISONS.fetch_add(comparisons, Ordering::Relaxed);
+    (Err(offset), probes)
+}
+
 /// Standard binary search (for comparison/benchmarking).
 ///
 /// This is provided as a reference implementation to compare against