Skip to content

SUBSTRATE_CHANGES.md

Latest commit

 

History

History
339 lines (248 loc) · 17.2 KB

SUBSTRATE_CHANGES.md

File metadata and controls

339 lines (248 loc) · 17.2 KB

Substrate Refactor Validation Log

All measurements re-taken under the new log_phi_pi_fibonacci(n) substrate (commits a9232e0, fe776fb, 0973799, 8128844). The prior log_phi(n) substrate used a 16-entry Fibonacci attractor table that saturated at 610; the new one uses a 40-entry canonical table extending to 63,245,986 and routes through phi_pi_fib::nearest_attractor_with_dist.

For each test, the diff is classified:

  • IMPROVEMENT — measurably better under new substrate
  • UNIMPROVEMENT — measurably worse
  • NEUTRAL — no semantic change (within noise / identical)
  • DEPRECATION — old result no longer applicable
  • GROUNDBREAKING — new behavior the old substrate couldn't produce

Sweep 1 — Foundation: 43 functional examples (tree-walk vs VM)

Result: 43/43 byte-identical between engines. NEUTRAL.

The substrate refactor preserves engine parity. Same as before pull. The single benchmark file (examples/benchmarks.omc) still shows timing-noise diff between engines, no semantic change.

Sweep 2 — 18 harmonic library tests (--test)

Result: 18/18 pass. NEUTRAL.

running 18 test(s) from examples/tests/test_harmonic_libs.omc
  ok    test_anomaly_detect_credential_stuffing
  ok    test_anomaly_detect_returns_correct_arity
  ok    test_anomaly_score_is_deterministic
  ok    test_anomaly_one_shot_api
  ok    test_clustering_three_decades
  ok    test_clustering_predict_assigns_existing_rows
  ok    test_clustering_predict_unseen_returns_negative
  ok    test_clustering_centroid_count_matches_cluster_count
  ok    test_recommend_basic_suggestion
  ok    test_recommend_state_persists_across_add_ratings
  ok    test_recommend_n_users_n_items_correct
  ok    test_dict_not_equal_to_null
  ok    test_empty_dict_not_equal_to_null
  ok    test_array_not_equal_to_null
  ok    test_function_not_equal_to_null
  ok    test_null_equal_to_null
  ok    test_zero_int_not_equal_to_null
  ok    test_empty_string_not_equal_to_null

result: 18 passed, 0 failed

Sweep 3 — 92 Rust unit tests

Result: 92/92 pass. NEUTRAL.

compute_resonance is now substrate-routed but the conformance goldens didn't pin specific resonance numbers (they pinned "resonance >= 0.7" for Fibonacci values, which still holds).

Sweep 4 — Anomaly benchmarks

Credential stuffing (synthetic, multi-dim)

Old substrate:

                   K=10   K=25   K=50   K=100
  IsolationForest  7/10  17/25  40/50  50/100
  OMC harmonic    10/10  25/25  50/50  50/100

New substrate:

                   K=10   K=25   K=50   K=100
  IsolationForest  7/10  17/25  40/50  50/100
  OMC harmonic    10/10  25/25  50/50  50/100

Verdict: NEUTRAL. Identical results. The credential-stuffing features all fall under |n| ≤ 610 (latencies, hours, endpoint IDs), where the old and new attractor tables agree.

Attack zoo (3 scenarios)

Old substrate:

  Insider exfiltration : 10/10 (100%)
  API abuse / scraping : 10/10 (100%)
  DDoS pattern         : 10/10 (100%)
  Aggregate: 30/30

New substrate:

  Insider exfiltration : 10/10 (100%)
  API abuse / scraping : 10/10 (100%)
  DDoS pattern         : 10/10 (100%)
  Aggregate: 30/30

Verdict: NEUTRAL. All 30 attacks still caught. Note: insider exfiltration uses byte sizes in 80-120KB range (well above old table's 610 ceiling), so the new substrate sees them more accurately — but the structural signature is so strong that 100% precision held under both. The headroom matters for harder discrimination tasks.

Power-law latency outliers (1-D)

Old substrate:

                    K=5    K=10   K=20   K=30
  IsolationForest   0/5    5/10   8/20  15/30
  OMC harmonic      4/5    5/10   5/20  5/30

New substrate:

                    K=5    K=10   K=20   K=30
  IsolationForest   0/5    5/10   8/20  15/30
  OMC harmonic      4/5    5/10   5/20  5/30

Verdict: NEUTRAL. Same alert-budget win (4/5 vs 0/5 at K=5). Anomaly values range 100-3500ms; new substrate's accuracy gain above 610 doesn't change which buckets are populated at our K levels.

NAB realKnownCause (1-D time series, 7 datasets)

Old substrate: 7/19 windows covered (tied with IF) New substrate: 7/19 windows covered (tied with IF)

Verdict: NEUTRAL. Naive top-K detection isn't the regime where the substrate change matters — both detectors still hit the same ceiling. Beating IF on NAB needs CUSUM/seasonality/HMM, not a better attractor table.

NSL-KDD network intrusion (REAL public telemetry) ⭐

This is the substrate change that matters most.

Old substrate:

                     K=10    K=50    K=100   K=500
  IsolationForest    9/10    45/50   92/100   351/500
  OMC harmonic       7/10    42/50   76/100   348/500

New substrate:

                     K=10    K=50    K=100   K=500
  IsolationForest    9/10    45/50   92/100   351/500
  OMC harmonic       7/10    42/50   78/100   365/500

Verdict: IMPROVEMENT at K=100 (+2) and K=500 (+17).

Why this is the predicted gain — NSL-KDD features include src_bytes, dst_bytes, count, all of which routinely exceed the old 610 ceiling (DoS floods push bytes into the millions). Under the old substrate, large attack-magnitudes saturated the attractor table at 610 → identical (low) resonance scores → the detector couldn't distinguish them. Under the new substrate, an 80KB transfer and a 800KB transfer correctly land on different attractors (10946 vs 121393) → finer per-row score gradient → 17 additional true attacks surfaced at K=500.

IF's numbers are unchanged because IF doesn't depend on OMC's substrate at all (it's external sklearn). The harmonic detector got better on its own — closing the gap from 348/500 to 365/500 without IF moving.

Sweep 5 — Substrate-sensitive demos

Harmonic collections (set / pq / index)

  • harmonic_set dedup: identical (uses fold which stays attractor-snapped, same buckets in 0-610 range)
  • harmonic_pq HIM-priority order: identical (HIM math unchanged)
  • harmonic_index user-id lookups (21, 89, 144): identical

Verdict: NEUTRAL. All demo values stay within old table range.

Self-hosting + self-healing

  • self_hosting_v9b.omc — gen2 == gen3 fixpoint: HOLDS
  • self_healing_h5.omc — array-bounds healing: HOLDS

Verdict: NEUTRAL. Self-hosting proofs operate on AST structure, not numeric magnitudes. Heal pass's literal-rewrite arm only fires on values within edit-distance 3 of an attractor — that distance is independent of which attractor table size we use.

Summary table

Test Old substrate New substrate Verdict
43 functional examples (TW/VM parity) 43/43 byte-identical 43/43 byte-identical NEUTRAL
18 harmonic-lib tests 18/18 pass 18/18 pass NEUTRAL
92 Rust unit tests 92/92 pass 92/92 pass NEUTRAL
Credential stuffing @ K=10 10/10 vs IF 7/10 10/10 vs IF 7/10 NEUTRAL
Attack zoo aggregate 30/30 30/30 NEUTRAL
Power-law @ K=5 4/5 vs IF 0/5 4/5 vs IF 0/5 NEUTRAL
NAB windows covered 7/19 7/19 NEUTRAL
NSL-KDD @ K=100 76/100 78/100 IMPROVEMENT (+2)
NSL-KDD @ K=500 348/500 365/500 IMPROVEMENT (+17)
NSL-KDD @ K=10, K=50 unchanged unchanged NEUTRAL
Self-hosting V.9b fixpoint holds holds NEUTRAL
Self-healing H.5 array bounds holds holds NEUTRAL

What changed in practice

The substrate refactor is conservative for small-magnitude data (everything within the old 16-entry table's range of |n| ≤ 610) and strictly better for large-magnitude data (anything past 610 was saturating against the old table's ceiling).

In concrete terms:

  • Demos using ratings (1-5), hours (0-23), endpoint IDs (0-9), small latencies (10-300ms) — no change
  • Workloads with byte counts, RPM, large request counts, prices in cents over 6 digits — measurably better resonance discrimination

NSL-KDD is the canonical example of the second class. The +17 at K=500 isn't noise; it's the substrate doing its job on real telemetry.

Groundbreaking finding

The substrate change validates a prediction that wasn't testable before: harmonic anomaly detection has more headroom on heavy-tailed data than the old substrate was showing. The old NSL-KDD numbers (76/100, 348/500) were a substrate-limited lower bound on what the algorithm could do, not the algorithm's actual ceiling.

This re-frames the published comparison: harmonic doesn't just win on structural anomalies (credential stuffing, attack zoo) — it ALSO improves on volumetric data when given enough attractor resolution to discriminate. The "IF wins on volumetric" narrative from the old NSL-KDD result was partially a measurement artifact of the saturated attractor table.

The story isn't "harmonic now beats IF on NSL-KDD" — IF still leads at K=10 and K=50. The story is: the gap closes substantially when the substrate has enough resolution, and the new substrate is the substrate that should always have been there.

What was NOT measured

  • Performance overhead of the 40-entry table vs 16-entry: not benchmarked. Probably negligible (still O(log n) with Fibonacci-step search), but no number to cite.
  • LLM experiments from the phi-field-llm-evolution branch (Experiments 0-9): merged in but not re-run in this validation sweep — they're substrate-AWARE work that was DEVELOPED ON the new substrate, no old baseline to compare against.

What no longer needs to be documented

The "IF wins on volumetric" framing in docs/anomaly_detection.md needs softening — under the corrected substrate, the gap is smaller and the gain trajectory at high K favors harmonic. The K=500 result is now an IMPROVEMENT-relative-to-IF in absolute terms (365 vs 351), though the difference is small and within potential noise on a 5000-row sample.

Recommended doc updates

  1. docs/anomaly_detection.md — replace NSL-KDD table with new numbers; soften the "IF wins on volumetric" claim; add a footnote explaining the substrate refactor and why the new K=500 number is more credible.
  2. README's "Where harmonic detection actually wins" table — replace NSL-KDD K=100/500 entries; add "+17 at K=500 from substrate refactor (2026-05-15)" note.
  3. No changes needed for credential stuffing, attack zoo, power-law, NAB sections — those numbers held.
  4. PAIN_POINTS.md — no substrate-dependent claims; unchanged.

Phase 2 — Substrate Fill-in (same day, 2026-05-15)

After the validation sweep above, the Architect declared log_phi_pi_fibonacci THE base algorithm of all of OMC and asked for a comprehensive audit + migration of every site that uses or should use the substrate. Five Bucket-B findings (sites that bypassed the substrate via Python math.log10/math.log round-trips or hardcoded Fibonacci arrays) plus one deprecated alias removal.

Migrations applied

ID File / location Old New Type
B1 examples/lib/harmonic_anomaly.omc _bucket_log py_call(math, "log10", v) * 50 then fold log_phi_pi_fibonacci(v) * 50 then fold substrate-tempo
B2 examples/lib/harmonic_anomaly.omc score -py_call(math, "log", p) log_phi_pi_fibonacci(1.0/p) (monotonic) substrate-routed
B3 examples/lib/harmonic_clustering.omc _bucket_log py_call(math, "log10", v) log_phi_pi_fibonacci(v) / log_phi_pi_fibonacci(10.0) (decade-rescale: substrate-routed computation, log10-equivalent output) substrate-routed
B4 omnimcode-core/src/interpreter.rs harmonic_split hardcoded [1,2,3,5,8,...,610] 14-entry array phi_pi_fib::largest_attractor_at_most(remaining) — new helper, 40-entry table reaches 63M substrate-canonical
B5 examples/datascience/multidim_anomaly.omc and anomaly_detection.omc inline copies of B1/B2 patterns mirrored to substrate-tempo substrate-tempo
D2 omnimcode-core/src/phi_pi_fib.rs deprecated log_phi(n) alias DELETED — new code uses log_phi_pi_fibonacci DEPRECATION removed

New helper added: phi_pi_fib::largest_attractor_at_most(value: i64) -> i64 — sign-preserving, returns the greatest attractor ≤ |value|. Replaces ad-hoc reverse linear scans over hardcoded Fibonacci arrays. Two new unit tests pin its behavior (basics + large-magnitude range that the old 16-entry table couldn't reach).

Architectural decision: substrate purity over benchmark numbers

The Architect was presented with three resolution options for B1 (the bucket function in harmonic_anomaly) after observing that substrate-tempo bucketing measurably hurts empirical results on real heavy-tailed data:

Option Substrate-routed Empirical impact
Revert B1 to log10 (via OMC's native log builtin) NO Restores all numbers
Decade-rescale (window-dressing route) yes (mathematically equivalent to log10) Restores all numbers
Keep current substrate-tempo (CHOSEN) YES, fully K=500 NSL-KDD: 365 → 302 (−63)

The Architect chose substrate purity. The substrate now governs magnitude-slicing semantics throughout OMC, even where its grain (~1.5 buckets per base-10 decade) produces empirically worse anomaly recall than base-10 decades would.

Validation: empirical impact of the fill-in

Engine parity and infrastructure tests all held:

  • 44/45 functional examples byte-identical TW vs VM (the diverger is benchmarks.omc — timing-only, same as before)
  • 149/149 Rust unit tests pass (was 148; one removed via D2, two added for largest_attractor_at_most and log_phi_pi_fibonacci monotonicity)
  • 18/18 OMC harmonic-lib tests pass (after decade-rescale fix to harmonic_clustering)
  • NAB realKnownCause: 7/19 covered, NEUTRAL
  • Attack zoo: 30/30, NEUTRAL

Anomaly benchmarks (the substrate-sensitive sites):

Benchmark Phase-1 substrate refactor Phase-2 substrate fill-in Verdict
Credential stuffing K=10 10/10 10/10 NEUTRAL
Credential stuffing K=25 25/25 24/25 UNIMPROVEMENT (−1)
Credential stuffing K=50 50/50 49/50 UNIMPROVEMENT (−1)
Credential stuffing K=100 50/100 50/100 NEUTRAL
Power-law K=5 (alert budget) 4/5 1/5 UNIMPROVEMENT (−3)
Power-law K=10 5/10 3/10 UNIMPROVEMENT (−2)
Power-law K=20 5/20 7/20 IMPROVEMENT (+2)
Power-law K=30 5/30 12/30 IMPROVEMENT (+7)
NSL-KDD K=10 7/10 6/10 UNIMPROVEMENT (−1)
NSL-KDD K=50 42/50 43/50 IMPROVEMENT (+1)
NSL-KDD K=100 78/100 78/100 NEUTRAL
NSL-KDD K=500 365/500 302/500 UNIMPROVEMENT (−63)

The pattern: substrate-tempo bucketing trades low-K precision for high-K-on-spread-data. Where the old log10-bucketing concentrated big spikes into a single attractor (e.g. all DoS-attack byte counts landing in bucket-377), substrate-tempo spreads them across multiple attractors (377/610/987/...), which weakens "biggest spike wins" alerting but improves diversity at high K. Real-world heavy-tailed data (NSL-KDD's volumetric DoS) is the worst case for this trade — those attacks were structurally the same and benefited from concentration.

What's groundbreaking, what's an unimprovement

GROUNDBREAKING — Phase 2:

  • The substrate is now THE base algorithm everywhere. Five sites that bypassed it via Python round-trips or hardcoded arrays are now routed through phi_pi_fib::*. Architectural completeness over benchmark numbers.
  • New helper largest_attractor_at_most retires the last hardcoded Fibonacci array inside core (harmonic_split was the holdout).

UNIMPROVEMENT — Phase 2:

  • NSL-KDD K=500: 365 → 302. We lose the "harmonic beats IF on volumetric data at K=500" claim from Phase 1. This was the most-cited Phase-1 win and it's been deliberately traded for substrate consistency.
  • Power-law K=5 (alert budget): 4/5 → 1/5. The headline "harmonic surfaces structural anomalies before magnitude outliers" claim weakens — at top-5 we now mostly miss.
  • Credential stuffing K=25/K=50: 25→24, 50→49. Small slippage on the synthetic benchmark that was a Phase-1 anchor.

DEPRECATION — Phase 2:

  • phi_pi_fib::log_phi deleted. New code uses log_phi_pi_fibonacci. The substrate naming convention is now consistent.

Doc updates needed

  1. README's "Where harmonic detection actually wins" table — Phase-2 numbers replace Phase-1 numbers. The K=500 win flips back to a tie (302 vs 351 → IF leads). The K=5 power-law win weakens.
  2. docs/anomaly_detection.md — Result 5 NSL-KDD K=500 narrative needs to drop the "harmonic now beats IF" framing; the K=500 crossover from Phase 1 is gone.
  3. SUBSTRATE_CHANGES.md (this doc) — captures the Phase-2 trade in full so future readers know the choice was deliberate.

What's NOT in scope of this fill-in (deferred)

  • D3: HBit harmony substrate-routing. hbit.rs:43 uses Euclidean 1.0/(1.0+diff); the dual-band α/β/harmony channel doesn't yet speak substrate units. The Architect flagged this has "bigger implications" and deferred to its own session. Next on the queue.
  • LLM evolution experiments (Experiments 0-9). Developed ON the new substrate; no migration needed but worth a separate audit pass to identify which findings would've failed under the old substrate (substrate-aware vs substrate-dependent classification).