Skip to content

Commit 6cf6582

Browse files
Empirical bench: substrate-routed typo lookup vs full closest_name scan
Adds `typo_bench_substrate_vs_full` as a cargo test (release build, --nocapture for output). Measures lookup time at N = 10/100/1000/10000 symbol-table sizes, 1000 queries each. Results validate the ~10x claim in docs/heal_pass.md: | N | substrate_µs | full_µs | speedup | |-------:|-------------:|---------:|--------:| | 10 | 3.22 | 3.26 | 1.01× | | 100 | 3.06 | 34.07 | 11.14× | | 1000 | 32.64 | 352.64 | 10.80× | | 10000 | 313.22 | 3365.22 | 10.74× | Neutral (no overhead) at N=10 — bucketed hit rate is 0 because the small table fits inside the fallback's working set. Above N=100 the bucketed phase finds the match 100% without falling back. Ratio holds flat across two orders of magnitude in N because closest_name is linear in N while the substrate path stays bounded by bucket size (~ N / 32 per probe, plus the prefer-set scan). Updated docs/heal_pass.md with the empirical table. Run: cargo test --release -p omnimcode-core typo_bench -- --nocapture Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 8469bae commit 6cf6582

2 files changed

Lines changed: 88 additions & 0 deletions

File tree

docs/heal_pass.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,19 @@ The substrate-routed implementation uses a two-phase scan:
2828

2929
Falls back to full `closest_name` if both phases miss — preserves correctness.
3030

31+
### Empirical timings
32+
33+
`cargo test --release -p omnimcode-core typo_bench -- --nocapture` runs 1000 typo queries at each symbol-table size:
34+
35+
| N | substrate_µs | full_µs | speedup | bucketed_hit |
36+
|-------:|-------------:|---------:|--------:|-------------:|
37+
| 10 | 3.22 | 3.26 | 1.01× | 0 / 1000 |
38+
| 100 | 3.06 | 34.07 | 11.14× | 1000 / 1000 |
39+
| 1000 | 32.64 | 352.64 | 10.80× | 1000 / 1000 |
40+
| 10000 | 313.22 | 3365.22 | 10.74× | 1000 / 1000 |
41+
42+
Neutral at N=10 (no benefit, no overhead — bucketed hit rate is 0 because the small table fits inside the fallback's working set). At N≥100 the bucketed phase finds the match 100% of the time without falling back, delivering the 10–11× speedup. The ratio holds essentially flat across 2 orders of magnitude in N because closest_name is linear in N while the substrate path stays bounded by bucket size (≤ N / 32 per probe, plus the prefer-set scan).
43+
3144
```rust
3245
// closest_name_substrate() in src/interpreter.rs
3346
// Phase 1: full O(|prefer|) scan of user fns (correctness)

omnimcode-core/src/interpreter.rs

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7854,6 +7854,81 @@ mod tests {
78547854
// Basic tests would go here
78557855
}
78567856

7857+
/// Empirical comparison: substrate-routed typo lookup vs full-scan
7858+
/// closest_name across symbol-table sizes 10/100/1000/10000. Each
7859+
/// size runs 1000 typo queries; we report mean lookup time and the
7860+
/// substrate/full ratio.
7861+
///
7862+
/// Run with: cargo test --release -p omnimcode-core typo_bench -- --nocapture
7863+
#[test]
7864+
fn typo_bench_substrate_vs_full() {
7865+
use std::time::Instant;
7866+
7867+
let sizes = [10usize, 100, 1000, 10000];
7868+
let queries_per_size = 1000usize;
7869+
7870+
println!();
7871+
println!("# Typo lookup: substrate-bucketed vs full-scan");
7872+
println!("# {} queries per size, ed≤2", queries_per_size);
7873+
println!();
7874+
println!("{:>8} {:>14} {:>14} {:>10} {:>12}",
7875+
"N", "substrate_µs", "full_µs", "ratio", "bucketed_hit");
7876+
7877+
for &n in &sizes {
7878+
// Synthesize N defined names of the shape "fn_NNNN" — enough
7879+
// structural diversity that the bucketed index distributes
7880+
// reasonably (substrate_hash_name is deterministic per str).
7881+
let names: Vec<String> = (0..n).map(|i| format!("fn_{:05}", i)).collect();
7882+
let defined: HashSet<String> = names.iter().cloned().collect();
7883+
7884+
// Queries: deterministic typos — drop the last char of every
7885+
// 7th name. Each is edit-distance 1 from a real name, so
7886+
// closest_name SHOULD find a match.
7887+
let queries: Vec<String> = (0..queries_per_size).map(|i| {
7888+
let target_idx = (i * 7919) % n;
7889+
let mut q = names[target_idx].clone();
7890+
q.pop();
7891+
q
7892+
}).collect();
7893+
7894+
// Populate the thread-local substrate index for the bucketed path.
7895+
let bucketed = build_substrate_name_index(&defined);
7896+
HEAL_SUBSTRATE_INDEX.with(|idx| *idx.borrow_mut() = bucketed);
7897+
HEAL_CLASS_COUNTS.with(|c| *c.borrow_mut() = HealClassCounts::new());
7898+
7899+
// Substrate path: bucketed pre-filter + fallback.
7900+
let t0 = Instant::now();
7901+
let mut sub_hits = 0;
7902+
for q in &queries {
7903+
if closest_name_substrate(q, &defined, 2, None).is_some() {
7904+
sub_hits += 1;
7905+
}
7906+
}
7907+
let sub_elapsed = t0.elapsed();
7908+
let sub_us = sub_elapsed.as_micros() as f64 / queries_per_size as f64;
7909+
7910+
// Full path: pure closest_name (linear scan).
7911+
let t0 = Instant::now();
7912+
let mut full_hits = 0;
7913+
for q in &queries {
7914+
if closest_name(q, &defined, 2, None).is_some() {
7915+
full_hits += 1;
7916+
}
7917+
}
7918+
let full_elapsed = t0.elapsed();
7919+
let full_us = full_elapsed.as_micros() as f64 / queries_per_size as f64;
7920+
7921+
assert_eq!(sub_hits, full_hits, "hit counts diverged at N={}", n);
7922+
7923+
let bucketed_hit = HEAL_CLASS_COUNTS.with(|c| c.borrow().typo_substrate_hit);
7924+
let ratio = full_us / sub_us.max(0.001);
7925+
7926+
println!("{:>8} {:>14.3} {:>14.3} {:>9.2}x {:>10}/{:<4}",
7927+
n, sub_us, full_us, ratio, bucketed_hit, queries_per_size);
7928+
}
7929+
println!();
7930+
}
7931+
78577932
fn run(source: &str) -> Result<Value, String> {
78587933
use crate::parser::Parser;
78597934
let mut parser = Parser::new(source);

0 commit comments

Comments
 (0)