Merge pull request #508 from AdaWorldAPI/claude/chaoda-ensemble-finding-v1

docs(genetics): D-GEN-CHAODA-ENSEMBLE increment 1 RUN — ensemble clears synthetic bar (AUC 0.62 -> 0.99, ndarray #220)

Author: AdaWorldAPI

.claude/plans/genetics-probes-v1.md

@@ -21,15 +21,20 @@
 
 | Phase | Probe | Cost | Status | Gates |
 |---|---|---|---|---|
-| **P0** | PROBE-CHAODA-1000G | ~3 days (after D-GEN-1+2) | ⚠ **spike RUN — AUC 0.624, BELOW bar** (ndarray #219) | The "CHAODA-as-novelty-detector" line of the entire plan |
+| **P0a** | D-GEN-CHAODA-ENSEMBLE (synthetic) | ~done | ✅ **ensemble RUN — AUC 0.991, CLEARS bar** (ndarray #220) | The CHAODA kernel's isolation capability |
+| **P0b** | PROBE-CHAODA-1000G (genomic) | ~3 days (after D-GEN-1+2) | ⏳ unblocked at kernel level; gated on real corpora | The "CHAODA-as-novelty-detector" line of the entire plan |
 | **P1** | PROBE-KRAS-COUNTERFACTUAL-DET | ~2 days (included in D-GEN-7) | queued | D-GEN-7 flagship dynamics-axis claim |
 | **P2** | PROBE-CAM-PQ-VS-BLAST | ~1 week | queued | D-GEN-3 sequence-fingerprint claim |
 
-**⚠ Blocker surfaced 2026-06-16:** the P0 spike (ndarray #219) shows the shipped
-single-method leaf-LFD `anomaly_scores` reaches only AUC 0.624 on ideal synthetic
-data. Porting the multi-method CHAODA ensemble (Ishaq et al. 2021) is now a
-**prerequisite for PROBE-CHAODA-1000G**, ahead of any genomic-fixture work. See
-the ⚠ FINDING under PROBE-CHAODA-1000G below.
+**⚠→✅ Blocker surfaced AND resolved at kernel level 2026-06-16:** the P0 spike
+(ndarray #219) showed the shipped single-method leaf-LFD `anomaly_scores` reaching
+only AUC 0.624 on ideal synthetic data. The multi-method CHAODA ensemble has now
+been built (ndarray #220, `ClamTree::ensemble_anomaly_scores`) and measured at
+**AUC 0.991** on the same synthetic fixture — clearing the 0.85 bar with a +0.367
+lift. **This resolves the kernel-level blocker** (the ensemble *approach* captures
+isolation where single-LFD did not). It does NOT yet prove genomic novelty
+detection: `PROBE-CHAODA-1000G` on real corpora remains gated on D-GEN-1 + D-GEN-2.
+See the FINDING under PROBE-CHAODA-1000G below.
 
 **Critical-path note:** PROBE-CHAODA-1000G is the single highest-leverage probe.
 If it fails (AUC < 0.85 on novel-variant detection against ClinVar Pathogenic
@@ -54,6 +59,30 @@ probe is a regression gate, not a discovery gate).
 > singletons from common population variants at ROC-AUC ≥ 0.85 on a
 > held-out test fold drawn from 1000-Genomes Phase 3 + ClinVar.
 
+### ✅ FOLLOW-UP (ensemble RUN 2026-06-16) — multi-method ensemble CLEARS the bar (synthetic)
+
+The blocker below has been resolved at the kernel level. The multi-method CHAODA
+ensemble is built (ndarray PR #220, `ClamTree::ensemble_anomaly_scores`) and
+measured on the *same* synthetic fixture:
+
+| signal | ROC-AUC |
+|---|---|
+| single-method leaf-LFD (baseline) | 0.6240 |
+| **multi-method ensemble** | **0.9906** |
+| lift | **+0.3667** |
+
+The dominant signal is the **parent-child path-minority ratio** — walking a leaf
+up to the root, the minimum `child/parent` cardinality ratio is tiny for a point
+that split off as a minority (an isolated outlier) and moderate for a dense-cluster
+member that always stayed in the majority. This is *immune to the leaf-fragmentation*
+that defeated the naïve first attempt (raw leaf cardinality + degree + component
+size → AUC 0.621, no lift). Averaged with connected-component cardinality.
+
+**This proves the ensemble approach, on synthetic data only.** It does NOT prove
+genomic novelty detection — `PROBE-CHAODA-1000G` on real corpora is still gated on
+D-GEN-1 + D-GEN-2. Random-walk stationary distribution remains deferred to a later
+ensemble increment.
+
 ### ⚠ FINDING (spike substitute RUN 2026-06-16) — single-method LFD is BELOW the bar
 
 The 1-day spike substitute (see §Cost below) has been **run** against the
@@ -353,21 +382,26 @@ the pattern match is sound, the single shipped signal is not yet sufficient,
 and the honest path is "port the ensemble, then re-run the spike, then build
 the fixture." A new candidate deliverable falls out of this:
 
-> **D-GEN-CHAODA-ENSEMBLE (new, prerequisite to PROBE-CHAODA-1000G):** add the
+> **D-GEN-CHAODA-ENSEMBLE (prerequisite to PROBE-CHAODA-1000G):** add the
 > multi-method CHAODA anomaly ensemble to `ndarray::hpc::clam` as a **new
-> scoring entry point** (e.g. `ensemble_anomaly_scores(...) -> Vec<AnomalyScore>`,
-> name TBD at implementation), combining the graph-based signals of Ishaq et
-> al. 2021. The existing single-method `anomaly_scores` is **kept unchanged as
-> the documented baseline / regression** (the ndarray #219 spike's `auc < 0.85`
-> tripwire stays green on it). **`PROBE-CHAODA-1000G` Step 3 must call the new
-> ensemble entry point, not `anomaly_scores`** — that wiring is part of this
+> scoring entry point**, combining the graph-based signals of Ishaq et al.
+> 2021. The existing single-method `anomaly_scores` is **kept unchanged as the
+> documented baseline / regression**. **`PROBE-CHAODA-1000G` Step 3 must call the
+> new ensemble entry point, not `anomaly_scores`** — that wiring is part of this
 > deliverable, otherwise the genomic probe would re-measure the known-bad
-> AUC-0.624 path. Re-run the ndarray #219 spike against the ensemble; gate at
-> AUC ≥ 0.85 on the synthetic mixture *before* genomic fixtures are built.
-> Lift: ~1 week (the graph-construction primitives — cluster cardinality,
-> neighbourhood, random-walk — are mostly present in the CLAM tree already;
-> the ensemble combination + per-method scoring + the probe-API wiring is the
-> new code).
+> AUC-0.624 path.
+>
+> **✅ INCREMENT 1 DONE 2026-06-16 — ndarray PR #220.** Shipped
+> `ClamTree::ensemble_anomaly_scores` = parent-child path-minority ratio ⊕
+> connected-component cardinality. Re-ran the ndarray #219 synthetic fixture:
+> **ensemble AUC 0.9906 vs single-LFD 0.6240 (+0.367), clears the 0.85 gate.**
+> Deterministic; built from shipped tree fields + public `dist()`; no new tree
+> state. **Remaining for a later increment:** (a) random-walk stationary
+> distribution method (deferred — needs power-iteration on the cluster graph);
+> (b) the actual `PROBE-CHAODA-1000G` Step 3 wiring lands with D-GEN-1+2 when the
+> VCF→feature-vector pipeline exists. Increment-1 lift came in at ~half-day, well
+> under the ~1-week estimate, because the path-minority signal needed only a
+> parent-map walk — no random-walk solver.
 
 **PROBE-CHAODA-1000G fires first, even though chronologically D-GEN-1..2 must
 ship first.** That ordering is a substrate-economic decision (cheaper to

docs/GENETIC_RESEARCH_VIA_STACK.md

@@ -71,7 +71,7 @@ impl ClamTree {
 
 **The composition:** build a CLAM tree on your per-variant feature vectors; CHAODA scores every variant against the local manifold's intrinsic dimensionality. A novel variant in a region of high LFD lights up as `AnomalyScore { score → 1.0, awareness → AwarenessState::Noise }` (the `score ≥ 0.75` quartile per `clam.rs:1556`) because its position differs from the population's local manifold — *without you having to train a classifier or annotate a truth set first*. This is *unsupervised* outlier detection on the same tree your range queries walk.
 
-> **⚠ MEASURED CAVEAT (2026-06-16, ndarray PR #219):** the *shipped* `anomaly_scores` implements **only the single-method leaf-LFD signal**, not the full multi-method CHAODA ensemble of Ishaq et al. 2021. A spike on ideal synthetic data (clean Gaussian clusters + far outliers) measured **ROC-AUC = 0.624** — well below the ≥ 0.85 bar a novelty detector needs. Leaf LFD captures *intra-leaf* geometry complexity, not *inter-leaf* isolation, so isolated outliers and dense-cluster points end up in the same score band. **As shipped today, this composition is NOT a working novel-variant detector.** Realising the claim requires porting the multi-method CHAODA ensemble (relative/component cardinality, graph neighbourhood, random-walk stationary distribution, vertex degree) — see `PROBE-CHAODA-1000G` in `.claude/plans/genetics-probes-v1.md`. The pattern match is real; the *single shipped signal* is not yet sufficient.
+> **⚠→✅ MEASURED CAVEAT (2026-06-16):** the *original* `anomaly_scores` implements **only the single-method leaf-LFD signal**. A spike (ndarray PR #219) on ideal synthetic data measured **ROC-AUC = 0.624** — below the ≥ 0.85 bar — because leaf LFD captures *intra-leaf* geometry complexity, not *inter-leaf* isolation. **The multi-method ensemble has since been built** (ndarray PR #220, `ClamTree::ensemble_anomaly_scores`: parent-child path-minority ⊕ connected-component cardinality) and measured at **ROC-AUC = 0.991** on the same fixture — clearing the bar. So the *kernel* now does isolation-aware novelty detection; use `ensemble_anomaly_scores`, not the single-LFD `anomaly_scores`, for this composition. **Still gated:** this is synthetic-only proof. Genomic novelty detection (`PROBE-CHAODA-1000G` on 1000-Genomes + ClinVar) remains unproven until the VCF→feature-vector pipeline (plan D-GEN-1+2) exists. The pattern match is real and the kernel capability is now demonstrated; the genomic claim is not yet measured.
 
 ### 1.5 minimap2 minimizers ↔ bgz17 11/17 X-Trans stride