Merge pull request #494 from AdaWorldAPI/claude/wonderful-hawking-lodtql

Entropy-ladder plateau: EdgeCodecFlavor selector + bgz17 OOB fix + SPO rung-ladder plan

Author: AdaWorldAPI

.claude/board/AGENT_LOG.md

@@ -1,3 +1,15 @@
+## 2026-06-14 — edge-codec flavors: all three implemented, class-selectable, measured (ICC/Pearson/Cronbach/Spearman)
+
+**Main thread (Opus 4.8 1M), branch `claude/wonderful-hawking-lodtql` (ndarray + lance-graph).** Operator (relaying a second session's 32×4-bit edge proposal + the key insight *"you could literally combine deterministic↔residue"*): *"implement all and allow the class>schema inheritance mapping to choose which flavor … measure all options and validate/invalidate ICC Pearson Cronbach alpha … using the JC crate and ndarray/crates/hpc/pillars."* Resolved the second session's false dichotomy: the deterministic part (nearest-centroid palette index, recomputed via AMX `matmul_i8_to_i32`) is the EdgeBlock byte unchanged; the residue is a value-slab 4-bit plane — so all flavors are *interpretations* of the locked 16-byte block, none changes `NODE_ROW_STRIDE`.
+
+**Shipped:**
+- **ndarray** (`d3b608f`): `hpc::reliability` (Pearson/Spearman/Cronbach α/ICC(2,1) + `FidelityReport`; the JC-consumable measurement layer — jc/pillar had only private Spearman, missing the other three) and `hpc::edge_codec` (Codebook k-means, `CoarseResidueCodec`, `ProductQuantizer`, `reconstruct_coarse`). Harness `examples/edge_codec_compare`. 16 unit + 9 doctests; lib clippy `-D warnings` clean.
+- **lance-graph contract**: `EdgeCodecFlavor` enum (`CoarseOnly`/`CoarseResidue`/`Pq32x4`) + defaulted `ClassView::edge_codec_flavor` selector (non-breaking). +3 tests, 609 lib green, clippy clean. LATEST_STATE contract inventory updated (same commit).
+
+**Measured (AMX host):** CoarseResidue dominates agreement (ICC 0.97–0.99, ρ 0.98, α 0.99 across blob+continuous); Pq32x4 preserves rank (ρ 0.60–0.67) but not absolute distance (ICC 0.11–0.29 — the Pearson-vs-ICC contrast working as designed); CoarseOnly collapses on continuous (ICC 0.003). AMX assign 100% vs scalar, 24–28 GMAC/s.
+
+**Deferred/flagged:** turbovec PQ4 *throughput* path blocked on #493 P2 (turbovec `ndarray-simd` feature removed in `7fa217c`; polyfill fns gone). Fidelity is kernel-independent, so throughput-only follow-up. Per-class flavor STORAGE (override `edge_codec_flavor` from a class config) = follow-up in `lance-graph-ontology`. Also fixed bgz17 SIMD gather OOB (P1 from #493, commit `6d48ced`).
+
 ## 2026-06-13 — turbovec ⇄ ndarray integration: fork-wired + ndarray::simd polyfill GEMM + measured AMX-vs-LUT
 
 **Main thread (Opus 4.8 1M) + 1 Opus general-purpose agent (bgz-tensor synergy map).** User: "create a crate in lance-graph for turbovec and check synergies; route SIMD through ndarray::simd (simd.rs→simd_amx/avx512/ops/soa); the polyfill does the work, ndarray ships AMX via byte-asm dispatch; pin rust 1.95." Cross-repo, branch `claude/wonderful-hawking-lodtql` in all three repos.

.claude/board/INTEGRATION_PLANS.md

@@ -1,3 +1,19 @@
+## 2026-06-14 — entropy-ladder-spo-rung-v1 (Staunen↔Wisdom entropy coordinate unifies SPO rungs + NARS reliability + edge-codec; R1 shipped, R2–R6 roadmap)
+
+**Status:** PLATEAU — R1 (foundation) SHIPPED; R2–R6 planned (probe-gated where marked). **Plan file:** `.claude/plans/entropy-ladder-spo-rung-v1.md`.
+**Owns:** 7 deliverables D-EL-1..6 + D-EL-COCA.
+- D-EL-1: entropy-ladder foundation — `ndarray::hpc::{reliability, edge_codec, entropy_ladder}` + `EdgeCodecFlavor` selector + bgz17 OOB fix (SHIPPED; ρ=−0.78 reliability proxy; ICC/Pearson/Cronbach/Spearman; edge-codec fidelity table)
+- D-EL-2: `entropy_class` → CausalEdge64 spare bits [63:61] (version-gated, field-isolation tests) — next
+- D-EL-3: CAM-PQ AMX centroid assignment (assignment-as-GEMM + 2×2/4×4 tiled grid) — queued
+- D-EL-4: HHTL+helix basin attraction (probe-gated, +15% recall vs HHTL-alone)
+- D-EL-5: Markov SPO rung-ladder → episodic context/basins/supporting edges (probe-gated)
+- D-EL-6: energy axis / particle↔wave (gated on Mailbox-SoA map)
+- D-EL-COCA: superposition 2/3 pruning (needs cluster-identity layer; I-VSA-IDENTITIES design-gate)
+**Key decisions:** read the 3×palette256 SPO already in CausalEdge64 (NO re-quant); Pearl 2³ mask = the 8 SPO iterations; flavors/classes are interpretation not layout (no NODE_ROW_STRIDE / no ENVELOPE_LAYOUT_VERSION bump); COCA pruning must bundle identities not PQ codes (I-VSA-IDENTITIES).
+**Repos:** ndarray (`d3b608f`, `83be7c3`) + lance-graph (`6d48ced`, `920671d`), branch `claude/wonderful-hawking-lodtql`. PRs: this plateau pair.
+
+---
+
 ## 2026-06-09 — polyglot-container-query-membrane-v1 (Node Container answers Cypher + SurrealQL AST + DataFusion UDF over one HHTL address space; mailbox = a normal cold path)
 
 **Status:** RESEARCH MAP + PLAN. Grounded by two parallel sweeps (lance-graph + surrealdb fork) with main-thread spot-verification; one agent claim caught false (SoaEnvelope has ZERO real impls — identity N3 stands live). **Plan file:** `.claude/plans/polyglot-container-query-membrane-v1.md`.

.claude/board/LATEST_STATE.md

@@ -512,3 +512,13 @@ PR sequence: #360 → #361 → post-#360 substrate-sweep (this PR).
 > **2026-06-01 — PR-in-flight (autoattended)** (D-EW64-3/4): `lance_graph_contract::episodic_edges` gains `EpisodicEdges64::{coldest, contains, promote_into}` + the `DemotionSink` trait. `coldest()` = the eviction victim (symmetric to `strongest()`); `contains()` = family-discriminating membership; `promote_into(e, sink)` = `promote` routing the evicted (coldest) edge to a `DemotionSink` — the hot→cold connectome exit. `DemotionSink` impls (surreal/LanceDB-LIVE "wingman", `E-SUBSTRATE-IS-THE-SCHEDULER`) are deferred + GATED on OQ-11.6. Zero-dep; contract lib 545 green; default clippy clean; `episodic_edges.rs` pedantic+nursery clean.
 
 > **2026-06-01 — Shipped (autoattended, 5-agent council)** (D-ATOM-4/RawEdge): `contract::counterfactual` wired into `lib.rs` (was orphaned); `RawEdge(i8)` mantissa-only **structural** impl of `EpisodicEdge` (`size_of==1` — a u64 newtype could read plasticity 50–52); `deposit_counterfactual` v2 filled (−6 on split). Closes the counterfactual seam (NOT the prefetch loop). +3 latent scaffold fixes. 550 contract lib green, clippy clean. The council REFUTED the prior "compose `Heel.plasticity` × MRU" ① resolution (`E-BASIN-NOT-EDGE-PLASTICITY`): coarse strength = MRU slot-order (shipped); per-edge Hebbian = per-plane `PlasticityState` (gated).
+
+## 2026-06-14 — Append: `EdgeCodecFlavor` selector + ndarray edge-codec/reliability layer (branch `claude/wonderful-hawking-lodtql`)
+
+(Per APPEND-ONLY rule: new top-of-inventory entry. Branch work; records the contract type so a new session does not re-derive it.)
+
+### Current Contract Inventory — new entry
+
+- **`canonical_node::EdgeCodecFlavor`** (NEW; re-exported from `lib.rs`). Per-class selector for how a node's 16-byte `EdgeBlock` (+ optional value-slab residue) is *interpreted* — `CoarseOnly` (1 B palette index, the canon zero-fallback default), `CoarseResidue` (1 + ⌈D/2⌉ B, value-slab signed-4-bit residue), `Pq32x4` (16 B = 32×4-bit product code, the turbovec PQ model). `bytes_per_vector(dim)` + `is_layout_preserving()` (always `true`). **Iron invariant:** the flavor is interpretation, NOT layout — every variant leaves `NODE_ROW_STRIDE = 512` untouched, so adoption needs no `ENVELOPE_LAYOUT_VERSION` bump (canon "registry-resolved via `classid → ClassView`"). Default `CoarseOnly` matches the all-zero bootstrap. Selection surface: new defaulted `ClassView::edge_codec_flavor(&self, ClassId) -> EdgeCodecFlavor` (non-breaking — `RegistryClassView` inherits the default; per-class override is the follow-up wiring in `lance-graph-ontology`). +3 tests; contract lib 609 green; clippy `-D warnings` clean.
+- **Encode/measure kernels live in `ndarray` (the hardware layer), not the contract:** `ndarray::hpc::edge_codec` (Codebook k-means, `CoarseResidueCodec`, `ProductQuantizer`, `reconstruct_coarse`) + `ndarray::hpc::reliability` (Pearson r, Spearman ρ, Cronbach α, ICC(2,1), `FidelityReport`). Harness `examples/edge_codec_compare` measures all flavors × {blob, continuous} regimes. **Measured:** CoarseResidue dominates agreement (ICC 0.97–0.99, ρ 0.98, α 0.99); Pq32x4 keeps rank (ρ 0.60–0.67) but not absolute distance (ICC 0.11–0.29); CoarseOnly collapses on continuous (ICC 0.003); AMX `matmul_i8_to_i32` assign 100% vs scalar, 24–28 GMAC/s. ndarray commit `d3b608f`.
+- **Deferred (flagged):** turbovec PQ4 *throughput* path (the FastScan nibble-LUT for the Pq32x4 flavor) blocked on the **#493 P2** build break — `lance-graph-turbovec` requests the `ndarray-simd` turbovec feature that was REMOVED (turbovec commit `7fa217c`); the polyfill fns are gone. turbovec's API is end-to-end (owns encode+scan), so it is a *PQ4 flavor*, not a residue-nibble-scan primitive. Fidelity (what ICC/Pearson/α measure) is independent of the fast kernel, so this is throughput-only follow-up.

.claude/board/STATUS_BOARD.md

@@ -1,3 +1,19 @@
+## entropy-ladder-spo-rung-v1 — Staunen↔Wisdom entropy coordinate unifies SPO rungs + NARS reliability (R1 shipped; R2–R6 roadmap)
+
+Plan path: `.claude/plans/entropy-ladder-spo-rung-v1.md`. Foundation: `ndarray::hpc::{reliability, edge_codec, entropy_ladder}`. Selector: `lance-graph-contract::EdgeCodecFlavor`.
+
+| D-id | Title | Crate(s) / repo | Risk | Status | PR / Evidence |
+|---|---|---|---|---|---|
+| D-EL-1 | Entropy-ladder foundation (reliability + edge_codec + entropy_ladder + EdgeCodecFlavor + bgz17 fix) | ndarray + lance-graph-contract + bgz17 | LOW | **In PR** | `d3b608f`,`83be7c3`,`920671d`,`6d48ced`; ρ=−0.78; ICC 0.97–0.99 |
+| D-EL-2 | `entropy_class` → CausalEdge64 spare bits [63:61] | `causal-edge` | MED | **Queued** | version-gated + field-isolation (I-LEGACY-API-FEATURE-GATED) |
+| D-EL-3 | CAM-PQ AMX centroid assignment (GEMM + 2×2/4×4 grid) | `ndarray` | MED | **Queued** | bit-exact + GMAC/s probe |
+| D-EL-4 | HHTL+helix basin attraction | `lance-graph` + `helix` | MED | **Probe queued** | +15% recall vs HHTL-alone gate |
+| D-EL-5 | Markov SPO rung-ladder → episodic context | `deepnsm` / `lance-graph` | MED | **Probe queued** | prune-without-recall-loss gate |
+| D-EL-6 | Energy axis / particle↔wave | `lance-graph` MailboxSoA | MED | **Blocked** | gated on Mailbox-SoA map |
+| D-EL-COCA | Superposition 2/3 pruning (cluster-identity layer) | `deepnsm` | HIGH | **Design** | I-VSA-IDENTITIES design-gate |
+
+---
+
 ## singleton-to-snapshot-nudge-v1 — every shared-mutable singleton → per-owner MailboxSoA + Arc-swap snapshot (7 deliverables; codebooks left as-is)
 
 Plan path: `.claude/plans/singleton-to-snapshot-nudge-v1.md`. Companions: `bindspace-singleton-to-mailbox-soa-v1` (BindSpace dissolution), `cycle-coherent-soa-snapshot-v1` (snapshot mechanism). Debt: TD-UNBUNDLE-FROM-1.

.claude/plans/entropy-ladder-spo-rung-v1.md

@@ -0,0 +1,105 @@
+# Entropy-Ladder SPO Rung Decomposition — v1
+
+> **Status:** PLATEAU — R1 (foundation) SHIPPED; R2–R6 planned (probe-gated where marked).
+> **Confidence:** R1 measured (ρ=−0.78 reliability-proxy); R2/R3 unblocked; R4/R5 probe-first; R6 gated on the Mailbox-SoA map.
+> **Date:** 2026-06-14
+> **Branch:** `claude/wonderful-hawking-lodtql` (ndarray + lance-graph)
+> **Owner:** main thread (Opus)
+
+## The Click — one coordinate unifies the stack
+
+**A fact's position on the cognitive ladder IS an entropy level.** The same
+scalar orders SPO triples, NARS truth, and the linguistic ladder:
+
+```
+  Staunen  ── high entropy ── raw stimulus, not yet crystallized ── SYNTAX
+     │                                                                 │
+  (semantics sits between)                                       (meaning)
+     │                                                                 │
+  Wisdom   ── low  entropy ── crystalline knowledge / settled fact ─ PRAGMATICS
+```
+
+`entropy = 1 − c·|2f − 1|` over a `CausalEdge64`'s NARS `(f, c)` — one minus the
+decisiveness of the NARS expectation. Validated as a **reliability proxy**:
+ρ(entropy, empirical prediction accuracy) = **−0.78** over a synthetic NARS
+population (the more crystalline the edge, the more reliably its belief matches
+fresh reality), grounded via `ndarray::hpc::reliability`.
+
+## Architecture decisions (locked this session)
+
+1. **No re-quantization.** The SPO rungs read the **3×palette-256 indices already
+   inside `CausalEdge64`** (`s_idx`/`p_idx`/`o_idx`) — "exact enough" (operator).
+   `cam_pq` is NOT used to re-encode SPO for the ladder.
+2. **Pearl 2³ mask = the 8 SPO iterations.** The existing causal mask at
+   `CausalEdge64` [42:40] enumerates the 8 `(S,P,O)` subsets; `decompose_spo`
+   zeroes inactive components and emits an HHTL-routable `basin_key`.
+3. **Flavors / classes are INTERPRETATION, not layout.** No `NODE_ROW_STRIDE`
+   change, no `ENVELOPE_LAYOUT_VERSION` bump (canon "registry-resolved via
+   `classid → ClassView`").
+4. **deterministic↔residue.** Coarse = nearest-centroid palette index
+   (recomputed via AMX `matmul_i8_to_i32`); residue = signed-4-bit correction.
+5. **I-VSA-IDENTITIES correction.** COCA "codebook superposition for 2/3 pruning"
+   must bundle cluster **identity fingerprints**, NOT the CAM-PQ codes
+   (superposing PQ codes is register-loss). Pruning needs a cluster-identity
+   layer that does not exist yet → design before build.
+6. **Placement.** `ndarray` = hardware/math (codecs, reliability, entropy);
+   `lance-graph-contract` = the selector + the edge type; `causal-edge` = the
+   bit-storage; `lance-graph` = thinking (orchestration, basins, context).
+
+## The Plateau (delivered) — R1
+
+| Artifact | Repo / path | Commit |
+|---|---|---|
+| `hpc::reliability` (Pearson, Spearman, Cronbach α, ICC(2,1), `FidelityReport`) | ndarray `src/hpc/reliability.rs` | `d3b608f` |
+| `hpc::edge_codec` (Codebook k-means, `CoarseResidueCodec`, `ProductQuantizer`, `reconstruct_coarse`) | ndarray `src/hpc/edge_codec.rs` | `d3b608f` |
+| `examples/edge_codec_compare` (measure all flavors × regimes) | ndarray | `d3b608f` |
+| `hpc::entropy_ladder` (`nars_entropy`, `EntropyRung`, `Quadrant`, `PEARL_SUBSETS`, `decompose_spo`, `entropy_class`) | ndarray `src/hpc/entropy_ladder.rs` | `83be7c3` |
+| `examples/entropy_ladder_probe` (rung/quadrant partition + SPO decomposition) | ndarray | `83be7c3` |
+| `EdgeCodecFlavor` + `ClassView::edge_codec_flavor` selector | lance-graph `lance-graph-contract` | `920671d` |
+| bgz17 SIMD gather OOB guard (P1) | lance-graph `crates/bgz17/src/simd.rs` | `6d48ced` |
+
+**Measured:**
+- Edge-codec fidelity: CoarseResidue dominates agreement (ICC 0.97–0.99, ρ 0.98,
+  α 0.99); Pq32x4 preserves rank (ρ 0.60–0.67) but not absolute distance
+  (ICC 0.11–0.29); CoarseOnly collapses on continuous data (ICC 0.003). AMX
+  assign 100% vs scalar, 24–28 GMAC/s.
+- Entropy ladder: ρ(entropy, accuracy) = −0.78; balanced rung/quadrant partitions;
+  crystalline SPO → H=0.107 (Pragmatics/Wisdom), ambiguous S-only → H=1.000
+  (Syntax/Staunen).
+
+Tests: ndarray 28 new unit + 14 doctests; contract +3 (609 lib green); bgz17 +1.
+All clippy `-D warnings` clean.
+
+## Roadmap — R2..R6 + COCA
+
+| Rung | What | Plugs into | Dependency / gate |
+|---|---|---|---|
+| **R1** | NARS f/c → validity/reliability + entropy coordinate | `nars_entropy` + `reliability` | ✅ **SHIPPED** (ρ=−0.78) |
+| **R2** | Store the entropy/reliability class where it belongs | `entropy_class` → `CausalEdge64` spare bits [63:61] | `causal-edge` (zero-dep bit ops), **version-gated** + field-isolation tests (I-LEGACY-API-FEATURE-GATED) |
+| **R3** | CAM-PQ AMX centroid assignment (the "scanning" win) | `matmul_i8_to_i32`; 2×2/4×4 tiled centroid grid | independent of the ladder; `cam_pq.rs:precompute_distances` → AMX variant + bit-exact/GMAC-s probe |
+| **R4** | HHTL + helix basin attraction | `decompose_spo.basin_key` + `helix` residue | 🔬 **probe**: residue-Δ from basin centroid predicts re-access > HHTL-tier alone (+15% recall gate) |
+| **R5** | Markov SPO rung-ladder → episodic context/basins/supporting edges | `EntropyRung` over a `CausalEdge64` stream | 🔬 **probe**: ladder prunes without recall loss; builds on R1+R2 |
+| **R6** | Energy axis / particle↔wave | real `MailboxSoA.energy` → `Quadrant`'s energy input | ⏳ gated on the Mailbox-SoA map (in flight) |
+| **COCA** | superposition 2/3 pruning between 2⁸ SPO | cluster-identity FP layer (NEW) | 🛑 **I-VSA-IDENTITIES**: bundle identities not PQ codes — design first |
+
+## Probe specs (the gates)
+
+- **R4 helix-basin** — synthetic/real episodic workload; for nodes in the same
+  HHTL basin, does a small Fisher-2z helix-residue Δ from the basin centroid
+  predict re-access better than HHTL-tier membership alone? Promote to FINDING
+  and wire only if ≥ +15% recall@k; else archive ("HHTL alone sufficient").
+- **R5 Markov rung-ladder** — build word→clause→discourse centroid-SPO levels;
+  measure whether checking the low-entropy (Pragmatics) rung first prunes ≥ X%
+  of candidates with < Y% recall loss vs flat scan. Entropy band = the prune key.
+
+## D-ids
+
+- **D-EL-1** (R1) — entropy-ladder foundation + reliability + edge-codec — **Shipped**
+- **D-EL-2** (R2) — entropy class in CausalEdge64 spare bits — **Queued** (next)
+- **D-EL-3** (R3) — CAM-PQ AMX assignment — **Queued**
+- **D-EL-4** (R4) — helix-basin attraction — **Probe queued**
+- **D-EL-5** (R5) — Markov SPO rung-ladder context — **Probe queued**
+- **D-EL-6** (R6) — energy axis / particle↔wave — **Blocked** (SoA map)
+- **D-EL-COCA** — superposition pruning (cluster-identity layer) — **Design**
+
+https://claude.ai/code/session_01D2WSmezQBNC3bUdHuGfGmo