Merge pull request #23 from AdaWorldAPI/claude/continue-session-0mAVa

docs: add integration plan awareness, progress tracker, update prompts 04/05

Cross-repo integration plan now tracked in /home/user/INTEGRATION_PLAN.md.
Updated lance-graph and consumer crate status in prompts 04 and 05.
Added PROGRESS.md for per-plateau task tracking.
Tracked 2 pre-existing doctest failures in blackboard (crystal_encoder, udf_kernels).

https://claude.ai/code/session_01CdqyUTUfjKZuk8YGJzv6LB

Author: AdaWorldAPI

.claude/blackboard.md

@@ -45,10 +45,70 @@
 | `Cargo.toml` blake3 dep | ✅ Done | `blake3 = "1"` |
 | `hpc/mod.rs` declarations | ✅ Done | 11 new modules with `#[allow(missing_docs)]` |
 
-### Test Summary
-- **286 lib tests passing** (209 original + 77 new cognitive layer tests)
+### Test Summary (STALE — see audit below)
+- ~~**286 lib tests passing** (209 original + 77 new cognitive layer tests)~~ → **880 lib tests passing** (2026-03-22 audit)
 - **Clippy clean** (`cargo clippy -- -D warnings`)
-- **All doctests passing**
+- ~~**All doctests passing**~~ → **2 doctest failures** out of 302
+
+---
+
+## Epoch 4 Completion Status (2026-03-22 Audit)
+
+> The original blackboard test counts and "must be ported" checklist were massively stale.
+> Every module has significantly more tests than originally documented. All porting work is complete.
+
+### HPC Module Inventory (55 files in src/hpc/)
+
+**Core types (Step 3a)** — all DONE, test counts grew:
+| Module | Original claim | Actual tests |
+|--------|---------------|-------------|
+| fingerprint.rs | 12 | 12 |
+| plane.rs | 8 | 16 |
+| seal.rs | 5 | 4 |
+| node.rs | 6 | 9 |
+| cascade.rs | 5 | 12 |
+| bf16_truth.rs | 8 | 23 |
+| causality.rs | 6 | 17 |
+| blackboard.rs | 10 | 36 |
+
+**Additional crates (Step 3b)** — all DONE:
+| Module | Original claim | Actual tests |
+|--------|---------------|-------------|
+| bnn.rs | 6 | 26 |
+| clam.rs | 7 | 46 |
+| arrow_bridge.rs | 5 | 26 |
+
+**BLAS / Numerical** — ALL DONE:
+- blas_level1.rs (11 tests), blas_level2.rs (10), blas_level3.rs (5)
+- fft.rs (3), lapack.rs (4), vml.rs (5), statistics.rs (11), quantized.rs (7), activations.rs (9)
+
+**Cognitive / Search / Advanced** — ALL DONE (27 additional modules, ~469 tests):
+- nars, qualia, qualia_gate, hdc, spo_bundle, cogrecord, graph, merkle_tree
+- cam_index, prefilter, clam_search, clam_compress, parallel_search
+- crystal_encoder, deepnsm, dn_tree, organic, substrate, tekamolo, vsa
+- bnn_cross_plane, bnn_causal_trajectory, binding_matrix
+- bgz17_bridge, palette_distance, layered_distance, surround_metadata
+- compression_curves, cyclic_bundle, packed, bitwise, kernels, udf_kernels, projection
+
+### Backend Module (6 files in src/backend/)
+- BlasFloat trait dispatch: DONE (mod.rs, native.rs)
+- MKL FFI: DONE (mkl.rs)
+- OpenBLAS FFI: DONE (openblas.rs)
+- SIMD compat layer: DONE (simd.rs, simd_avx512.rs, simd_avx2.rs — LazyLock<Tier> AVX-512/AVX2/Scalar)
+- AVX-512 kernels: DONE (kernels_avx512.rs)
+
+### Build Status
+- Build currently fails (exit 101) — needs investigation
+- 880 lib tests pass when build succeeds
+- 2 doctest failures out of 302:
+  - `src/hpc/crystal_encoder.rs` line 251 — `distill` doctest (compile error)
+  - `src/hpc/udf_kernels.rs` line 200 — `udf_sigma_classify` doctest (assertion: `"noise" != "exact"`)
+
+### Architecture Notes
+- `LinalgBackend` trait from CLAUDE.md spec → actual impl is `BlasFloat` trait (different name, same purpose)
+- `src/simd/` directory from spec → actual is `src/simd.rs`, `src/simd_avx512.rs`, `src/simd_avx2.rs` (three top-level files)
+- `src/vector/` directory from spec → not created (functionality in hpc/)
+- Blackboard uses `HashMap<String, Box<dyn Any>>`, not a true 64-byte aligned arena
 
 ---
 
@@ -68,24 +128,24 @@
 - [x] **SIMD binary**: hamming_batch, hamming_top_k (VPOPCNTDQ + raw-slice API)
 - [x] **Cognitive layer**: Fingerprint, Plane, Node, Seal, Cascade, BF16Truth, Causality, Blackboard, BNN, CLAM, ArrowBridge
 
-### Must be ported from rustynum:
-- [ ] **Backend trait** (LinalgBackend) — pluggable BLAS dispatch
-- [ ] **BLAS L1**: sdot/ddot, saxpy/daxpy, sscal/dscal, snrm2/dnrm2, sasum/dasum, isamax/idamax, scopy/dcopy, sswap/dswap
-- [ ] **BLAS L1 SIMD**: add/sub/mul/div scalar+vec (f32/f64) — 16 functions
-- [ ] **BLAS L2**: sgemv/dgemv, sger/dger, ssymv/dsymv, strmv/dtrmv, strsv/dtrsv
-- [ ] **BLAS L3**: sgemm/dgemm, ssyrk/dsyrk, strsm, ssymm/dsymm
-- [ ] **BF16 GEMM**: BF16 type, conversions, bf16_gemm_f32, mixed_precision_gemm
-- [ ] **Int8 GEMM**: quantize_f32_to_u8/i8/i4, int8_gemm_i32/f32, per_channel variants
-- [ ] **LAPACK**: LU (getrf/getrs), Cholesky (potrf/potrs), QR (geqrf)
-- [ ] **FFT**: fft/ifft f32/f64, rfft_f32
-- [ ] **VML**: vsexp/vdexp, vsln/vdln, vssqrt/vdsqrt, vsabs/vdabs, vsadd, vsmul, vsdiv, vssin, vscos, vspow
-- [ ] **Statistics**: median, var, std, percentile (with axis variants)
-- [ ] **Array ops**: argmin, argmax, top_k, cumsum, sigmoid, softmax, log_softmax, cosine_similarity, norm(p,axis,keepdims)
-- [ ] **HDC**: bind, permute, bundle, bundle_byte_slices, dot_i8
-- [ ] **Projection**: simhash_project, simhash_batch_project, simhash_int8_project
-- [ ] **CogRecord**: 4-channel struct, new/zeros/container, hamming_4ch, sweep, to/from_bytes
-- [ ] **Graph**: VerbCodebook, encode_edge, decode_target, causality_asymmetry, causality_check, find_non_causal_edges, infer_verb
-- [ ] **Binding matrix**: binding_popcount_3d, find_holographic_sweet_spot, find_discriminative_spots
+### Must be ported from rustynum (ALL DONE as of 2026-03-22):
+- [x] **Backend trait** (BlasFloat — renamed from LinalgBackend) — src/backend/mod.rs + native.rs
+- [x] **BLAS L1** — hpc/blas_level1.rs (11 tests)
+- [x] **BLAS L1 SIMD** — hpc/blas_level1.rs (ScalarArith + VecArith traits)
+- [x] **BLAS L2** — hpc/blas_level2.rs (10 tests)
+- [x] **BLAS L3** — hpc/blas_level3.rs (5 tests)
+- [x] **BF16 GEMM** — hpc/quantized.rs (7 tests)
+- [x] **Int8 GEMM** — hpc/quantized.rs
+- [x] **LAPACK** — hpc/lapack.rs (4 tests)
+- [x] **FFT** — hpc/fft.rs (3 tests)
+- [x] **VML** — hpc/vml.rs (5 tests)
+- [x] **Statistics** — hpc/statistics.rs (11 tests)
+- [x] **Array ops** — hpc/statistics.rs + hpc/activations.rs (9 tests)
+- [x] **HDC** — hpc/hdc.rs (5 tests)
+- [x] **Projection** — hpc/projection.rs (4 tests)
+- [x] **CogRecord** — hpc/cogrecord.rs (4 tests)
+- [x] **Graph** — hpc/graph.rs (4 tests)
+- [x] **Binding matrix** — hpc/binding_matrix.rs (9 tests)
 
 ---
 
@@ -102,37 +162,4 @@
 <!-- savant-architect writes here -->
 - LinalgBackend trait: generic monomorphized (no Box<dyn> in hot paths)
 - SIMD dispatch: runtime detection via is_x86_feature_detected!
-- Feature gates: native (default), intel-mkl, openblas — mutually exclusive
-- Extension traits on ArrayBase for new operations
-- Cognitive layer: all SIMD through `hpc/bitwise.rs` dispatch (no separate SIMD paths)
-- Fingerprint<N>: zero-copy `as_bytes()` via unsafe ptr cast (SAFETY reviewed)
-- Blackboard arena: 64-byte aligned allocations, PhantomData !Send/!Sync
-- Node RNG: inline SplitMix64 (avoids rustynum dependency)
-
----
-
-## QA Audit Log
-<!-- sentinel-qa writes here -->
-- [2026-03-15] VPOPCNTDQ hamming: kernel existed in kernels_avx512.rs but dispatch_hamming() in bitwise.rs only checked AVX2. FIXED: tiered dispatch AVX-512 → AVX2 → scalar. Benchmark: 64Kbit 1.84x → 1.14x.
-- [2026-03-15] GEMM: native.rs used naive axpy-based tiling (16-20 GFLOPS). FIXED: ported Goto BLAS with 6×16 f32 / 6×8 f64 microkernels from rustyblas. Benchmark: 31-50 GFLOPS, matches reference.
-- [2026-03-15] Correctness: all kernels bit-exact or 1-ULP (FMA rounding). sgemm 64×64 max_abs_err = 0.
-- [2026-03-16] Cognitive layer migration: 11 modules ported, 286 lib tests passing, clippy clean, doctests passing.
-
----
-
-## Loose Ends
-- [x] Define feature-gate hierarchy (native/mkl/openblas) → DONE: mutually exclusive
-- [x] Backend trait: generic (monomorphized) vs enum dispatch → DONE: monomorphized
-- [x] Benchmark harness: custom bench binary with GFLOP/s reporting (see .claude/BENCHMARK_RESULTS.md)
-- [ ] CI matrix: which feature combinations to test
-- [x] Benchmark all areas at parity with rustynum (see .claude/BENCHMARK_RESULTS.md)
-- [x] Cognitive layer migration from rustynum → DONE: 11 modules, 77 tests
-- [ ] End-to-end pipeline verification (Step 6 from migration plan)
-
----
-
-## Agent Handoff Log
-<!-- Format: [agent] → [agent]: reason -->
-- [2026-03-16] cognitive-architect: Migrated Fingerprint, Plane, Seal, Node
-- [2026-03-16] cascade-architect: Migrated Cascade, PackedDatabase
-- [2026-03-16] truth-architect: Migrated BF16Truth, Causality
+- Feature gates: native (default), intel-mkl, openblas — mutu

.claude/prompts/04_lance_graph_integration.md

@@ -45,3 +45,28 @@ Use formal causal identifiability tests:
 - Arrow buffers flow through `hpc/arrow_bridge.rs` zero-copy
 - UDF registration is external to ndarray (in lance-graph or consumer crate)
 - ndarray provides the compute kernels; integration crate provides the glue
+
+---
+
+## Status Update (2026-03-22)
+
+### lance-graph Actual State (verified)
+
+The lance-graph repo has progressed beyond what this prompt originally described:
+
+- **Phase 1 (blasgraph CSC/Planner):** DONE — CscStorage, TypedGraph, blasgraph_planner, TruthGate
+- **Phase 2 (bgz17 container/semiring):** DONE — 121 tests passing, PaletteSemiring, PaletteMatrix, PaletteCsr, TypedPaletteGraph all complete
+- **Phase 3 (dual-path):** NOT STARTED — bgz17-codec feature flag, NdarrayFingerprint::plane_to_base17(), parallel_search()
+- **Phase 4 (FalkorDB retrofit):** NOT STARTED
+
+### What ndarray needs to provide for Phase 3
+
+1. `Fingerprint<256>` → Base17 encoding bridge (plane_to_base17)
+2. Cascade search results compatible with TruthGate filtering
+3. CLAM tree compatible with PaletteDistance (W125 palette indices from containers)
+
+### UDF Registration Note
+
+The UDFs listed above are defined in ndarray but registered externally in lance-graph.
+This separation is correct and should be maintained. ndarray provides kernels; lance-graph
+provides the DataFusion UDF wrappers.

.claude/prompts/05_cross_repo_map.md

@@ -74,3 +74,30 @@ Downstream crates that use ndarray's HPC modules:
 | Rung 1 (proven) | XOR group properties | Algebraic property tests |
 | Rung 2 (statistical) | Cascade bands separate signal | Distributional tests |
 | Rung 3 (needs theorem) | SPO recovers causal structure | Formal proof required |
+
+---
+
+## Status Update (2026-03-22)
+
+### Consumer Crate Status (verified audit)
+
+| Consumer | Status | Notes |
+|----------|--------|-------|
+| lance-graph | Phase 2 DONE, Phase 3-4 OPEN | bgz17 complete (121 tests), ndarray_bridge.rs exists |
+| ladybug-rs | NOT STARTED | Still uses rustynum. Migration planned as Plateau 2 in INTEGRATION_PLAN.md |
+| rs-graph-llm | NOT STARTED | graph-flow-memory crate planned (AriGraph schema port) |
+| crewai-rust | NOT STARTED | Currently uses rustynum indirectly via BindSpace |
+
+### Integration Order (from INTEGRATION_PLAN.md)
+
+1. **Plateau 0:** ndarray itself must compile clean (fix exit-101 build, 2 doctest failures)
+2. **Plateau 1:** rs-graph-llm creates graph-flow-memory, references ndarray for embeddings
+3. **Plateau 2:** ladybug-rs migrates from rustynum to ndarray (highest risk phase)
+4. **Plateau 3:** Full pipeline — ndarray compute → lance-graph query → rs-graph-llm orchestrate
+
+### rustynum Supersession
+
+rustynum's 57K LOC has been transcoded into ndarray's src/hpc/ (55 modules, 880 tests).
+The migration in consumer crates (ladybug-rs, crewai-rust via BindSpace) is tracked
+under Plateau 2 of the master plan. rustynum will remain as a path dep during the
+transition period — both deps coexist until all tests pass with ndarray alone.

.claude/prompts/research_quantized_graph_algebra.md

@@ -337,7 +337,7 @@ All code is in `AdaWorldAPI/lance-graph` and `AdaWorldAPI/ndarray`:
 - `crates/lance-graph/src/graph/blasgraph/hdr.rs` — Cascade with 2D progressive
 - `crates/bgz17/src/generative.rs` — LFD correction (generative decoding)
 - `crates/bgz17/src/similarity.rs` — SimilarityTable (CDF inversion)
-- `src/backend/simd_compat.rs` — portable SIMD for reproducible benchmarks
+- `src/simd.rs`, `src/simd_avx512.rs`, `src/simd_avx2.rs` — portable SIMD for reproducible benchmarks
 
 ## Key Equations to Derive
 

.claude/prompts/session_bgz17_similarity.md

@@ -133,7 +133,7 @@ impl SimilarityTable {
         for (i, &d) in distances.iter().enumerate() {
             out[i] = self.similarity(d);
         }
-        // TODO: SIMD path with VGATHERDPS + VCVTPH2PS when simd_compat lands
+        // TODO: SIMD path with VGATHERDPS + VCVTPH2PS when the SIMD compat layer lands
     }
 
     /// Raw table access (for inspection/debugging).
@@ -225,7 +225,7 @@ impl PaletteEdge {
 }
 ```
 
-## DELIVERABLE 4: Batch Similarity with SIMD (future, after simd_compat)
+## DELIVERABLE 4: Batch Similarity with SIMD (future, after SIMD compat layer)
 
 When the SIMD compat layer lands:
 
@@ -245,7 +245,7 @@ impl SimilarityTable {
         distances: &[u32],
         out: &mut [f32],
     ) {
-        // With simd_compat types:
+        // With crate::simd types:
         // let bucket_width = U32x16::splat(self.bucket_width);
         // for chunk in distances.chunks_exact(16).zip(out.chunks_exact_mut(16)) {
         //     let d = U32x16::from_slice(chunk.0);

.claude/prompts/session_ndarray_migration_inventory.md

@@ -94,7 +94,7 @@ It saves: write-once kernels for all architectures, clean std::simd
 migration path, simpler kernel authoring.
 
 RECOMMENDATION: Port the compat layer as P1. Put it in
-    src/backend/simd_compat.rs
+    src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs
 Wire kernels_avx512.rs to use F32x16 etc. instead of raw __m512.
 Add #[cfg(target_arch = "aarch64")] backing using NEON intrinsics later.
 Kernels become architecture-portable without rewriting.
@@ -316,10 +316,10 @@ server tick from 1.5 CPU to 0.2 CPU. Cross-referenced against existing code:
 
 ### What This Reveals About Priorities
 
-The compat layer (rustynum simd_avx512.rs → `src/backend/simd_compat.rs`)
+The compat layer (rustynum simd_avx512.rs → `src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs`)
 is the FOUNDATION for items 1, 3, 4, 6, 8. Without portable F32x16/U8x64
 types, none of the user-facing SIMD APIs can be implemented portably.
-This reinforces simd_compat as P1 — it unblocks both bgz17 integration
+This reinforces the SIMD compat layer (src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs) as P1 — it unblocks both bgz17 integration
 AND the Pumpkin feature set.
 
 Items that already have internal implementations but lack user-facing API:
@@ -415,9 +415,9 @@ SIMD: DECOMPOSED (not missing), but COMPAT LAYER not ported:
         simd_avx512.rs (2643)  → backend/kernels_avx512.rs           962
                                   ↑ MISSING: compat layer types (F32x16 etc.)
                                     kernels use raw __m512 instead
-                                    Port as src/backend/simd_compat.rs (P1)
+                                    Port as src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs (P1)
         simd_isa.rs (215)      → backend/mod.rs Tier enum             165
-        simd_compat.rs (4)     → (not needed)
+        simd_compat.rs (4)     → replaced by src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs
 ```
 
 ### Cross-Reference: rustynum-rs → ndarray
@@ -679,7 +679,7 @@ Ordered list based on:
    palette distance in kernels_avx512.rs
 2. **P1 — SIMD compat layer:** port rustynum simd_avx512.rs type system
    (F32x16, F64x8, U8x64, I32x16, SimdFloat trait) into
-   `src/backend/simd_compat.rs`. Refactor kernels_avx512.rs to use
+   `src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs`. Refactor kernels_avx512.rs to use
    compat types instead of raw `__m512`. Zero runtime cost. Unlocks:
    aarch64/NEON support, std::simd migration, simpler kernel authoring.
    ALSO UNBLOCKS Pumpkin items 1,3,4,6,8 (simd_map, xor_diff, gather,

.claude/prompts/session_unified_vector_search.md

@@ -352,7 +352,7 @@ for chunk in packed_db.array_windows::<RECORD_BYTES>() { ... }
 
 ## DELIVERABLE 9: Redirect lance-graph Raw Intrinsics
 
-After simd_compat lands and bgz17 joins workspace:
+After the SIMD compat layer (src/simd.rs, src/simd_avx512.rs, src/simd_avx2.rs) lands and bgz17 joins workspace:
 
 ```
 ndarray_bridge.rs (6 Hamming functions, ~200 lines)

CLAUDE.md

@@ -36,29 +36,39 @@ When summarizing this conversation, preserve:
 - Which agents have been consulted and their verdicts
 - Any BLOCK findings from sentinel-qa
 
-## Repository Structure (Target)
+## Repository Structure (Actual as of 2026-03-22)
 ```
 src/
 ├── lib.rs              # Re-exports, feature gates
 ├── backend/
-│   ├── mod.rs          # LinalgBackend trait
-│   ├── native.rs       # Pure Rust + SIMD
+│   ├── mod.rs          # BlasFloat trait (was planned as LinalgBackend)
+│   ├── native.rs       # Pure Rust + SIMD microkernels
 │   ├── mkl.rs          # Intel MKL FFI (feature = "intel-mkl")
-│   └── openblas.rs     # OpenBLAS FFI (feature = "openblas")
-├── linalg/
-│   ├── gemm.rs         # General matrix multiply
-│   ├── syrk.rs         # Symmetric rank-k update
-│   └── trsm.rs         # Triangular solve
-├── simd/
-│   ├── mod.rs          # Runtime detection, dispatch
-│   ├── avx512.rs       # AVX-512 kernels
-│   ├── avx2.rs         # AVX2 fallback
-│   └── sse42.rs        # SSE4.2 minimum
-└── vector/
-    ├── distance.rs     # Cosine, L2, dot product
-    ├── batch.rs        # Pairwise, top-k
-    └── index.rs        # VectorIndex trait
-benches/
-├── gemm_bench.rs       # criterion benchmarks
-└── vector_bench.rs
+│   ├── openblas.rs     # OpenBLAS FFI (feature = "openblas")
+├── simd.rs             # Consumer-facing SIMD module, re-exports all types
+├── simd_avx512.rs      # AVX-512 type definitions (11 types from rustynum)
+├── simd_avx2.rs        # AVX2 functions
+│   └── kernels_avx512.rs  # AVX-512 kernel implementations
+├── hpc/                # 55 modules — ALL DONE (880 lib tests)
+│   ├── blas_level1.rs  # BLAS L1 (dot, axpy, scal, nrm2, asum, etc.)
+│   ├── blas_level2.rs  # BLAS L2 (gemv, ger, symv, trmv, trsv)
+│   ├── blas_level3.rs  # BLAS L3 (gemm, syrk, trsm, symm)
+│   ├── quantized.rs    # BF16 GEMM, Int8 GEMM
+│   ├── lapack.rs       # LU, Cholesky, QR
+│   ├── fft.rs          # FFT/IFFT (Cooley-Tukey radix-2)
+│   ├── vml.rs          # Vector math (exp, ln, sqrt, etc.)
+│   ├── statistics.rs   # Median, var, std, percentile, top_k
+│   ├── activations.rs  # Sigmoid, softmax, log_softmax
+│   ├── fingerprint.rs, plane.rs, seal.rs, node.rs  # Cognitive core
+│   ├── cascade.rs, bf16_truth.rs, causality.rs     # Truth/cascade
+│   ├── blackboard.rs   # Typed slot arena
+│   ├── bnn.rs, clam.rs, arrow_bridge.rs            # Additional crates
+│   ├── hdc.rs, nars.rs, qualia.rs, spo_bundle.rs   # Cognitive extensions
+│   └── ... (27 more modules)
 ```
+
+## Status (2026-03-22 Audit)
+- **All "must be ported" items: DONE** — see `.claude/blackboard.md` for full inventory
+- **880 lib tests passing**, 2 doctest failures out of 302
+- **Build currently fails (exit 101)** — needs investigation
+- See blackboard for detailed per-module test counts

MIGRATION_INVENTORY.md

@@ -249,7 +249,7 @@ These modules exist in the upstream rustynum workspace but have **no counterpart
 | `simd_avx2.rs` | 600 | AVX2-specific kernels |
 | `simd_avx512.rs` | 2 643 | AVX-512 kernels |
 | `simd_isa.rs` | 215 | ISA detection |
-| `simd_compat.rs` | 4 | Compat shim |
+| `simd.rs` / `simd_avx512.rs` / `simd_avx2.rs` | — | SIMD compat layer (re-exports, AVX-512 types, AVX2 functions) |
 | `hybrid.rs` | 2 355 | Hybrid compute pipeline |
 | `jitson.rs` | 1 688 | JIT JSON/binary codec |
 | `jit_scan.rs` | 385 | JIT scan operations |