Merge pull request #151 from AdaWorldAPI/claude/compare-rustynum-ndarray-5ePRn

docs: update 11 markdown files for 16K container layout

Author: AdaWorldAPI

ARCHITECTURE.md

@@ -410,22 +410,23 @@ https://github.com/AdaWorldAPI/ladybug-rs
 
 The sections below document the deep container substrate that underlies
 the CAM/scent architecture above.  Where Part I describes the indexing
-and query surface, Part II describes the 8192-bit cognitive geometry,
+and query surface, Part II describes the 16,384-bit cognitive geometry,
 the DN tree, NARS reasoning, qualia modules, and the Friston free energy
 loop that ties them together.
 
 ---
 
 ## 11. Container Geometry
 
-The **Container** is the atomic unit.  8192 bits = 128 × u64 words = 1 KB.
+The **Container** is the atomic unit.  16,384 bits = 256 × u64 words = 2 KB.
 Stack-allocated, SIMD-aligned (`#[repr(C, align(64))]`), zero-copy.
+Each CogRecord IS one container. A node has N containers (meta, content, embeddings...).
 
 ```text
 ┌──────────────────────────────────────────────────────────┐
-│  128 words × 64 bits = 8192 bits = 1 KB                 │
-│  16 AVX-512 iterations cover the full container          │
-│  Expected random Hamming distance: 4096 (σ ≈ 45)        │
+│  256 words × 64 bits = 16,384 bits = 2 KB               │
+│  32 AVX-512 iterations cover the full container          │
+│  Expected random Hamming distance: 8192 (σ = 64)        │
 └──────────────────────────────────────────────────────────┘
 ```
 
@@ -1114,9 +1115,9 @@ SNN/ANN/GNN approaches:
    relationships compose via XOR without loss.  GNNs lose information at
    each message-passing layer.
 
-3. **Constant-size representation**: An edge between two 8192-bit containers
-   is another 8192-bit container.  No matter how complex the relationship,
-   it fits in 1 KB.  Neural edge representations grow with model size.
+3. **Constant-size representation**: An edge between two 16,384-bit containers
+   is another 16,384-bit container.  No matter how complex the relationship,
+   it fits in 2 KB.  Neural edge representations grow with model size.
 
 4. **Information content is measurable**: `popcount(a xor b)` = how many bits
    differ = the energy of the transformation.  This is an exact count, not
@@ -1248,7 +1249,7 @@ The key transcoding: RedisGraph stored adjacency as integer node IDs in CSR
 format.  The holograph step replaced integer IDs with Container fingerprints
 and integer edge weights with XOR deltas.  BlasGraph formalized this as
 sparse-adjacent-vector operations.  ContainerGraph (ladybug-rs) is the final
-form: pure Container-native, everything is 8192 bits, all operations are
+form: pure Container-native, everything is 16,384 bits, all operations are
 XOR/Hamming/popcount.
 
 The adjacency encoding in W16-31 (inline) and W96-111 (CSR overflow) is the

HANDOVER.md

@@ -140,7 +140,7 @@ order:
 
 Extended from 402 → 1,649 lines. Preserved existing CAM/scent-index sections
 (1-10). Added 17 new sections covering:
-- Container Geometry (8192-bit atom, XOR/Hamming/popcount)
+- Container Geometry (16,384-bit atom, XOR/Hamming/popcount)
 - CogRecord (2 KB holy grail layout)
 - Container 0 Metadata Map (W0-W127 complete)
 - DN Tree (PackedDn 7×8-bit)
@@ -399,7 +399,7 @@ Complete mapping between the Python ecosystem and ladybug-rs:
 | File | What |
 |------|------|
 | **Container substrate** | |
-| `crates/ladybug-contract/src/container.rs` | CONTAINER_BITS=8192, EXPECTED_DISTANCE=4096, SIGMA=45.25 |
+| `crates/ladybug-contract/src/container.rs` | CONTAINER_BITS=16384, EXPECTED_DISTANCE=8192, SIGMA=64.0 |
 | `crates/ladybug-contract/src/record.rs` | CogRecord (2 KB = meta + content), cross_hydrate, extract_perspective |
 | `crates/ladybug-contract/src/nars.rs` | TruthValue, revision/deduction/induction/abduction/analogy/comparison |
 | `src/container/meta.rs` | W0-W127 metadata layout, MetaView/MetaViewMut |

INTEGRATION_PLAN.md

@@ -37,10 +37,9 @@ It survives context resets and serves as the task skeleton for agents.
                              ▼
                     ┌──────────────────┐
                     │ ladybug-contract │
-                    │ Container 8192b  │
-                    │ WideContainer    │
-                    │ CogRecord8K     │
-                    │ CogPacket wire  │
+                    │ Container 16384b │
+                    │ CogRecord 16K    │
+                    │ CogPacket wire   │
                     └────────┬─────────┘
                              ▼
                     ┌──────────────────┐
@@ -296,15 +295,14 @@ UnifiedExecution
 #### 5.2 Future Primitives [TODO]
 - [ ] Majority vote bundle (used by CognitiveKernel.bundle_recent)
 - [ ] Hamming similarity (portable version of AVX-512 VPOPCNTDQ)
-- [ ] Fingerprint XOR-fold (Container 8192 ↔ Fingerprint 16384)
+- [ ] Fingerprint XOR-fold (Container = Fingerprint = 16384 bits)
 - [ ] Focus mask: dimension selection based on codebook crystallization history
 
 ### 6. ladybug-contract (Pure Types)
 
 #### 6.1 Current State [DONE]
-- [x] Container (8192-bit, 128 × u64)
-- [x] WideContainer (16384-bit, 256 × u64)
-- [x] CogRecord / CogRecord8K
+- [x] Container (16384-bit, 256 × u64) *(updated Feb 2026 from 8192-bit)*
+- [x] CogRecord (each container = 16K = 2 KB)
 - [x] CogPacket wire protocol
 - [x] EmbeddingFormat enum
 

REALITY_CHECK.md

@@ -92,6 +92,11 @@ src/container/mod.rs:73        — [u64; 128] = 8,192 bits = 1 KB
 crates/ladybug-contract/src/container.rs:29 — [u64; 128] = 8,192 bits = 1 KB
 ```
 
+> **UPDATE (Feb 2026):** Container has been widened to 16,384 bits (256 × u64 = 2 KB).
+> Container = Fingerprint = same width. No more truncation or zero-extension.
+> The dual-Container type issue (src/container vs contract crate) remains a
+> cleanup target, but the width mismatch is resolved.
+
 They are **identical in layout** but **different Rust types**. You cannot pass one
 where the other is expected without conversion. This means:
 
@@ -102,44 +107,47 @@ where the other is expected without conversion. This means:
 - `src/container/cache.rs`, `src/container/graph.rs` use the local one
 - Type mismatch between modules that should be the same
 
-**AND** there is `Fingerprint` (256 u64 = 16,384 bits = 2 KB) in `src/core/fingerprint.rs`.
+~~**AND** there is `Fingerprint` (256 u64 = 16,384 bits = 2 KB) in `src/core/fingerprint.rs`.
 Two `From` impls exist to convert:
 - Fingerprint → Container: truncation (copy first 128 words, discard upper 128)
 - Container → Fingerprint: zero-extension (copy 128 words, pad 128 zeros)
 
 This means **half the fingerprint is thrown away** when going to storage.
-Or storage records carry **128 zero words** when promoted to Fingerprint.
+Or storage records carry **128 zero words** when promoted to Fingerprint.~~
+
+> **RESOLVED (Feb 2026):** Container and Fingerprint are now both 256 × u64 = 16,384 bits.
+> Conversion is direct copy — no truncation, no zero-extension.
 
 ### THE FIX
 
-The correct layout (which you identified in a previous session) is:
+> **UPDATE (Feb 2026):** The canonical layout is now implemented:
+> - Every Container = 16,384 bits = 256 × u64 = 2 KB. Always.
+> - Container 0 = Metadata CogRecord (16K): W0-W127 MetaView fields, W224-W255 SchemaSidecar
+> - Container 1 = Content CogRecord (16K): All 256 words = searchable VSA fingerprint
+> - Container N = Additional CogRecords (Jina embeddings, etc.)
+> - Constants: CONTAINER_BITS=16384, CONTAINER_WORDS=256, CONTENT_OFFSET=0, CONTENT_WORDS=256
+>
+> The old "8192+8192 = one Fingerprint" model is superseded. Each CogRecord IS one
+> full-width container. A node is composed of separate CogRecords (Container 0 meta,
+> Container 1 content, etc.), each 2 KB.
 
-```
+~~The correct layout (which you identified in a previous session) is:~~
+
+~~```
 CogRecord = 8,192-bit metadata (W0-W127) + 8,192-bit content (W0-W127)
            = 2 Containers = 2 KB total
            = Exactly 1 Fingerprint
-```
+```~~
 
-This means:
+Remaining work:
 1. **Delete `src/container/mod.rs:73` Container** — re-export from contract crate
-2. **Make CogRecord = exactly 1 Fingerprint** — upper 128 words = metadata, lower 128 = content
-   OR: Container 0 (meta) + Container 1 (content), serialized as one Fingerprint
-3. **DN tree in Redis** — each key maps to exactly 2 KB = 1 Fingerprint = 1 CogRecord
-4. **Spine** — XOR of content containers IS the tree spine, same as Redis DN tree
-
-What changes:
-- `crates/ladybug-contract/src/record.rs` — CogRecord becomes `[Container; 2]` not `meta + Vec<Container>`
-- `src/core/fingerprint.rs` — Fingerprint IS a CogRecord (upper=meta, lower=content)
-- Kill `src/container/mod.rs` — everything uses `ladybug_contract::container::Container`
-- `ContainerGeometry::Cam` (the default, most common) stays as 1 meta + 1 content = 2 KB
-- Multi-container geometries (Xyz, Chunked, Tree) become linked lists of 2 KB records via DN tree
+2. Kill duplicate Container type — everything uses `ladybug_contract::container::Container`
 
 ### WHY THIS MATTERS
 
-Right now searching requires loading Container (1 KB) then separately loading metadata.
-With 8192+8192, every record is self-contained. One 2 KB read gives you everything:
-identity, NARS truth, edges, AND the searchable content fingerprint.
-Zero joins. Zero second lookups. The record IS the DN tree node IS the Redis value.
+Each container is self-contained at 2 KB. One read gives you the full 16,384-bit
+searchable fingerprint (Container 1) or the full metadata (Container 0).
+Zero truncation. Zero zero-extension. The record IS the DN tree node IS the storage value.
 
 ---
 
@@ -420,45 +428,41 @@ Wire them into the cognitive kernel (`src/cognitive/cognitive_kernel.rs`):
 
 ---
 
-## THE HOLY GRAIL: 8192 META + 8192 CONTENT
+## THE HOLY GRAIL: 16K-PER-CONTAINER MODEL
+
+> **UPDATE (Feb 2026):** This section described the old "8192+8192" model.
+> The canonical layout is now: every Container = 16,384 bits = 256 × u64 = 2 KB.
+> A node has separate containers (Container 0 = metadata, Container 1 = content, etc.),
+> each one a full 16K-bit CogRecord.
 
-### Current State (Wrong)
+### Previous State (Fixed)
 
 ```
 Fingerprint = 256 u64 = 16,384 bits = 2 KB    (src/core/)
-Container   = 128 u64 =  8,192 bits = 1 KB    (contract + src/container/ DUPLICATE)
+Container   = 256 u64 = 16,384 bits = 2 KB    (contract — updated from 128 u64)
 CogRecord   = 1 meta Container + Vec<Container>  (variable size, heap allocated)
 CogPacket   = 8-word header + 1-2 Containers     (wire protocol)
 ```
 
-Problems:
-- Fingerprint → Container loses half the data (truncation at conversion)
-- CogRecord is heap-allocated Vec (variable size = no zero-copy, no mmap)
-- Two Container types cause type confusion
-- Wire protocol adds its own 64-byte header, different from meta.rs W0-W127
-
-### Target State (8192 + 8192)
+### Current State (Implemented)
 
 ```
-Container   = 128 u64 = 8,192 bits = 1 KB     (ONE type, in contract)
-CogRecord   = [Container; 2] = 2 KB fixed      (meta + content, stack allocated)
-Fingerprint = type alias for CogRecord          (or From<CogRecord> zero-cost)
+Container   = 256 u64 = 16,384 bits = 2 KB     (ONE type, in contract)
+Fingerprint = 256 u64 = 16,384 bits = 2 KB     (same width as Container)
+CogRecord   = separate Containers (meta + content + ...)
 DN tree key = PackedDn (8 bytes)
-DN tree val = CogRecord (2 KB fixed)
-Redis key   = DN address
-Redis value = 2 KB blob (identical to CogRecord)
+DN tree val = Container(s) (2 KB each)
 ```
 
 ### What this gives you
 
-1. **Zero-copy everything**: mmap a file, cast to `&[CogRecord]`, done
-2. **No heap allocation**: `[Container; 2]` lives on the stack
-3. **DN tree = Redis = Storage**: exact same 2 KB blob everywhere
-4. **Spine = XOR of content containers**: `spine = records.iter().fold(Container::zero(), |s, r| s.xor(&r.content))`
-5. **SIMD on full record**: 2 x 16 AVX-512 iterations = 32 iterations per record
-6. **One lookup per node**: GET dn_addr → 2 KB → you have meta + content + edges + NARS
-7. **CLAM tree over CogRecords**: one tree indexes both metadata and content
-8. **panCAKES compression on content container**: XOR-diff from cluster center, 5-70x ratio
+1. **No truncation**: Container = Fingerprint = 16,384 bits, direct copy
+2. **Full-width SIMD**: 32 AVX-512 iterations per Container (256 words / 8)
+3. **σ = 64.0 exactly**: sqrt(16384/4) = 64.0, simplifies all threshold math
+4. **Spine = XOR of content containers**: `spine = records.iter().fold(Container::zero(), |s, r| s.xor(&r.cam))`
+5. **One lookup per container**: GET dn_addr → 2 KB → you have the full 16K fingerprint
+6. **CLAM tree over Containers**: one tree indexes content directly
+7. **panCAKES compression on content container**: XOR-diff from cluster center, 5-70x ratio
 
 ### Migration Path
 
@@ -491,10 +495,10 @@ Week 2: crewai-rust Resurrection
   [6]  Wire task.execute_sync() to real execution (Issue #2)
   [7]  Delete dead wire_bridge or call it (Issue #2)
 
-Week 3: The Holy Grail — 8192+8192
-  [8]  CogRecord = [Container; 2] (Issue #1, steps 2-5)
-  [9]  Update storage to fixed 2 KB records (steps 6-8)
-  [10] Fingerprint = CogRecord alias (step 9)
+Week 3: The Holy Grail — 16K per Container *(DONE Feb 2026)*
+  [8]  ~~CogRecord = [Container; 2]~~ → Container widened to 16384 bits ✓
+  [9]  ~~Update storage to fixed 2 KB records~~ → Constants updated ✓
+  [10] ~~Fingerprint = CogRecord alias~~ → Same width, direct copy ✓
 
 Week 4: Connect the Pipes
   [11] Bridge Grammar → CausalSearch (Issue #3)
@@ -582,9 +586,9 @@ into "scientific breakthrough."
 
 ### The gap between "looks impressive" and "actually works" is ~8 weeks of focused work.
 
-The 8192+8192 change is the architectural unlock (weeks 1-3).
+The 16K-per-container change is the architectural unlock *(completed Feb 2026)*.
 CLAM integration is the scientific unlock (weeks 5-8).
-Everything else follows from having one canonical 2 KB record type
+Everything else follows from having one canonical 2 KB container type
 that IS the fingerprint, IS the DN tree node, IS the Redis value,
 IS the search vector, IS the CLAM tree leaf, IS the storage unit.
 

docs/BINDSPACE_UNIFICATION.md

@@ -2,6 +2,14 @@
 
 > **Date**: 2026-02-13
 > **Status**: PLAN — awaiting implementation
+>
+> **NOTE (Feb 2026):** This document was written when Container = 128 × u64 = 8,192 bits = 1 KB.
+> Container is now 256 × u64 = 16,384 bits = 2 KB. The "8K vs 16K" comparison table
+> and the 128/128 metadata/content split described in §6-§7 are **superseded**.
+> Canonical layout: every Container = 16,384 bits = 256 words. Container 0 = metadata,
+> Container 1 = content (full 256 words = searchable fingerprint), Container N = additional.
+> The BindSpace concepts (typed lens, Arrow backing, DnIndex) remain valid.
+> Specific word-offset references to [0..128] metadata / [128..256] content need updating.
 > **Branch for work**: Create fresh branch from `claude/code-review-SMMuY`
 > **Rollback**: `git revert` the implementation commit(s); all consolidation files remain untouched on the parent branch
 
@@ -1875,7 +1883,7 @@ type Node {
   children: [Node!]!              # dn_index.children(addr)
   parent: Node                    # dn.parent() → resolve
   content: Fingerprint            # content_container()
-  similarity(to: Fingerprint!): Float  # content.hamming() → 1.0 - dist/8192
+  similarity(to: Fingerprint!): Float  # content.hamming() → 1.0 - dist/16384
 }
 
 type Query {

docs/COGNITIVE_RECORD_192.md

@@ -1,7 +1,19 @@
 # The 192×u64 Cognitive Record
 
 **Date**: 2026-02-05
-**Status**: Proposal (alternate to COMPOSITE_FINGERPRINT_SCHEMA.md)
+**Status**: ~~Proposal~~ **SUPERSEDED**
+
+> **NOTE (Feb 2026):** This document proposed a 192-word record with an 8,192-bit
+> (128 × u64) fingerprint lane. The canonical layout is now:
+> - Every Container = 16,384 bits = 256 × u64 = 2 KB
+> - Container 0 = Metadata (16K), Container 1 = Content (16K), Container N = Additional
+> - The 192-word layout was never implemented. See `width_16k/mod.rs` for the current spec.
+> - Many of the GraphBLAS/semiring concepts remain valid and are implemented in
+>   `src/container/semiring.rs` and `src/width_16k/search.rs`.
+>
+> Kept for historical reference — the adjacency bitvector concept and DataFusion operator
+> model described here informed later design decisions.
+
 **Core idea**: One 1,536-byte fixed-size record is the node, the edge row, the sparse adjacency vector, the NARS belief, the scent, AND the VSA fingerprint. Every cognitive operation reduces to SIMD on a lane of the same buffer. Zero-copy from storage through compute through transport.
 
 ---

docs/COGRECORD_65536.md

@@ -10,17 +10,18 @@ February 21, 2026 — AdaWorldAPI / Claude Architecture Audit
 
 ## 0. The Insight
 
-Current: `Container = [u64; 128]` = 8,192 bits = 1 KB.
-Current: `CogRecord = [Container; 2]` = 16,384 bits = 2 KB.
-Current: `Fingerprint = [u64; 256]` = 16,384 bits = 2 KB.
+> **UPDATE (Feb 2026):** Phase 0 of this plan is now implemented.
+> `CONTAINER_BITS = 16_384`, `CONTAINER_WORDS = 256`, `SIGMA = 64.0`.
+> Container = Fingerprint = same width. No truncation. No zero-extension.
+> The "Current" state below reflects the pre-change baseline.
 
-Container is **half-width Fingerprint**. Every `From<&Fingerprint> for Container` truncates. Every `From<&Container> for Fingerprint` zero-extends. This is wasted information.
-
-**New: Container = Fingerprint = `[u64; 256]` = 16,384 bits = 2 KB.**
+~~Current: `Container = [u64; 128]` = 8,192 bits = 1 KB.~~
+~~Current: `CogRecord = [Container; 2]` = 16,384 bits = 2 KB.~~
+**Implemented: Container = Fingerprint = `[u64; 256]` = 16,384 bits = 2 KB.**
 
 One type. Full width. No truncation. No zero-extension. The conversion functions become identity operations.
 
-Then: `CogRecord = [Container; 4]` = 65,536 bits = 8 KB.
+Future: `CogRecord = [Container; 4]` = 65,536 bits = 8 KB.
 
 With AVX-512 `VPOPCNTDQ` (confirmed present on this hardware), a full 65,536-bit sweep costs 128 instructions — the same throughput class as the current 8,192-bit sweep's 16 instructions. The pipeline fills identically; only the iteration count changes.
 
@@ -159,16 +160,10 @@ match meta.embedding_metric() {
 
 The entire change starts at one file: `crates/ladybug-contract/src/container.rs`.
 
-### Phase 0: Change Constants (1 hour)
+### Phase 0: Change Constants ~~(1 hour)~~ **DONE (Feb 2026)**
 
 ```rust
-// BEFORE:
-pub const CONTAINER_BITS: usize = 8_192;
-pub const CONTAINER_WORDS: usize = CONTAINER_BITS / 64;  // 128
-pub const CONTAINER_BYTES: usize = CONTAINER_WORDS * 8;  // 1024
-pub const CONTAINER_AVX512_ITERS: usize = CONTAINER_WORDS / 8;  // 16
-
-// AFTER:
+// IMPLEMENTED in crates/ladybug-contract/src/container.rs:
 pub const CONTAINER_BITS: usize = 16_384;
 pub const CONTAINER_WORDS: usize = CONTAINER_BITS / 64;  // 256
 pub const CONTAINER_BYTES: usize = CONTAINER_WORDS * 8;  // 2048
@@ -177,15 +172,10 @@ pub const CONTAINER_AVX512_ITERS: usize = CONTAINER_WORDS / 8;  // 32
 
 Because the code uses `CONTAINER_WORDS` everywhere (not hardcoded `128`), **most code compiles immediately**. The 17 files importing these constants just get wider containers.
 
-### Phase 0.1: Statistical Constants
+### Phase 0.1: Statistical Constants **DONE (Feb 2026)**
 
 ```rust
-// BEFORE:
-pub const EXPECTED_DISTANCE: u32 = 4096;  // 8192/2
-pub const SIGMA: f64 = 45.254833995939045;  // √(8192/4)
-pub const SIGMA_APPROX: u32 = 45;
-
-// AFTER:
+// IMPLEMENTED:
 pub const EXPECTED_DISTANCE: u32 = 8192;  // 16384/2
 pub const SIGMA: f64 = 64.0;  // √(16384/4) = √4096 = 64.0 exactly
 pub const SIGMA_APPROX: u32 = 64;