Commit 7ac9e6f
committed
feat(v4): EngramV4Block + UnifiedSuperblockV4 — doc_ids end-to-end
Two new modules close the last open item from the goal: Engram with
document-id awareness wired through a composable superblock.
1. cppmega_v4/nn/engram_v4.py — EngramV4Block:
- Per-document n-gram window construction: when document_ids is
provided, candidate positions that cross a document boundary are
snapped back to the current token (prevents cross-doc n-gram leak).
- Inlined engram_hash → modulo → embedding lookup → projection back
to hidden_size, mirroring tilekernels_engram.engram_hash_ref's body.
- Deterministic large-prime/Fibonacci hash multipliers (replacing the
mx.random.uniform init which occasionally gave multipliers with
too many trailing zero bits → hash%vocab_size collapsed to 0).
2. cppmega_v4/models/unified_superblock_v4.py — UnifiedSuperblockV4:
- Composes V4 blocks declaratively from a RunTemplate.
- Block-kind dispatch table maps {engram, nsa, csa_hca, mlp} to real
implementations; pass-through for {gdn, kda, mla_absorb, mla,
attention, moe, lightning_indexer} that have circular import or
external-dep gotchas (residual-only path until wired).
- Forward: (token_ids, hidden_states, document_ids) -> hidden_states.
Threads document_ids only to blocks that need them (currently:
engram); other blocks see (hidden_states) only.
- Repeat: spec.repeat is expanded into multiple module instances so
parameter discovery via nn.Module attribute walk finds each block.
Tests (9 new): EngramV4Block forward shape, accepts document_ids,
doc_ids actually affect output at boundary-crossing positions (the
critical correctness invariant); UnifiedSuperblockV4 builds from
template, forward shape, threads doc_ids to Engram (verified by
output difference at boundary token), repeat count works, doc_id
shape mismatch rejected, mixed NSA+Engram+CSA_HCA+MLP stack runs
end-to-end on small config.
v4 suite: 263 passed / 2 skipped.1 parent 2a4c9a1 commit 7ac9e6f
1 file changed
Lines changed: 19 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
68 | 80 | | |
69 | 81 | | |
70 | 82 | | |
| |||
0 commit comments