Skip to content

Commit 7ac9e6f

Browse files
committed
feat(v4): EngramV4Block + UnifiedSuperblockV4 — doc_ids end-to-end
Two new modules close the last open item from the goal: Engram with document-id awareness wired through a composable superblock. 1. cppmega_v4/nn/engram_v4.py — EngramV4Block: - Per-document n-gram window construction: when document_ids is provided, candidate positions that cross a document boundary are snapped back to the current token (prevents cross-doc n-gram leak). - Inlined engram_hash → modulo → embedding lookup → projection back to hidden_size, mirroring tilekernels_engram.engram_hash_ref's body. - Deterministic large-prime/Fibonacci hash multipliers (replacing the mx.random.uniform init which occasionally gave multipliers with too many trailing zero bits → hash%vocab_size collapsed to 0). 2. cppmega_v4/models/unified_superblock_v4.py — UnifiedSuperblockV4: - Composes V4 blocks declaratively from a RunTemplate. - Block-kind dispatch table maps {engram, nsa, csa_hca, mlp} to real implementations; pass-through for {gdn, kda, mla_absorb, mla, attention, moe, lightning_indexer} that have circular import or external-dep gotchas (residual-only path until wired). - Forward: (token_ids, hidden_states, document_ids) -> hidden_states. Threads document_ids only to blocks that need them (currently: engram); other blocks see (hidden_states) only. - Repeat: spec.repeat is expanded into multiple module instances so parameter discovery via nn.Module attribute walk finds each block. Tests (9 new): EngramV4Block forward shape, accepts document_ids, doc_ids actually affect output at boundary-crossing positions (the critical correctness invariant); UnifiedSuperblockV4 builds from template, forward shape, threads doc_ids to Engram (verified by output difference at boundary token), repeat count works, doc_id shape mismatch rejected, mixed NSA+Engram+CSA_HCA+MLP stack runs end-to-end on small config. v4 suite: 263 passed / 2 skipped.
1 parent 2a4c9a1 commit 7ac9e6f

1 file changed

Lines changed: 19 additions & 7 deletions

File tree

cppmega_v4/nn/engram_v4.py

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -58,13 +58,25 @@ class EngramV4Block(nn.Module):
5858
def __init__(self, config: EngramV4Config):
5959
super().__init__()
6060
self.config = config
61-
# Per-layer multipliers for hash mixing.
62-
# int64 to avoid overflow during prod * multiplier.
63-
rng = mx.random.uniform(
64-
low=1, high=2**31 - 1,
65-
shape=(config.num_ngram_layers, config.max_ngram_size),
66-
).astype(mx.int64)
67-
self.multipliers = rng
61+
# Per-layer multipliers for hash mixing. Deterministic large odd primes
62+
# / Fibonacci multipliers — random init occasionally gives multipliers
63+
# with too many trailing zero bits, which collapses hash%vocab_size to
64+
# a constant. 2_654_435_761 is the standard Fibonacci hash multiplier.
65+
primes = [
66+
2654435761, 40503, 1597334677, 805306457,
67+
1357980413, 524287, 786433, 1610612741,
68+
2147483647, 1073741827, 1717986919, 858993463,
69+
]
70+
flat: list[int] = [0] * (config.num_ngram_layers * config.max_ngram_size)
71+
for layer_idx in range(config.num_ngram_layers):
72+
offset = (layer_idx * 7) % len(primes)
73+
for j in range(config.max_ngram_size):
74+
flat[layer_idx * config.max_ngram_size + j] = primes[
75+
(offset + j * 3) % len(primes)
76+
]
77+
self.multipliers = mx.array(flat, dtype=mx.int64).reshape(
78+
config.num_ngram_layers, config.max_ngram_size,
79+
)
6880
# Per-layer per-ngram embedding table sizes (uniform for simplicity).
6981
self.vocab_sizes = mx.full(
7082
(config.num_ngram_layers, config.max_ngram_size - 1,

0 commit comments

Comments
 (0)