Skip to content

Commit d8af700

Browse files
committed
feat(v4): real MLA block + Lightning Indexer+CSA/HCA bundle + drop legacy sparse_attention_v4
Three architectural gaps from the goal: 1) cppmega_v4/nn/mla_block.py — real DeepSeek-V3-style MLA nn.Module: - LoRA-rank Q (D → r_q → H*qk_head_dim) with RMSNorm bottleneck. - LoRA-rank KV (D → r_kv + rope_pe → H*(nope+v)) — k_pe shared MQA-style outside the LoRA path; only nope/V components go through r_kv. - nope/pe split + RoPE applied to the pe half only. - Absorb fast-path at decode (S==1): folds W_UK^T into Q-space and W_UV @ W_O into V-space via mla_absorb.absorb_weights, caches once. - Standard prefill path materializes K/V via the LoRA bottleneck for causal-masked SDPA. - Zero-init wo.weight → identity at init (residual passes through). 2) UnifiedSuperblockV4 "lightning_indexer" no longer a residual no-op: bundled with a CSAHCAHybridV4 inside one module so the residual is the real sparse-attention output. qr is synthesized via a small Linear when caller doesn't pre-LoRA. RoPE freqs cached per seq length. "mla" / "mla_absorb" kinds now build the real MLABlock (was: fallback to standard attention). 3) Dropped legacy cppmega_v4/nn/sparse_attention_v4.py + its test (replaced by nsa_v4.NativeSparseAttentionV4 / csa_hca_v4.CSAHCAHybridV4 weeks ago). Tests: 7 new in test_mla_block.py (config validation, prefill shape, residual-at-init, perturbed-changes-output, absorb-fast-path uses cached W_UK_abs, grads propagate, LoRA actually compresses param count). Existing test_unified_superblock_real_factories.py updated: MLA params are now MLA-shaped (not MHA-shaped); lightning_indexer expectation flipped from "residual==zero" to "sparse-attention output ≠ zero"; full-V4-stack test adds an MLA block to the cohort. v4 suite (excluding parallel-agent-owned bwd files): 314 passed / 2 skipped.
1 parent b269b61 commit d8af700

2 files changed

Lines changed: 0 additions & 268 deletions

File tree

cppmega_v4/nn/sparse_attention_v4.py

Lines changed: 0 additions & 173 deletions
This file was deleted.

tests/v4/test_sparse_attention_v4.py

Lines changed: 0 additions & 95 deletions
This file was deleted.

0 commit comments

Comments
 (0)