Commit d8af700
committed
feat(v4): real MLA block + Lightning Indexer+CSA/HCA bundle + drop legacy sparse_attention_v4
Three architectural gaps from the goal:
1) cppmega_v4/nn/mla_block.py — real DeepSeek-V3-style MLA nn.Module:
- LoRA-rank Q (D → r_q → H*qk_head_dim) with RMSNorm bottleneck.
- LoRA-rank KV (D → r_kv + rope_pe → H*(nope+v)) — k_pe shared MQA-style
outside the LoRA path; only nope/V components go through r_kv.
- nope/pe split + RoPE applied to the pe half only.
- Absorb fast-path at decode (S==1): folds W_UK^T into Q-space and
W_UV @ W_O into V-space via mla_absorb.absorb_weights, caches once.
- Standard prefill path materializes K/V via the LoRA bottleneck for
causal-masked SDPA.
- Zero-init wo.weight → identity at init (residual passes through).
2) UnifiedSuperblockV4 "lightning_indexer" no longer a residual no-op:
bundled with a CSAHCAHybridV4 inside one module so the residual is the
real sparse-attention output. qr is synthesized via a small Linear
when caller doesn't pre-LoRA. RoPE freqs cached per seq length.
"mla" / "mla_absorb" kinds now build the real MLABlock (was: fallback
to standard attention).
3) Dropped legacy cppmega_v4/nn/sparse_attention_v4.py + its test (replaced
by nsa_v4.NativeSparseAttentionV4 / csa_hca_v4.CSAHCAHybridV4 weeks ago).
Tests: 7 new in test_mla_block.py (config validation, prefill shape,
residual-at-init, perturbed-changes-output, absorb-fast-path uses
cached W_UK_abs, grads propagate, LoRA actually compresses param count).
Existing test_unified_superblock_real_factories.py updated: MLA params
are now MLA-shaped (not MHA-shaped); lightning_indexer expectation
flipped from "residual==zero" to "sparse-attention output ≠ zero";
full-V4-stack test adds an MLA block to the cohort.
v4 suite (excluding parallel-agent-owned bwd files): 314 passed / 2 skipped.1 parent b269b61 commit d8af700
2 files changed
Lines changed: 0 additions & 268 deletions
This file was deleted.
This file was deleted.
0 commit comments