PoR: Deepseek v4 Roadmap

[weijiac0619]

End-to-end training support for DeepSeek-V4-Flash (~285B MoE) and DeepSeek-V4-Pro (~600B MoE) in Megatron-Bridge, covering H100 (Hopper) and GB200 (Blackwell) clusters.

Status: ✅ validated · 🚧 pending validation · 📋 planned · ❌ known broken

1. Core Training

Feature	H100 (Hopper)	GB200 (Blackwell)	Notes
Basic training (PP + EP + MTP + MLA)	✅	✅	PR #3562
Fused RoPE (apply_rope_fusion=true)	✅	✅	Fixed in #4271
DSA Indexer Loss (dsa_indexer_loss_coeff > 0)	🚧	🚧	Previously hit ICE when enabled. Validation in progress

2. Fusion & Performance

Feature	H100	GB200	Notes
Fused mHC (use_fused_mhc=true)	📋	🚧	Now MCore supports sm_100 kernel (Blackwell-only). GB200 validation in progress
Fused DSA (via cudnn-frontend 1.24.0)	📋	🚧	H100 blocked by SM90 assertion in transformer_config.py. GB200 not yet tested in Bridge. cudnn-frontend PR#263 merged Jun 8
GroupedGEMM + ClampedSwiGLU	🚧	🚧	MCore merged (#5130). Bridge verification needed
Packed sequence (THD format)	🚧	🚧	MCore merged (#5011). Bridge verification needed
HybridEP dispatcher (THD)	🚧	🚧	MCore merged (#4816). Bridge verification needed
CUDA graphs	🚧	🚧	Per-layer MCore merged (#4359); chunk-wise whole-block in progress (#5258). Requires packed seq + HybridEP. Bridge verification needed

3. Parallelism

Feature	H100	GB200	Notes
PP + EP	✅	✅
Context Parallel (CP)	📋	🚧	MCore merged (#5087), depends on THD stack
mHC + EP overlap	📋	📋	MCore in progress (#5471)

4. Optimizer

Feature	H100	GB200	Notes
Adam	✅	✅
Muon	✅	✅	Validated. MCore integration mostly merged
Muon FP8 param gather	📋	📋	MCore in progress (#5470)

6. Recipes

Recipe	Smoke	Short convergence run	Long convergence run	Notes
Flash H100	✅	📋	📋	PP=16 EP=16, unfused mHC, full recompute
Flash GB200, Adam	✅	✅	📋	PP=4 EP=8, MXFP8, fused DSA. Loss curve healthy
Flash GB200, Muon	✅	✅	📋	PP=4 EP=8, BF16, fused DSA. Loss curve healthy
Pro H100	📋	📋	📋	PP=16 EP=8. Import/export/inference verified
Pro GB200	📋	📋	📋	PP=4 EP=8. Import/export/inference verified