[tracking] Long-context (128k) extension example for Megatron-Bridge

[cuichenx]

Summary

Umbrella tracker for the long-context extension example in Megatron-Bridge, targeted for the 26.08 release. The goal: a runnable, documented example that takes an existing checkpoint and extends its context length to 128k tokens in Bridge's training loop, with CP + sequence-packing enabled and convergence verified.

Carried from 26.06.

Target models

• Qwen3-30B — P0
• Moonlight-16B — P1
• GLM — TBD (decide whether to include in this example or split). Confirm which GLM variant once Qwen3-30B path is working.

Reference dataset

• togethercomputer/CoderForge-Preview — long-context coding dataset, suitable for stressing the 128k path (real long-tail sequence lengths beyond synthetic data). Use as the primary convergence/throughput dataset for the example.

Scope

• Recipe(s) under src/megatron/bridge/recipes/ for Qwen3-30B context extension @ 128k
• Recipe(s) for Moonlight-16B @ 128k
• Example launch script(s) under examples/ (Slurm + torchrun variants where applicable)
• CP + sequence-packing configuration verified end-to-end (seq_length=131072, packed THD path, calculate_per_token_loss=True, ddp.average_in_collective=False, micro_batch_size=1)
• Data preprocessing path for togethercomputer/CoderForge-Preview (tokenization + packing); document the workflow
• Short convergence run + loss curve published (W&B link in this issue)
• Memory / throughput numbers for the documented parallelism plan (TP/PP/CP/EP) on target HW
• Docs: tutorial page describing the workflow, parallelism choices, and known caveats
• Functional smoke test (CI-affordable: small toy data, ≤ 2 GPUs)

Related / coordination

• Existing broader issue: #1926 "CP + Seq packing in LLM & VLM" — currently a stub. This new tracker is the concrete 26.08 deliverable; decide whether CP + Seq packing in LLM & VLM #1926 should fold into this issue or stay as the umbrella for the broader CP+packing surface.
• Dynamic CP (Hybrid CP) — #4586. If Dynamic CP lands in time, the long-context example should adopt it (rather than static CP) for the long-tail dataset distribution. Coordinate scope.
• Qwen3-VL 128k SFT dynamic-padding bug — #4143. Not blocking (this issue is LLM, not VLM), but the shape contracts surfaced there are worth checking for the LLM path too.
• SFT dataset unification — #4596. If the new upstream SFTDataset lands first, this example should use it.

References

• 26.08 PoR tracker: track this issue from the Megatron Bridge 26.08 PoR tracker (MB-585) §6 Examples & Tutorials.
• PIC: Jonas team / Rayen