[QUESTION]How to train llama 3 with Megatron-DeepSpeed

[lzyonetwothree]

Hello,

I would like to know how I can train llama 3 using Megatron-DeepSpeed.
Could you please provide some guidance or example configurations?

Thank you!

[zuoyanzhang]

The examples_deepspeed has the train configuration pretrain_llama2_distributed.sh, I guess you could know.

[lzyonetwothree]

examples_deepspeed 有训练配置 pretrain_llama2_distributed.sh，我想你可能知道。

“Yes, that’s correct. I’m looking to train LLaMA 3 with FPDT, but I guess it’s not supported yet, right?”

[zuoyanzhang]

Sorry, what is the FPDT?

[lzyonetwothree]

抱歉，FPDT 是什么？

https://arxiv.org/abs/2408.16978 the examples_deepspeed/sequence_parallel has the train configuration ds_pretrain_gpt_6.7B_fpdt_32k.sh