Guidance on adapting a fine-tuning recipe into a pretraining recipe

[wswaq]

Hi NeMo AutoModel team,

Thanks a lot for this excellent repository. I think the project is very well designed, and the provided recipes are very useful.

I noticed that there are already many fine-tuning recipes in the repo, while the number of pretraining recipes seems relatively limited. My impression is that fine-tuning and pretraining recipes may share a lot of common structure, so I am wondering how to properly adapt a fine-tuning recipe into a pretraining recipe.

Could you please share some suggestions on what parts should typically be changed?

For instance, I assume the modifications may involve things like:

• training data format and pipeline
• objective / loss configuration
• optimizer and scheduler
• model initialization
• masking / packing strategy

I would really appreciate any guidance, recommendations, or pointers to relevant examples.

Thank you very much for your time and for maintaining this great repo.

[akoumpa]

Hi @wswaq, thanks for the kind words and sorry for the slow reply!

Good news: in Automodel, SFT and pretraining share the same recipe entry point — TrainFinetuneRecipeForNextTokenPrediction (nemo_automodel/recipes/llm/train_ft.py). Both examples/llm_finetune/finetune.py and examples/llm_pretrain/pretrain.py instantiate the same class. What changes between the two is really just the YAML config.

For concrete references, examples/llm_pretrain/ has a few you can diff against the finetune configs:

• nanogpt_pretrain.yaml — minimal from-scratch GPT-2 style
• llama3_70b_pretrain.yaml — continued-pretraining-style (loads HF weights)
• deepseekv3_pretrain.yaml, megatron_pretrain_gpt2.yaml, megatron_pretrain_moonlight_16b_te_slurm.yaml — larger-scale examples

Addressing your specific points:

• Dataset + pipeline: SFT uses prompt/response datasets (e.g. squad.make_squad_dataset → collater emits a loss mask over the response tokens). Pretraining uses a raw token stream — see NanogptDataset (nemo_automodel/components/datasets/llm/nanogpt_dataset.py), which reads packed .bin shards into fixed-length sequences. For your own corpus, the easiest path is to tokenize into .bin shards (modded-nanogpt format) and reuse NanogptDataset, or write a small IterableDataset that yields {input_ids, labels} with labels = input_ids.clone() (no masking).
• Loss / objective: same MaskedCrossEntropy — the difference is just which tokens are masked out. Pretraining masks nothing (all tokens contribute).
• Model initialization: `from_pretrained` for continued pretraining, or `from_config` / a `build_*_model` shorthand for from-scratch. See the commented "Option 2" block at the top of the pretrain configs.
• Optimizer / scheduler: same YAML structure. For from-scratch you typically want higher LR + longer warmup + cosine decay (see the `lr_scheduler` block). For continued pretraining, closer to finetune settings.
• Masking / packing: `seq_len` on the dataset controls packing. No instruction-tuning-style attention masking; sequences are usually packed back-to-back with BOS/EOS separators.

The fastest way to get going: pick the pretrain config closest to your model, then swap in your dataset block. Happy to answer more specific questions once you try it.