feat(training): launch canonical performance recipes (#4620)

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Author: yaoyu-33

scripts/performance/README.md

@@ -2,7 +2,16 @@
 
 ## NOTE: This directory will change a lot over the coming weeks
 
-- Scripts defined in `scripts/performance` are recipes optimized for performance. These scripts can launch pre-training experiments on Slurm based clusters.
+New runs of exact exported flat recipes should use `scripts/training/train.sh --recipe <function_name>`. The launcher
+discovers text pretraining, text SFT/PEFT, Qwen-VL pretraining, and Wan pretraining recipes automatically and selects
+their forward step. This directory remains the compatibility path for selector-based invocation, dataset replacement,
+topology resizing, and specialized benchmark controls. The training launcher preserves total GPU-count validation,
+the recipe process environment, and mock-data defaults for text SFT/PEFT; it does not inject offline defaults. The
+performance compatibility launcher continues to own its benchmark offline environment.
+Cluster-specific CPU/NUMA binding, Slurm segment sizing, NCCL fabric settings, and `srun` arguments remain user
+supplied.
+
+- Scripts defined in `scripts/performance` launch performance-optimized experiments on Slurm-based clusters.
 
 ## Performance recipe configs
 
@@ -11,7 +20,7 @@ launcher resolves recipes from that package by model, task, GPU count, GPU type,
 and config variant.
 
 `setup_experiment.py` launches `bootstrap.py` on each rank. The bootstrap resolves and
-applies recipe-owned process settings before importing Torch, then replaces itself with
+applies recipe-owned process settings before importing the training loop, then replaces itself with
 either `run_script.py` for flat performance recipes or `run_recipe.py` for model recipes.
 Each training entrypoint therefore executes only once.
 

scripts/training/README.md

@@ -6,21 +6,22 @@ Megatron Bridge training provides a small public Slurm launcher:
 ./scripts/training/train.sh [launch options] [runner options] [KEY=VALUE overrides]
 ```
 
-`train.sh` invokes `setup_experiment.py` on a Slurm login node. Library recipes run through `run_recipe.py`; an exact
-flat performance recipe name is translated to the selector interface already supported by `run_script.py`. The setup
-layer owns resources, the container, explicitly forwarded environment variables, and explicit mounts. Without an
-active virtual environment, the shell entry point creates an isolated `nemo-run` environment rather than resolving
-the full GPU training dependency set on the login node.
+`train.sh` invokes `setup_experiment.py` on a Slurm login node and submits `run_recipe.py` directly through Slurm. The
+setup layer only owns resources, the container, explicitly forwarded environment variables, and explicit mounts.
+Recipe selection, dataset construction, and ConfigContainer overrides are resolved inside the training environment.
+Without an active virtual environment, the shell entry point creates an isolated `nemo-run` environment rather than
+resolving the full GPU training dependency set on the login node.
 
 `launch_with_nemo_run.py` and `launch_with_sbatch.sh` remain available for their existing specialized workflows; `train.sh` is the compact recipe-oriented path.
 
 ## Selection rules
 
-Choose exactly one of a complete recipe or a model selector. `--recipe` and `--model` are mutually exclusive. The mode
-is inferred from a conventional complete recipe name; pass one of `--mode pretrain`, `--mode sft`, `--mode lora`, or
-`--mode dora` when using `--model` or when the name does not encode it.
+Choose exactly one of a complete recipe or a model selector. `--recipe` and `--model` are mutually exclusive. A complete
+recipe is discovered automatically from its exported function name, whether it is a library or benchmark recipe;
+there is no separate source flag. A model selector requires one of `--mode pretrain`, `--mode sft`, `--mode lora`, or
+`--mode dora`; a conventional complete recipe name infers its mode when `--mode` is omitted.
 
-### Complete recipe
+### Library recipe
 
 A complete recipe already identifies the model and default training configuration, so do not also pass `--model`:
 
@@ -48,38 +49,61 @@ requested adapter scheme.
     --mode pretrain --dataset mock
 ```
 
-The default forward step is `llm_step`. Pass `--step-func NAME` explicitly for a recipe that needs another registered
-forward step. Common training, sequence-length, parallelism, optimization, and checkpoint fields also have convenience
-flags such as `-ms`/`--max_steps`, `-sl`/`--seq_length`, `-tp`/`--tensor_model_parallel_size`, and `--save_dir`.
-Use trailing `KEY=VALUE` overrides for every other `ConfigContainer` field.
+### Benchmark recipe
 
-### Performance recipe
+The training launcher can run exact exported recipes from `src/megatron/bridge/perf_recipes`, including text
+pretraining, text SFT/PEFT, Qwen-VL pretraining, and Wan pretraining. The total allocation must match the GPU count
+encoded by the recipe name; the user selects the node shape, and the Slurm partition must provide the requested
+hardware:
 
-Pass the complete exported performance recipe name through the same `--recipe` option; recipe type is inferred:
+`benchmark` is the unified runner's user-facing term. The existing `perf_recipes` package and `scripts/performance/`
+compatibility paths retain their legacy names.
 
 ```bash
 ./scripts/training/train.sh \
     --nodes 2 --gpus-per-node 8 \
     --account ACCOUNT --partition PARTITION \
-    --container-image /path/to/container.sqsh \
-    --env HF_TOKEN \
-    --recipe qwen3_30b_a3b_pretrain_16gpu_h100_bf16_config
+    --container-image IMAGE \
+    --recipe qwen3_30b_a3b_pretrain_16gpu_h100_bf16_config \
+    --mode pretrain
 ```
 
-The GPU count encoded in the recipe name must equal `--nodes × --gpus-per-node`; the user-provided
-`--gpus-per-node` determines the node shape. This compact route supports exact exported recipe names only;
-model-family selectors and advanced performance-launcher controls remain on `scripts/performance` during migration.
-The launcher discovers the exact public export from source and derives the existing runner selectors from its name;
-the performance parser and runner remain unchanged. Use an image containing the same Bridge revision and recipe
-export. Two SFT/PEFT names exported by both recipe packages retain their existing library route. Forward `HF_TOKEN`
-with `--env HF_TOKEN` when the selected recipe needs online Hugging Face access.
+Benchmark recipes provide canonical defaults for their dataset, parallelism topology, batch sizes, sequence length,
+precision, dispatcher, CUDA-graph settings, and process environment. Trailing `KEY=VALUE` overrides are applied to their
+`ConfigContainer` in the same way as library recipes; an overridden run no longer represents the canonical benchmark
+configuration. Benchmark recipes retain their selected dataset type, so `--dataset` is not supported on this path.
+Recipe environment defaults are installed before the launcher enters the training stack; values explicitly set by the
+shell or Slurm environment retain precedence.
+
+The launcher does not infer offline mode, cluster-specific CPU/NUMA binding, Slurm segment sizing, or NCCL fabric
+settings from the recipe name. Supply those deployment settings explicitly through the target cluster's launcher
+configuration, repeated `--srun-arg` options, or exported values forwarded with `--env NAME`.
+
+The runner selects the registered text, multimodal, audio, omni, or diffusion forward step from the recipe identity,
+regardless of which recipe package exports it. The compatibility launcher at
+`scripts/performance/setup_experiment.py` remains available for selector-based invocation, dataset replacement, and
+specialized legacy controls that are not part of the compact training CLI.
+
+Text SFT/PEFT benchmark recipes retain the flat runner's mock-data default. Qwen-VL and Wan recipes retain their
+model-specific dataset configuration. Exported benchmark PEFT recipes are fixed LoRA configs; DoRA remains available
+through configurable library recipes. Explicit benchmark dataset replacement remains on the compatibility launcher.
+
+Five legacy duplicate names resolve to the benchmark definition; their library workloads remain available through
+the corresponding generic recipe aliases. New recipe names should be unique across both packages.
+
+Text recipes default to `llm_step`; all recipes infer their modality-specific forward step from the same registry.
+Pass `--step-func NAME` to override that selection. Common training,
+sequence-length, parallelism, optimization, and checkpoint fields also have convenience flags such as
+`-ms`/`--max_steps`, `-sl`/`--seq_length`, `-tp`/`--tensor_model_parallel_size`, and `--save_dir`. Use trailing
+`KEY=VALUE` overrides for every other `ConfigContainer` field.
 
 ## Dataset selection
 
-`--dataset` accepts source selectors and named dataset presets rather than internal dataset config class names.
+For library recipes, `--dataset` accepts source selectors and named dataset presets rather than internal dataset config class names.
 Each name selects a `DatasetConfig` preset; the config type selects the existing runtime builder. Trailing
 `dataset.*` overrides are applied directly after preset selection. Use typed fields for source, preprocessing, packing,
 and loader settings; use `dataset.dataset_kwargs={...}` only for extra options consumed by a dataset implementation.
+Benchmark recipes retain their recipe-owned dataset and reject `--dataset`.
 
 | Value | Kind | Mode | Behavior |
 |---|---|---|---|
@@ -139,7 +163,8 @@ shared mounted storage.
 
 The hosted VLM names use the existing Hugging Face dataset adapters and retain the processor and in-batch packing
 settings from the selected VLM recipe. `raven`, `rdr`, and `llava-video-178k` derive deterministic 95/5 train and
-validation slices; `cord-v2` and `medpix` use their published validation splits. Pass a VLM forward step explicitly.
+validation slices; `cord-v2` and `medpix` use their published validation splits. The runner selects the registered VLM
+forward step from the recipe name; pass `--step-func` only to override it.
 
 For local JSON/JSONL, select `local-vlm` and set
 `dataset.source.load_kwargs.data_files.train=/path/to/train.jsonl`. Optional split overrides are
@@ -179,9 +204,9 @@ directory:
 
 ## Overrides
 
-Every `ConfigContainer` field can be set with trailing `KEY=VALUE` arguments. Common fields also accept the convenience
-flags listed by `run_recipe.py --help`. For example, batch sizes and training duration belong under `train`, not
-`model`:
+Every selected recipe's `ConfigContainer` fields can be set with trailing `KEY=VALUE` arguments. Common fields also
+accept the convenience flags listed by `run_recipe.py --help`. For example, batch sizes and training duration belong
+under `train`, not `model`:
 
 ```bash
 ./scripts/training/train.sh \
@@ -199,6 +224,9 @@ Precedence is recipe defaults, the selected dataset configuration, common conven
 `ConfigContainer` overrides. A trailing override therefore wins when it targets the same field as a convenience
 argument.
 
+Overrides take a benchmark recipe outside its canonical configuration. Use
+`scripts/performance/setup_experiment.py` for specialized workflows that the unified launcher has not migrated yet.
+
 ## Slurm and containers
 
 Required Slurm arguments are:
@@ -215,10 +243,8 @@ Set `CONTAINER_IMAGE` to avoid repeating `--container-image`. On clusters that a
 forwarded implicitly. Export credentials in the launcher environment, then repeat `--env NAME` to forward names without
 materializing their values in the generated sbatch script. Repeat `--mount HOST` for the same host and container path, or use
 `--mount HOST:CONTAINER` when the paths differ. Mount every dataset, checkpoint, cache, and output path the job needs.
-
-The launcher adds no cluster-specific `srun` flags. Repeat
-`--srun-arg=ARG` for every flag required by the target cluster. For example,
-a Pyxis/Enroot cluster may use:
+The launcher adds no cluster-specific `srun` flags by default. Repeat `--srun-arg=ARG` for every flag required by the
+target cluster. For example, a Pyxis/Enroot cluster may use:
 
 ```bash
 --srun-arg=--mpi=pmix \
@@ -228,8 +254,9 @@ a Pyxis/Enroot cluster may use:
 
 The `=` form is required when `ARG` begins with `-`.
 
-Common NCCL, Transformer Engine, tokenizer, and allocator environment defaults are applied to both library and
-performance recipes. Explicitly inherited environment names take precedence over these defaults.
+The compact launcher does not add rank-command prefixes or extra sbatch parameters such as `segment`. Configure those
+through the target cluster integration, or use `scripts/performance/setup_experiment.py` when its compatibility NUMA
+and segment policies are required.
 
 The launcher submits the experiment in detached mode and returns after Slurm accepts the job. Inspect its state and
 logs with the cluster's normal `squeue`, `sacct`, and log-file workflow.
@@ -244,6 +271,10 @@ uv run python scripts/training/run_recipe.py \
     --dry-run logger.save_config_filepath=/tmp/config.yaml
 ```
 
+For a benchmark dry run, use the complete flat recipe name and omit `--dataset`. The rank-local dry run discovers the
+recipe and validates the final config against the total GPU count encoded in that name without
+requiring a live allocation; the submission dry run additionally validates `--nodes` and `--gpus-per-node`.
+
 ## Rank-local entry point
 
 `run_recipe.py` remains available for existing distributed environments that already own process launch:
@@ -257,4 +288,5 @@ uv run python -m torch.distributed.run --nproc_per_node=2 \
     model.sequence_parallel=true
 ```
 
-This entry point loads library recipes for functional pretraining, SFT, LoRA, and DoRA.
+This entry point loads library recipes for pretraining, SFT, LoRA, and DoRA, and all exact exported
+benchmark recipes when selected explicitly.