NVIDIA-NeMo
diff --git a/‎docs/guides/audio/qwen3-omni-asr.md‎
Lines changed: 61 additions & 4 deletions b/‎docs/guides/audio/qwen3-omni-asr.md‎
Lines changed: 61 additions & 4 deletions
diff --git a/‎docs/guides/dataset-overview.md‎
Lines changed: 1 addition & 1 deletion b/‎docs/guides/dataset-overview.md‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎examples/audio_finetune/qwen2_5_omni_asr/ami_sft_3b.yaml‎
Lines changed: 2 additions & 2 deletions b/‎examples/audio_finetune/qwen2_5_omni_asr/ami_sft_3b.yaml‎
Lines changed: 2 additions & 2 deletions
diff --git a/‎examples/audio_finetune/qwen2_5_omni_asr/ami_sft_7b.yaml‎
Lines changed: 2 additions & 2 deletions b/‎examples/audio_finetune/qwen2_5_omni_asr/ami_sft_7b.yaml‎
Lines changed: 2 additions & 2 deletions
diff --git a/‎examples/audio_finetune/qwen2_5_omni_asr/multi_en_sft_3b.yaml‎
Lines changed: 125 additions & 0 deletions b/‎examples/audio_finetune/qwen2_5_omni_asr/multi_en_sft_3b.yaml‎
Lines changed: 125 additions & 0 deletions
diff --git a/‎examples/audio_finetune/qwen3_omni_asr/ami_sft.yaml‎
Lines changed: 2 additions & 2 deletions b/‎examples/audio_finetune/qwen3_omni_asr/ami_sft.yaml‎
Lines changed: 2 additions & 2 deletions
@@ -20,7 +20,7 @@ The workflow has two stages:
 
 ### Built-In Builder: `make_hf_audio_asr_dataset`
 
-`nemo_automodel.components.datasets.vlm.datasets.make_hf_audio_asr_dataset`
+`nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset`
 returns a Hugging Face `Dataset` whose `__getitem__` lazily produces a single
 `{"conversation": [...]}` dict suitable for `qwen3_omni_asr_collate_fn`. Key
 design points:
@@ -41,7 +41,7 @@ design points:
   WenetSpeech out of the box; per-dataset overrides go in the recipe YAML.
 
 ```python
-from nemo_automodel.components.datasets.vlm.datasets import (
+from nemo_automodel.components.datasets.audio.datasets import (
     make_hf_audio_asr_dataset,
 )
 
@@ -62,7 +62,7 @@ dataset = make_hf_audio_asr_dataset(
 
 ### Built-In Collate: `qwen3_omni_asr_collate_fn`
 
-`nemo_automodel.components.datasets.vlm.collate_fns.qwen3_omni_asr_collate_fn`
+`nemo_automodel.components.datasets.audio.collate_fns.qwen3_omni_asr_collate_fn`
 batches the lazy samples into model inputs without depending on
 `qwen_omni_utils`:
 
@@ -98,7 +98,7 @@ YAML override snippet for CommonVoice (note `text_column: sentence`):
 
 ```yaml
 dataset:
-  _target_: nemo_automodel.components.datasets.vlm.datasets.make_hf_audio_asr_dataset
+  _target_: nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset
   path_or_dataset: mozilla-foundation/common_voice_18_0
   name: en
   text_column: sentence
@@ -109,6 +109,33 @@ dataset:
 Audio columns are universally named `audio` across these datasets, so the
 default `audio_column="audio"` rarely needs an override.
 
+### Mixture of Datasets: `multi_en`
+
+For a stronger general-purpose English model, train on a **mixture** of public
+ASR corpora rather than a single dataset.
+`nemo_automodel.components.datasets.audio.multi_en` concatenates several HF
+sources into one training set, normalizing each to `{audio, text, source}` with
+per-source transcript cleanup (e.g. stripping GigaSpeech bracket tags such as
+`<COMMA>` / `<SIL>`). The default English composition is ~500k clips:
+
+| Source | HF repo | config / split | clips |
+|---|---|---|---:|
+| AMI IHM | `edinburghcstr/ami` | `ihm` / `train` | 108,502 |
+| Earnings22 | `sanchit-gandhi/earnings22_split` | `train` | 52,006 |
+| VoxPopuli (en) | `facebook/voxpopuli` | `en` / `train` (capped) | 4,000 |
+| GigaSpeech (s) | `speechcolab/gigaspeech` | `s` / `train` | 230,068 |
+| SPGISpeech (S) | `kensho/spgispeech` | `S` / `train` | 77,073 |
+| LibriSpeech | `openslr/librispeech_asr` | `clean` / `train.100` | 28,539 |
+| **Total** | | | **~500,188** |
+
+GigaSpeech and SPGISpeech are gated on the Hub — accept their terms (and allow
+`trust_remote_code` for GigaSpeech) before launching. The source list is fully
+overridable from YAML via `dataset.sources` (pass a trimmed list to drop gated
+corpora), and `dataset.max_audio_duration_seconds` caps clip length to bound
+activation memory. Ready-to-run recipe:
+`examples/audio_finetune/qwen3_omni_asr/multi_en_sft.yaml` (and
+`examples/audio_finetune/qwen2_5_omni_asr/multi_en_sft_3b.yaml` for the 3B).
+
 ---
 
 ## Train
@@ -251,3 +278,33 @@ roughly half:
 |-----------------|-------------------------------------------------------|--------------------|
 | Before training | Base `Qwen/Qwen3-Omni-30B-A3B-Instruct` (zero-shot)   | 15.81%             |
 | After training  | 1 epoch full FT (audio tower trainable)               | **8.31%**          |
+
+## Results: `multi_en` mixture
+
+Training the same model for 3 epochs on the ~500k-clip `multi_en` mixture (see
+[Mixture of Datasets](#mixture-of-datasets-multi_en)) generalizes across all 7
+[open-ASR-leaderboard](https://huggingface.co/datasets/hf-audio/open-asr-leaderboard)
+English test subsets, not just AMI. WER below is Whisper-normalized
+(`EnglishTextNormalizer`), greedy decode, comparing the zero-shot base against
+the `multi_en` fine-tune:
+
+| Subset | N | Base (zero-shot) | `multi_en` FT |
+|---|---:|---:|---:|
+| LibriSpeech test.clean | 2,620 | 1.49 | 1.89 |
+| LibriSpeech test.other | 2,939 | 2.62 | 3.54 |
+| SPGISpeech | 39,341 | 3.12 | **2.11** |
+| VoxPopuli | 1,842 | 7.07 | **6.67** |
+| GigaSpeech | 19,931 | 8.54 | 9.46 |
+| Earnings22 | 2,741 | 9.79 | **8.89** |
+| AMI (IHM) | 12,643 | 11.07 | **8.22** |
+| **Macro avg** | — | 6.24 | **5.83** |
+
+Fine-tuning concentrates its gains on the harder conversational / domain sets
+(AMI −2.85, Earnings22 −0.90, SPGISpeech −1.01, VoxPopuli −0.40), while the
+strong base keeps a small edge on clean read speech (LibriSpeech, GigaSpeech) —
+a hint that the mix can be rebalanced toward those styles. Net macro WER
+improves from 6.24% to **5.83%**.
+
+The same recipe on **Qwen2.5-Omni-3B** (`multi_en_sft_3b.yaml`) shows a much
+larger fine-tuning gain, since the small model's zero-shot baseline is weaker:
+macro WER **8.97% → 6.55%** (−2.42).
@@ -451,7 +451,7 @@ Built-in dataset makers (return lists of `conversation` dicts):
 - **RDR items**: `nemo_automodel.components.datasets.vlm.datasets.make_rdr_dataset` (HF: `quintend/rdr-items`)
 - **CORD-V2 receipts (Consolidated Receipt Dataset for Post-OCR Parsing)**: `nemo_automodel.components.datasets.vlm.datasets.make_cord_v2_dataset` (HF: `naver-clova-ix/cord-v2`)
 - **MedPix-VQA (Medical Pixel Question Answering)**: `nemo_automodel.components.datasets.vlm.datasets.make_medpix_dataset`
-- **CommonVoice 17 (CV17) (audio)**: `nemo_automodel.components.datasets.vlm.datasets.make_cv17_dataset`
+- **CommonVoice 17 (CV17) (audio)**: `nemo_automodel.components.datasets.audio.datasets.make_cv17_dataset`
 
 
 Each example follows the conversation schema expected by `apply_chat_template`, e.g.:
 
@@ -71,7 +71,7 @@ loss_fn:
 #   user(  text=user_prompt + audio=<waveform>  )
 #   assistant(  text=<transcript>  )
 dataset:
-  _target_: nemo_automodel.components.datasets.vlm.datasets.make_hf_audio_asr_dataset
+  _target_: nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset
   path_or_dataset: edinburghcstr/ami
   name: ihm
   split: train
@@ -91,7 +91,7 @@ dataloader:
   _target_: torchdata.stateful_dataloader.StatefulDataLoader
   num_workers: 2
   collate_fn:
-    _target_: nemo_automodel.components.datasets.vlm.collate_fns.qwen2_5_omni_asr_collate_fn
+    _target_: nemo_automodel.components.datasets.audio.collate_fns.qwen2_5_omni_asr_collate_fn
 
 optimizer:
   _target_: torch.optim.AdamW
 
@@ -74,7 +74,7 @@ loss_fn:
 #   user(  text=user_prompt + audio=<waveform>  )
 #   assistant(  text=<transcript>  )
 dataset:
-  _target_: nemo_automodel.components.datasets.vlm.datasets.make_hf_audio_asr_dataset
+  _target_: nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset
   path_or_dataset: edinburghcstr/ami
   name: ihm
   split: train
@@ -94,7 +94,7 @@ dataloader:
   _target_: torchdata.stateful_dataloader.StatefulDataLoader
   num_workers: 2
   collate_fn:
-    _target_: nemo_automodel.components.datasets.vlm.collate_fns.qwen2_5_omni_asr_collate_fn
+    _target_: nemo_automodel.components.datasets.audio.collate_fns.qwen2_5_omni_asr_collate_fn
 
 optimizer:
   _target_: torch.optim.AdamW
 
@@ -0,0 +1,125 @@
+# Copyright (c) 2026, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Full-FT English ASR of Qwen/Qwen2.5-Omni-3B on a six-corpus English mix
+# (AMI + earnings22 + voxpopuli_en + gigaspeech + spgispeech + librispeech).
+#
+# NOTE: gigaspeech and spgispeech are gated on the Hugging Face Hub. Accept
+# their dataset terms (and allow trust_remote_code for gigaspeech) before
+# launching, or trim the mix by passing a `dataset.sources` subset.
+
+recipe: FinetuneRecipeForVLM
+
+step_scheduler:
+  global_batch_size: 64
+  local_batch_size: 4         # long clips inflate audio activations; 4 keeps headroom on 80GB
+  ckpt_every_steps: 4000
+  num_epochs: 3
+  max_steps: 30000            # safety cap (~3 epochs of the ~500k-sample mix at GBS 64)
+
+dist_env:
+  backend: nccl
+  # Generous timeout: the first run materializes + caches the ~500k-sample mix
+  # (per-rank, serialized by the HF cache lock). Pre-warm the cache once (single
+  # process) so subsequent launches build in seconds. See the guide.
+  timeout_minutes: 60
+
+rng:
+  _target_: nemo_automodel.components.training.rng.StatefulRNG
+  seed: 1111
+  ranked: true
+
+model:
+  _target_: nemo_automodel.NeMoAutoModelForMultimodalLM.from_pretrained
+  pretrained_model_name_or_path: Qwen/Qwen2.5-Omni-3B
+  backend:
+    _target_: nemo_automodel.components.models.common.BackendConfig
+    attn: sdpa
+    linear: te
+    rms_norm: torch_fp32
+    rope_fusion: false
+    enable_hf_state_dict_adapter: true
+
+checkpoint:
+  enabled: true
+  checkpoint_dir: result/checkpoints/qwen2_5_omni_asr_multi_en
+  model_save_format: safetensors
+  save_consolidated: true
+
+distributed:
+  strategy: fsdp2
+  tp_size: 1
+  cp_size: 1
+  pp_size: 1
+  sequence_parallel: false
+  # GBS 64 / (LBS 4 x 8 GPUs) = 2 grad-accum microbatches. The bundled torch 2.11
+  # FSDP2 build crashes in the deferred-grad-sync path on non-final microbatches
+  # (AttributeError: 'FSDPParam' has no attribute '_unsharded_param'), so sync
+  # gradients every microbatch instead of deferring.
+  defer_fsdp_grad_sync: false
+
+# Full fine-tune: audio tower + language model trainable; vision tower frozen
+# (we never feed images on these English ASR corpora).
+freeze_config:
+  freeze_embeddings: false
+  freeze_vision_tower: true
+  freeze_audio_tower: false
+  freeze_language_model: false
+
+loss_fn:
+  _target_: nemo_automodel.components.loss.masked_ce.MaskedCrossEntropy
+
+# Conversation shape (per sample):
+#   user(  text=user_prompt + audio=<waveform>  )
+#   assistant(  text=<transcript>  )
+#
+# The builder owns the six-source list, the prototype-faithful transcript
+# normalization, and the torchcodec-free soundfile decode; it needs no
+# path_or_dataset / name here.
+dataset:
+  _target_: nemo_automodel.components.datasets.audio.multi_en.make_multi_en_asr_dataset
+  sampling_rate: 16000
+  system_prompt: null
+  user_prompt: >-
+    Transcribe the following English audio verbatim. Output only the raw
+    transcript text with no leading or trailing commentary.
+  # Qwen-Omni's Whisper feature extractor has a known off-by-one between
+  # ``input_features`` and ``feature_attention_mask`` on sub-second clips.
+  # Filtering at 1.0 s costs only the header-only sf.info probe per sample.
+  min_audio_duration_seconds: 1.0
+  # Cap clip length so long-audio batches don't OOM a rank (the audio encoder's
+  # input_features scale with duration). Header-only probe; no decode.
+  max_audio_duration_seconds: 30.0
+
+dataloader:
+  _target_: torchdata.stateful_dataloader.StatefulDataLoader
+  num_workers: 2
+  collate_fn:
+    _target_: nemo_automodel.components.datasets.audio.collate_fns.qwen2_5_omni_asr_collate_fn
+
+optimizer:
+  _target_: torch.optim.AdamW
+  betas: [0.9, 0.95]
+  eps: 1e-8
+  lr: 2.0e-5
+  weight_decay: 0.0
+
+wandb:
+  project: ${oc.env:WANDB_PROJECT,qwen2_5-omni-asr-multi-en}
+  name: ${oc.env:WANDB_NAME,qwen2_5_omni_3b_asr_multi_en_fullft}
+  dir: ${oc.env:WANDB_DIR,result/wandb}
+
+ci:
+  recipe_owner: yuekaizhang
+  time: "00:30:00"
@@ -101,7 +101,7 @@ loss_fn:
 #   assistant(  text=<transcript>  )
 # No system turn (system_prompt: null).
 dataset:
-  _target_: nemo_automodel.components.datasets.vlm.datasets.make_hf_audio_asr_dataset
+  _target_: nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset
   path_or_dataset: edinburghcstr/ami
   name: ihm
   split: train
@@ -118,7 +118,7 @@ dataloader:
   _target_: torchdata.stateful_dataloader.StatefulDataLoader
   num_workers: 2
   collate_fn:
-    _target_: nemo_automodel.components.datasets.vlm.collate_fns.qwen3_omni_asr_collate_fn
+    _target_: nemo_automodel.components.datasets.audio.collate_fns.qwen3_omni_asr_collate_fn
 
 optimizer:
   _target_: torch.optim.AdamW