Qwen3-Asr GSPO

[xyqsgdog-ctrl]

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

How can I using ms-swift to train Qwen3-Asr with GSPO?

How to Reproduce / 如何复现

How can I using ms-swift to train Qwen3-Asr with GSPO?

Additional Information / 补充信息

No response

[hjh0119]

You can base this on the existing GRPO example, replacing the model and dataset. For the dataset format, refer to https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html

If you encounter any errors, let me know.

[xyqsgdog-ctrl]

You can base this on the existing GRPO example, replacing the model and dataset. For the dataset format, refer to https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html

If you encounter any errors, let me know.

my train.sh

`#!/usr/bin/env bash
set -euo pipefail

CUDA_VISIBLE_DEVICES=0,1
WANDB_API_KEY=your_wandb_key
NPROC_PER_NODE=2
swift rlhf
--rlhf_type grpo
--model /nas/xyq/qwen3-asr-finetune/ckpts/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train_20260422_173123/checkpoint-14000-wer0.3439-infer
--model_type qwen3_asr
--external_plugins examples/train/grpo/plugin/plugin.py
--reward_funcs qwen3_asr_orm format
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port 8001
--tuner_type full
--torch_dtype bfloat16
--dataset "/nas/xyq/qwen3-asr-finetune/data/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train.jsonl"
--val_dataset "/nas/xyq/qwen3-asr-finetune/data/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_test3000.jsonl"
--load_from_cache_file true
--importance_sampling_level sequence
--max_length 2048
--max_completion_length 1024
--num_train_epochs 1
--per_device_train_batch_size 8
--per_device_eval_batch_size 8
--learning_rate 5e-7
--gradient_accumulation_steps 8
--eval_steps 500
--save_steps 100
--save_total_limit 20
--logging_steps 1
--output_dir output/GRPO_Qwen3_Asr
--warmup_ratio 0.01
--dataloader_num_workers 4
--num_generations 8
--temperature 1.0
--system ''
--deepspeed zero3
--log_completions true
--report_to wandb
--beta 0.001
--num_iterations 1`

my env
(ms-swift) ➜ ms-swift git:(main) ✗ pip list | grep transformers transformers 4.57.6 transformers-stream-generator 0.0.5

Error:
(ms-swift) ➜ ms-swift git:(main) ✗ bash qwen3_Asr_GSPO_train.sh run sh: /nas/syh/miniconda3/envs/ms-swift/bin/python3.11 -m torch.distributed.run --nproc_per_node 2 /nas/xyq/ms-swift/swift/cli/rlhf.py --rlhf_type grpo --model /nas/xyq/qwen3-asr-finetune/ckpts/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train_20260422_173123/checkpoint-14000-wer0.3439-infer --model_type qwen3_asr --external_plugins examples/train/grpo/plugin/plugin.py --reward_funcs qwen3_asr_orm format --use_vllm true --vllm_mode server --vllm_server_host 127.0.0.1 --vllm_server_port 8001 --tuner_type full --torch_dtype bfloat16 --dataset /nas/xyq/qwen3-asr-finetune/data/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train.jsonl --val_dataset /nas/xyq/qwen3-asr-finetune/data/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_test3000.jsonl --load_from_cache_file true --importance_sampling_level sequence --max_length 2048 --max_completion_length 1024 --num_train_epochs 1 --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --learning_rate 5e-7 --gradient_accumulation_steps 8 --eval_steps 500 --save_steps 100 --save_total_limit 20 --logging_steps 1 --output_dir output/GRPO_Qwen3_Asr --warmup_ratio 0.01 --dataloader_num_workers 4 --num_generations 8 --temperature 1.0 --system --deepspeed zero3 --log_completions true --report_to wandb --beta 0.001 --num_iterations 1`

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

[INFO:swift] Successfully registered /nas/xyq/ms-swift/swift/dataset/data/dataset_info.json.
[INFO:swift] Setting args.remove_unused_columns: False
[INFO:swift] Start connecting to vLLM server
[INFO:swift] Connected to vLLM server
[INFO:swift] Successfully imported external_plugins: ['examples/train/grpo/plugin/plugin.py'].
[INFO:swift] rank: 0, local_rank: 0, world_size: 2, local_world_size: 2
[INFO:swift] Because len(args.val_dataset) > 0, setting split_dataset_ratio: 0.0
[INFO:swift] Setting args.lazy_tokenize: True
[INFO:swift] freeze_parameters: ['thinker.audio_tower', 'thinker.audio_tower.proj1', 'thinker.audio_tower.proj2']
[INFO:swift] Using deepspeed: {'fp16': {'enabled': 'auto', 'loss_scale': 0, 'loss_scale_window': 1000, 'initial_scale_power': 16, 'hysteresis': 2, 'min_loss_scale': 1}, 'bf16': {'enabled': 'auto'}, 'zero_optimization': {'stage': 3, 'offload_optimizer': {'device': 'none', 'pin_memory': True}, 'offload_param': {'device': 'none', 'pin_memory': True}, 'overlap_comm': False, 'contiguous_gradients': True, 'sub_group_size': 1000000000.0, 'reduce_bucket_size': 'auto', 'zero_quantized_weights': False, 'zero_quantized_gradients': False, 'stage3_prefetch_bucket_size': 'auto', 'stage3_param_persistence_threshold': 'auto', 'stage3_max_live_parameters': 1000000000.0, 'stage3_max_reuse_distance': 1000000000.0, 'stage3_gather_16bit_weights_on_model_save': True}, 'gradient_accumulation_steps': 'auto', 'gradient_clipping': 'auto', 'steps_per_print': 2000, 'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'wall_clock_breakdown': False}
[INFO:swift] output_dir: /nas/xyq/ms-swift/output/GRPO_Qwen3_Asr/v1-20260427-152206
[INFO:swift] args: RLHFArguments(
_n_gpu=-1,
acc_strategy=token,
accelerator_config={'dispatch_batches': False},
adafactor=False,
adalora_beta1=0.85,
adalora_beta2=0.85,
adalora_deltaT=1,
adalora_init_r=12,
adalora_orth_reg_weight=0.5,
adalora_target_r=8,
adalora_tfinal=0,
adalora_tinit=0,
adam_beta1=0.9,
adam_beta2=0.95,
adam_epsilon=1e-08,
adapter_act=gelu,
adapter_length=128,
adapters=[],
add_non_thinking_prefix=True,
add_version=True,
advantage_estimator=grpo,
agent_template=None,
aligner_lr=None,
async_generate=False,
attn_impl=None,
auto_find_batch_size=False,
average_tokens_across_devices=True,
batch_eval_metrics=False,
beta=0.001,
bf16=True,
bf16_full_eval=False,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_quant_storage=None,
bnb_4bit_quant_type=nf4,
bnb_4bit_use_double_quant=True,
boft_block_num=0,
boft_block_size=4,
boft_dropout=0.0,
boft_n_butterfly_factor=1,
cached_dataset=[],
cached_val_dataset=[],
callbacks=[],
center_rewards_coefficient=None,
check_model=True,
chord_enable_phi_function=False,
chord_mu_decay_steps=None,
chord_mu_peak=None,
chord_mu_valley=None,
chord_mu_warmup_steps=None,
chord_sft_dataset=[],
chord_sft_per_device_train_batch_size=None,
cliprange=0.2,
cliprange_value=0.2,
columns={},
completion_length_limit_scope=per_round,
cosine_max_len=None,
cosine_max_len_value_correct=0.5,
cosine_max_len_value_wrong=0.0,
cosine_min_len_value_correct=1.0,
cosine_min_len_value_wrong=-0.5,
cpo_alpha=1.0,
create_checkpoint_symlink=False,
custom_dataset_info=[],
custom_register_path=[],
data_seed=42,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
dataset=['/nas/xyq/qwen3-asr-finetune/data/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train.jsonl'],
dataset_num_proc=1,
dataset_shuffle=True,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=18000000,
debug=None,
deepspeed={'fp16': {'enabled': 'auto', 'loss_scale': 0, 'loss_scale_window': 1000, 'initial_scale_power': 16, 'hysteresis': 2, 'min_loss_scale': 1}, 'bf16': {'enabled': 'auto'}, 'zero_optimization': {'stage': 3, 'offload_optimizer': {'device': 'none', 'pin_memory': True}, 'offload_param': {'device': 'none', 'pin_memory': True}, 'overlap_comm': False, 'contiguous_gradients': True, 'sub_group_size': 1000000000.0, 'reduce_bucket_size': 'auto', 'zero_quantized_weights': False, 'zero_quantized_gradients': False, 'stage3_prefetch_bucket_size': 'auto', 'stage3_param_persistence_threshold': 'auto', 'stage3_max_live_parameters': 1000000000.0, 'stage3_max_reuse_distance': 1000000000.0, 'stage3_gather_16bit_weights_on_model_save': True}, 'gradient_accumulation_steps': 'auto', 'gradient_clipping': 'auto', 'steps_per_print': 2000, 'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'wall_clock_breakdown': False},
deepspeed_autotp_size=None,
delta=None,
desirable_weight=1.0,
device_groups=None,
device_map=None,
disable_tqdm=None,
discopop_tau=0.05,
do_eval=False,
do_predict=False,
do_train=False,
download_mode=reuse_dataset_if_exists,
ds3_gather_for_generation=True,
dynamic_sample=False,
early_stop_interval=None,
enable_channel_loss=False,
enable_dft_loss=False,
enable_flattened_weight_sync=True,
enable_thinking=None,
epsilon=0.2,
epsilon_high=None,
eval_accumulation_steps=None,
eval_dataset=[],
eval_dataset_args=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_generation_config=None,
eval_limit=None,
eval_metric=None,
eval_on_start=False,
eval_steps=500.0,
eval_strategy=steps,
eval_use_evalscope=False,
eval_use_gather_object=False,
experts_impl=None,
external_plugins=['examples/train/grpo/plugin/plugin.py'],
extra_eval_args=None,
fourier_n_frequency=2000,
fourier_scaling=300.0,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
freeze_aligner=True,
freeze_llm=False,
freeze_parameters=['thinker.audio_tower', 'thinker.audio_tower.proj1', 'thinker.audio_tower.proj2'],
freeze_parameters_ratio=0.0,
freeze_parameters_regex=None,
freeze_vit=True,
fsdp=[],
fsdp_config=None,
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
galore_cos_threshold=0.4,
galore_gamma_proj=2,
galore_optim_per_parameter=False,
galore_proj_bits=4,
galore_proj_group_size=256,
galore_proj_quant=False,
galore_proj_type=std,
galore_quantization=False,
galore_queue_size=5,
galore_rank=128,
galore_scale=1.0,
galore_target_modules=None,
galore_update_proj_gap=50,
galore_with_embedding=False,
gamma=1.0,
generation_batch_size=None,
generation_config=None,
generation_max_length=None,
generation_num_beams=None,
gkd_logits_topk=None,
gradient_accumulation_steps=8,
gradient_checkpointing=True,
gradient_checkpointing_kwargs=None,
greater_is_better=False,
group_by_length=False,
half_precision_backend=auto,
hqq_axis=None,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=None,
hub_revision=None,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_args_error=False,
ignore_data_skip=False,
importance_sampling_level=sequence,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
init_strategy=None,
init_weights=True,
interleave_prob=None,
jit_mode_eval=False,
kl_coef=0.05,
kl_in_reward=False,
label_names=None,
label_smoothing=0,
label_smoothing_factor=0.0,
lam=0.95,
lazy_tokenize=True,
ld_alpha=None,
learning_rate=5e-07,
length_column_name=length,
liger_kernel_config=None,
lisa_activated_layers=0,
lisa_step_interval=20,
llamapro_num_groups=None,
llamapro_num_new_blocks=4,
lmbda=0.5,
load_args=False,
load_best_model_at_end=False,
load_data_args=False,
load_from_cache_file=True,
local_rank=0,
local_repo_path=None,
local_rollout_forward_batch_size=64,
log_completions=True,
log_entropy=False,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
log_rollout_offpolicy_metrics=False,
logging_dir=/nas/xyq/ms-swift/output/GRPO_Qwen3_Asr/v1-20260427-152206/runs,
logging_first_step=True,
logging_nan_inf_filter=True,
logging_steps=1,
logging_strategy=steps,
logprobs=False,
lora_alpha=32,
lora_bias=none,
lora_dropout=0.05,
lora_dtype=None,
lora_ga_batch_size=2,
lora_ga_direction=ArB2r,
lora_ga_iters=2,
lora_ga_max_length=1024,
lora_ga_scale=stable,
lora_ga_stable_gamma=16,
lora_rank=8,
lorap_lr_ratio=None,
loss_scale=last_round,
loss_type=grpo,
loss_weights=None,
lr_scheduler_kwargs=None,
lr_scheduler_type=cosine,
max_completion_length=1024,
max_epochs=None,
max_grad_norm=1.0,
max_length=2048,
max_memory={},
max_model_len=None,
max_new_tokens=1024,
max_pixels=None,
max_resample_times=3,
max_shard_size=5GB,
max_steps=-1,
max_turns=None,
metric_for_best_model=loss,
missing_eos_penalty=None,
model=/nas/xyq/qwen3-asr-finetune/ckpts/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train_20260422_173123/checkpoint-14000-wer0.3439-infer,
model_author=None,
model_kwargs={},
model_name=None,
model_revision=None,
model_type=qwen3_asr,
modules_to_save=[],
move_model_batches=None,
mp_parameters=,
multi_turn_scheduler=None,
neftune_noise_alpha=None,
new_special_tokens=[],
no_cuda=False,
norm_bbox=None,
num_beams=1,
num_generations=8,
num_generations_eval=None,
num_iterations=1,
num_labels=None,
num_mini_batches=1,
num_ppo_epochs=4,
num_sample_generations=10,
num_train_epochs=1.0,
off_policy_sequence_mask_delta=None,
offload_model=False,
offload_optimizer=False,
offload_teacher_model=False,
optim=adamw_torch_fused,
optim_args=None,
optim_target_modules=None,
optimizer=None,
output_dir=/nas/xyq/ms-swift/output/GRPO_Qwen3_Asr/v1-20260427-152206,
overlong_filter=False,
overwrite_output_dir=False,
packing=False,
packing_length=None,
packing_num_proc=1,
padding_free=False,
padding_side=right,
parallelism_config=None,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
predict_with_generate=False,
prediction_loss_only=False,
problem_type=None,
project=huggingface,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
quant_bits=None,
quant_method=None,
ray_exp_name=None,
ray_scope=last,
real_tau=0.5,
ref_adapters=[],
ref_model=/nas/xyq/qwen3-asr-finetune/ckpts/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_train_20260422_173123/checkpoint-14000-wer0.3439-infer,
ref_model_mixup_alpha=0.6,
ref_model_revision=None,
ref_model_sync_steps=512,
ref_model_type=qwen3_asr,
reft_args=None,
reft_intervention_type=LoreftIntervention,
reft_layer_key=None,
reft_layers=None,
reft_rank=4,
remove_unused_columns=False,
repetition_max_penalty=-1.0,
repetition_n_grams=3,
repetition_penalty=1.0,
report_to=['wandb'],
response_length=1024,
response_prefix=None,
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
resume_only_model=False,
reward_adapters=[],
reward_funcs=['qwen3_asr_orm', 'format'],
reward_model=None,
reward_model_plugin=None,
reward_model_revision=None,
reward_model_type=None,
reward_template=None,
reward_weights=None,
rlhf_type=grpo,
rollout_importance_sampling_mode=None,
rollout_importance_sampling_threshold=2.0,
rope_scaling=None,
router_aux_loss_coef=0.0,
rpo_alpha=None,
run_name=/nas/xyq/ms-swift/output/GRPO_Qwen3_Asr/v1-20260427-152206,
safe_serialization=True,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=100.0,
save_strategy=steps,
save_total_limit=20,
scale_rewards=group,
seed=42,
seq_kd=False,
sequence_parallel_size=1,
sft_alpha=0,
shuffle_buffer_size=1000,
simpo_gamma=1,
skip_memory_metrics=True,
sleep_level=0,
soft_cache_length=None,
soft_max_length=None,
sortish_sampler=False,
split_dataset_ratio=0.0,
steps_per_generation=None,
stop_words=[],
stopping_strategy=first_exhausted,
stream=False,
streaming=False,
strict=False,
structured_outputs_regex=None,
swanlab_email_language=zh,
swanlab_exp_name=None,
swanlab_mode=cloud,
swanlab_notification_method=None,
swanlab_project=ms-swift,
swanlab_receiver_email=None,
swanlab_secret=None,
swanlab_sender_email=None,
swanlab_smtp_port=None,
swanlab_smtp_server=None,
swanlab_token=<SWANLAB_TOKEN>,
swanlab_webhook_url=None,
swanlab_workspace=None,
sync_ref_model=False,
system=,
target_modules=['all-linear'],
target_parameters=None,
target_regex=None,
task_type=causal_lm,
tau_neg=1.05,
tau_pos=1.0,
teacher_adapters=[],
teacher_deepspeed=None,
teacher_model=None,
teacher_model_revision=None,
teacher_model_server=None,
teacher_model_type=None,
temperature=1.0,
template=qwen3_asr,
template_backend=swift,
tf32=None,
top_entropy_quantile=1.0,
top_k=-1,
top_logprobs=None,
top_p=1.0,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_dtype=torch.bfloat16,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
trackio_space_id=trackio,
train_dataloader_shuffle=True,
trainable_parameters=[],
trainable_parameters_regex=None,
truncation_strategy=left,
tuner_backend=peft,
tuner_type=full,
undesirable_weight=1.0,
use_chat_template=True,
use_cpu=False,
use_dora=False,
use_flash_ckpt=False,
use_galore=False,
use_hf=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_logits_to_keep=None,
use_mps_device=False,
use_ray=False,
use_rslora=False,
use_swift_lora=False,
use_vllm=True,
val_dataset=['/nas/xyq/qwen3-asr-finetune/data/audio_train_manifest_0.1-0.9_400w_plus_perfect_plus_batch001_010_enzh_dedup_audiotext_audio_asrtext_only_test3000.jsonl'],
val_dataset_shuffle=False,
vera_d_initial=0.1,
vera_dropout=0.0,
vera_projection_prng_key=0,
vera_rank=256,
vf_coef=0.1,
vit_gradient_checkpointing=False,
vit_lr=None,
vllm_data_parallel_size=1,
vllm_disable_cascade_attn=False,
vllm_disable_custom_all_reduce=True,
vllm_enable_expert_parallel=False,
vllm_enable_lora=False,
vllm_enable_prefix_caching=True,
vllm_enforce_eager=False,
vllm_engine_kwargs={},
vllm_gpu_memory_utilization=0.9,
vllm_limit_mm_per_prompt=None,
vllm_max_lora_rank=16,
vllm_max_model_len=None,
vllm_max_num_seqs=None,
vllm_mm_processor_cache_gb=None,
vllm_mode=server,
vllm_pipeline_parallel_size=1,
vllm_quantization=None,
vllm_reasoning_parser=None,
vllm_server_base_url=None,
vllm_server_group_port=None,
vllm_server_host=['127.0.0.1'],
vllm_server_pass_dataset=False,
vllm_server_port=[8001],
vllm_server_timeout=240.0,
vllm_speculative_config=None,
vllm_tensor_parallel_size=1,
vllm_use_async_engine=None,
wandb_log_unique_prompts=None,
warmup_ratio=0.01,
warmup_steps=0,
weight_decay=0.1,
whiten_rewards=False,
zero_hpz_partition_size=None,
)
[INFO:swift] Global seed set to 42
[rank0]: Traceback (most recent call last):
[rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1360, in from_pretrained
[rank0]: config_class = CONFIG_MAPPING[config_dict["model_type"]]
[rank0]: ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1048, in getitem
[rank0]: raise KeyError(key)
[rank0]: KeyError: 'qwen3_asr'

[rank0]: During handling of the above exception, another exception occurred:

[rank0]: Traceback (most recent call last):
[rank0]: File "/nas/xyq/ms-swift/swift/cli/rlhf.py", line 7, in
[rank0]: rlhf_main()
[rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 246, in rlhf_main
[rank0]: return SwiftRLHF(args).main()
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/sft.py", line 29, in init
[rank0]: self._prepare_model_tokenizer()
[rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 130, in _prepare_model_tokenizer
[rank0]: result = self._prepare_single_model(model_key, key, model_type, model_revision)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 81, in _prepare_single_model
[rank0]: task_type, num_labels = self._get_model_task_type(model_dir)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 41, in _get_model_task_type
[rank0]: model_config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1362, in from_pretrained
[rank0]: raise ValueError(
[rank0]: ValueError: The checkpoint you are trying to load has model type qwen3_asr but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[rank0]: You can update Transformers with the command pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command pip install git+https://github.com/huggingface/transformers.git
[rank1]: Traceback (most recent call last):
[rank1]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1360, in from_pretrained
[rank1]: config_class = CONFIG_MAPPING[config_dict["model_type"]]
[rank1]: ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1048, in getitem
[rank1]: raise KeyError(key)
[rank1]: KeyError: 'qwen3_asr'

[rank1]: During handling of the above exception, another exception occurred:

[rank1]: Traceback (most recent call last):
[rank1]: File "/nas/xyq/ms-swift/swift/cli/rlhf.py", line 7, in
[rank1]: rlhf_main()
[rank1]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 246, in rlhf_main
[rank1]: return SwiftRLHF(args).main()
[rank1]: ^^^^^^^^^^^^^^^
[rank1]: File "/nas/xyq/ms-swift/swift/pipelines/train/sft.py", line 29, in init
[rank1]: self._prepare_model_tokenizer()
[rank1]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 130, in _prepare_model_tokenizer
[rank1]: result = self._prepare_single_model(model_key, key, model_type, model_revision)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 81, in _prepare_single_model
[rank1]: task_type, num_labels = self._get_model_task_type(model_dir)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 41, in _get_model_task_type
[rank1]: model_config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1362, in from_pretrained
[rank1]: raise ValueError(
[rank1]: ValueError: The checkpoint you are trying to load has model type qwen3_asr but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[rank1]: You can update Transformers with the command pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command pip install git+https://github.com/huggingface/transformers.git
[rank0]:[W427 15:22:06.736872555 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
W0427 15:22:08.003000 1491042 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 1491207 closing signal SIGTERM
E0427 15:22:08.118000 1491042 site-packages/torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 0 (pid: 1491206) of binary: /nas/syh/miniconda3/envs/ms-swift/bin/python3.11
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/run.py", line 995, in
main()
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 362, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/run.py", line 991, in main
run(args)
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/run.py", line 982, in run
elastic_launch(
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 170, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/nas/xyq/ms-swift/swift/cli/rlhf.py FAILED

Failures:
[1]:
time : 2026-04-27_15:22:08
host : iZl4vdqozw0b94f5yrqsboZ
rank : 1 (local_rank: 1)
exitcode : -15 (pid: 1491207)
error_file: <N/A>
traceback : Signal 15 (SIGTERM) received by PID 1491207

Root Cause (first observed failure):
[0]:
time : 2026-04-27_15:22:08
host : iZl4vdqozw0b94f5yrqsboZ
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1491206)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================`

No qwen3_asr [rank1]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1360, in from_pretrained
[rank1]: config_class = CONFIG_MAPPING[config_dict["model_type"]]
[rank1]: ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1048, in getitem
[rank1]: raise KeyError(key)
[rank1]: KeyError: 'qwen3_asr'

[xyqsgdog-ctrl]

If I change beta to 0 " --beta 0.0 \ ", the KeyError: 'qwen3_asr' will not exist, but new problem:

CPU Virtual Memory: used = 95.22 GB, percent = 9.5% Train: 0%| | 0/3125 [00:00<?, ?it/s[WARNING:swift] IndexError occurs in the _extend_tokens function: index 1 is out of bounds for dimension 0 with size 1. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:400: masked_scatter_size_check: block: [0,0,0], thread: [0,0,0] Assertion totalElements <= srcSizefailed. [INFO:swift] last_model_checkpoint: None [INFO:swift] best_model_checkpoint: None [rank0]: Traceback (most recent call last): [rank0]: File "/nas/xyq/ms-swift/swift/cli/rlhf.py", line 7, in <module> [rank0]: rlhf_main() [rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/rlhf.py", line 246, in rlhf_main [rank0]: return SwiftRLHF(args).main() [rank0]: ^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/pipelines/base.py", line 52, in main [rank0]: result = self.run() [rank0]: ^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/ray/base.py", line 168, in wrapper [rank0]: return func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/sft.py", line 197, in run [rank0]: return self.train(trainer) [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/pipelines/train/sft.py", line 270, in train [rank0]: trainer.train(resume_checkpoint) [rank0]: File "/nas/xyq/ms-swift/swift/trainers/mixin.py", line 896, in train [rank0]: res = super().train(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train [rank0]: return inner_training_loop( [rank0]: ^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop [rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 1917, in training_step [rank0]: return super().training_step(model, inputs, num_items_in_batch) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 4014, in training_step [rank0]: inputs = self._prepare_inputs(inputs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/utils.py", line 613, in wrapper [rank0]: return func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 203, in _prepare_inputs [rank0]: generation_batch = self._generate_and_score_completions(generation_batch) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/utils.py", line 613, in wrapper [rank0]: return func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 244, in _generate_and_score_completions [rank0]: batch_encoded_inputs = self._prepare_batch_inputs(inputs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/utils.py", line 613, in wrapper [rank0]: return func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 935, in _prepare_batch_inputs [rank0]: self._get_per_token_logps_and_entropies(self.model, batch_encoded_inputs)[0]) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/utils.py", line 613, in wrapper [rank0]: return func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 1708, in _get_per_token_logps_and_entropies [rank0]: return self._get_per_token_logps_and_entropies_single(model, inputs, compute_entropy=compute_entropy) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 1754, in _get_per_token_logps_and_entropies_single [rank0]: logps, entropies = self._get_logps_via_local_forward( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/rlhf_trainers/grpo_trainer.py", line 1660, in _get_logps_via_local_forward [rank0]: logits = model(**model_inputs).logits [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1882, in _call_impl [rank0]: return inner() [rank0]: ^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1830, in inner [rank0]: result = forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/ms-swift/swift/model/utils.py", line 101, in _new_func [rank0]: res = _old_func(submodel, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/utils/generic.py", line 918, in wrapper [rank0]: output = func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/Qwen3-ASR/qwen_asr/core/transformers_backend/modeling_qwen3_asr.py", line 1230, in forward [rank0]: outputs = self.model( [rank0]: ^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1882, in _call_impl [rank0]: return inner() [rank0]: ^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1830, in inner [rank0]: result = forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/utils/generic.py", line 1072, in wrapper [rank0]: outputs = func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/xyq/Qwen3-ASR/qwen_asr/core/transformers_backend/modeling_qwen3_asr.py", line 1027, in forward [rank0]: attention_mask = create_causal_mask( [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/masking_utils.py", line 825, in create_causal_mask [rank0]: causal_mask = mask_interface( [rank0]: ^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/masking_utils.py", line 374, in sdpa_mask_recent_torch [rank0]: if allow_is_causal_skip and _ignore_causal_mask_sdpa(padding_mask, q_length, kv_length, kv_offset, local_size): [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/transformers/masking_utils.py", line 244, in _ignore_causal_mask_sdpa [rank0]: if ( [rank0]: torch.AcceleratorError: CUDA error: device-side assert triggered [rank0]: Search forcudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank0]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Train: 0%| | 0/3125 [00:18<?, ?it/s]
W0427 17:52:00.798000 1661049 site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 1661201 closing signal SIGTERM
E0427 17:52:00.966000 1661049 site-packages/torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 0 (pid: 1661200) of binary: /nas/syh/miniconda3/envs/ms-swift/bin/python3.11
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/run.py", line 995, in
main()
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 362, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/run.py", line 991, in main
run(args)
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/run.py", line 982, in run
elastic_launch(
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 170, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nas/syh/miniconda3/envs/ms-swift/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/nas/xyq/ms-swift/swift/cli/rlhf.py FAILED

Failures:
[1]:
time : 2026-04-27_17:52:00
host : iZl4vdqozw0b94f5yrqsboZ
rank : 1 (local_rank: 1)
exitcode : -15 (pid: 1661201)
error_file: <N/A>
traceback : Signal 15 (SIGTERM) received by PID 1661201

Root Cause (first observed failure):
[0]:
time : 2026-04-27_17:52:00
host : iZl4vdqozw0b94f5yrqsboZ
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1661200)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================`