Add support for LLaVA-OneVision-1.5 model

[randydl]

python tests/test_mllm.py
[INFO:swift] Conv3d patched successfully
[INFO:swift] Successfully registered `/nas_train/app.e0016372/projects/ms-swift/swift/dataset/data/dataset_info.json`.
[INFO:swift] rank: 0, local_rank: 0, world_size: 1, local_world_size: 1
`torch_dtype` is deprecated! Use `dtype` instead!
[INFO:swift] Setting args.lazy_tokenize: True
[INFO:swift] args.output_dir: `/nas_train/app.e0016372/projects/mcore-bridge/LLaVA-OneVision-1.5-4B-Instruct-mcore`
[INFO:swift] args: ExportArguments(use_ray=False, ray_exp_name=None, device_groups=None, model='/nas_train/app.e0016372/models/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct', model_type='llava_onevision1_5', model_revision=None, task_type='causal_lm', torch_dtype=torch.bfloat16, attn_impl=None, experts_impl=None, new_special_tokens=[], num_labels=None, problem_type=None, rope_scaling=None, device_map=None, max_memory={}, max_model_len=None, local_repo_path=None, init_strategy=None, template='llava_onevision1_5', system=None, max_length=262144, truncation_strategy='delete', max_pixels=None, agent_template=None, norm_bbox=None, use_chat_template=True, padding_side='right', padding_free=False, loss_scale='default', sequence_parallel_size=1, template_backend='swift', response_prefix=None, enable_thinking=None, add_non_thinking_prefix=True, dataset=[], val_dataset=[], cached_dataset=[], cached_val_dataset=[], split_dataset_ratio=0.0, data_seed=42, dataset_num_proc=1, load_from_cache_file=False, dataset_shuffle=True, val_dataset_shuffle=False, streaming=False, interleave_prob=None, stopping_strategy='first_exhausted', shuffle_buffer_size=1000, download_mode='reuse_dataset_if_exists', columns={}, strict=False, remove_unused_columns=True, model_name=None, model_author=None, custom_dataset_info=[], quant_method=None, quant_bits=None, hqq_axis=None, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, bnb_4bit_quant_storage=None, max_new_tokens=None, temperature=None, top_k=None, top_p=None, repetition_penalty=None, num_beams=1, stream=False, stop_words=[], logprobs=False, top_logprobs=None, structured_outputs_regex=None, tuner_backend='peft', tuner_type='lora', adapters=[], external_plugins=[], custom_register_path=[], seed=42, model_kwargs={}, load_args=True, load_data_args=False, packing=False, packing_length=None, packing_num_proc=1, lazy_tokenize=True, use_hf=False, hub_token=None, ddp_timeout=18000000, ddp_backend=None, ignore_args_error=False, use_swift_lora=False, merge_lora=False, safe_serialization=True, max_shard_size='5GB', output_dir='/nas_train/app.e0016372/projects/mcore-bridge/LLaVA-OneVision-1.5-4B-Instruct-mcore', quant_n_samples=256, quant_batch_size=1, group_size=128, to_cached_dataset=False, template_mode='train', to_ollama=False, to_mcore=True, to_hf=False, mcore_model=None, mcore_adapter=None, thread_count=None, test_convert_precision=True, test_convert_dtype=torch.float32, push_to_hub=False, hub_model_id=None, hub_private_repo=False, commit_message='update files', to_peft_format=False, exist_ok=True)
[INFO:swift] Global seed set to 42
[INFO:swift] Start time of running main: 2026-04-28 18:42:38.732901
[INFO:swift] swift.__version__: 4.2.0.dev0
[INFO:swift] mcore_bridge.__version__: 1.3.0.dev0
[INFO:swift] megatron.core.__version__: 0.16.2
[INFO:mcore_bridge] Setting USE_MCORE_GDN: True. You can adjust this hyperparameter through the environment variable: `USE_MCORE_GDN`.
[INFO:swift] Patch tp_plan.
/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  warnings.warn(  # warn only once
[rank0]:[W428 18:42:44.679933343 ProcessGroupNCCL.cpp:5072] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
The tokenizer you are loading from '/nas_train/app.e0016372/models/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[INFO:swift] model_kwargs: {'device_map': 'auto', 'dtype': torch.bfloat16}
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.17it/s][INFO:swift] default_system: 'You are a helpful assistant.'
[INFO:swift] max_length: 262144
[INFO:swift] response_prefix: ''
[INFO:swift] agent_template: hermes
[INFO:swift] norm_bbox: norm1000
[INFO:swift] Setting ROOT_IMAGE_DIR: None. You can adjust this hyperparameter through the environment variable: `ROOT_IMAGE_DIR`.
[INFO:swift] Setting QWENVL_BBOX_FORMAT: legacy. You can adjust this hyperparameter through the environment variable: `QWENVL_BBOX_FORMAT`.
[INFO:swift] Setting torch_dtype: torch.bfloat16
[INFO:swift] freeze_parameters: ['visual.visual', 'visual.visual.merger']
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[INFO:swift] TP: 1, PP: 1, VPP: None, CP: 1, EP: 1, ETP: 1
[INFO:swift] Setting random seeds to 42.
/nas_train/app.e0016372/projects/Megatron-LM/megatron/core/transformer/transformer_config.py:1225: UserWarning: If you are using transformer_engine as the transformer implementation, the core_attn is from transformer_engine and may be the fused version. For fused attention, you have no need to set 'core_attn' to recompute. Please check that the core_attn recompute is really needed.
  warnings.warn(
/nas_train/app.e0016372/projects/Megatron-LM/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
[INFO:swift] Megatron model created successfully.
Loading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36/36 [00:00<00:00, 63.44it/s][INFO:swift] Successfully transferred HF model weights to MG model.
[INFO:swift] n_parameter: 590
[INFO:swift] total_sum: 84295811.41383028
[INFO:swift] zero_count: 0
[INFO:swift] n_parameter: 698
[INFO:swift] total_sum: 84295811.59742403
[INFO:swift] zero_count: 0
You shouldn't move a model that is dispatched using accelerate hooks.
token_mean_diff: tensor([[0.0013, 0.0025, 0.0030, 0.0013, 0.0025, 0.0009, 0.0022, 0.0019, 0.0012,
         0.0005, 0.0016, 0.0014, 0.0026, 0.0017, 0.0007, 0.0022, 0.0016, 0.0016,
         0.0027, 0.0021, 0.0021, 0.0022, 0.0016, 0.0016, 0.0026, 0.0013, 0.0026,
         0.0018, 0.0028, 0.0016, 0.0043, 0.0021, 0.0016, 0.0014, 0.0026, 0.0030,
         0.0019, 0.0017, 0.0023, 0.0030, 0.0012, 0.0019, 0.0018, 0.0026, 0.0027,
         0.0036, 0.0028, 0.0022, 0.0021, 0.0016, 0.0028, 0.0022, 0.0016, 0.0017,
         0.0019, 0.0037, 0.0020, 0.0023, 0.0017, 0.0030, 0.0021, 0.0019, 0.0025,
         0.0026, 0.0030, 0.0020, 0.0017, 0.0017, 0.0039, 0.0024, 0.0024, 0.0012,
         0.0027, 0.0025, 0.0021, 0.0023, 0.0054, 0.0028, 0.0026, 0.0013, 0.0022,
         0.0015, 0.0014, 0.0019, 0.0020, 0.0019, 0.0020, 0.0021, 0.0035, 0.0019,
         0.0015, 0.0021, 0.0032, 0.0032, 0.0025, 0.0025, 0.0023, 0.0036, 0.0040,
         0.0016, 0.0024, 0.0039, 0.0021, 0.0024, 0.0028, 0.0037, 0.0030, 0.0042,
         0.0018, 0.0020, 0.0019, 0.0035, 0.0021, 0.0028, 0.0019, 0.0025, 0.0034,
         0.0028, 0.0019, 0.0027, 0.0036, 0.0028, 0.0019, 0.0027, 0.0017, 0.0018,
         0.0021, 0.0025, 0.0020, 0.0026, 0.0022, 0.0022, 0.0034, 0.0029, 0.0033,
         0.0019, 0.0034, 0.0020, 0.0018, 0.0045, 0.0017, 0.0023, 0.0031, 0.0018,
         0.0028, 0.0038, 0.0023, 0.0022, 0.0015, 0.0043, 0.0023, 0.0019, 0.0028,
         0.0026, 0.0047, 0.0027, 0.0041, 0.0037, 0.0024, 0.0017, 0.0019, 0.0022,
         0.0042, 0.0034, 0.0017, 0.0029, 0.0018, 0.0056, 0.0018, 0.0032, 0.0016,
         0.0029, 0.0025, 0.0026, 0.0024, 0.0041, 0.0032, 0.0021, 0.0054, 0.0030,
         0.0019, 0.0020, 0.0047, 0.0021, 0.0024, 0.0019, 0.0019, 0.0016, 0.0018,
         0.0019, 0.0019, 0.0022, 0.0020, 0.0023, 0.0013, 0.0021, 0.0027, 0.0026,
         0.0017, 0.0013, 0.0021, 0.0024, 0.0027, 0.0020, 0.0039, 0.0016, 0.0016,
         0.0015, 0.0026, 0.0025, 0.0030, 0.0041, 0.0014, 0.0061, 0.0026, 0.0033,
         0.0025, 0.0022, 0.0020, 0.0028, 0.0040, 0.0023, 0.0020, 0.0017, 0.0013,
         0.0042, 0.0014, 0.0015, 0.0048, 0.0021, 0.0018, 0.0016, 0.0020, 0.0024,
         0.0026, 0.0025, 0.0025, 0.0022, 0.0015, 0.0018, 0.0025, 0.0019, 0.0020,
         0.0029, 0.0020, 0.0048, 0.0028, 0.0019, 0.0024, 0.0021, 0.0044, 0.0044,
         0.0038, 0.0027, 0.0031, 0.0076, 0.0033, 0.0017, 0.0042, 0.0029, 0.0021,
         0.0016, 0.0015, 0.0035, 0.0017, 0.0018, 0.0031, 0.0018, 0.0032, 0.0034,
         0.0032, 0.0015, 0.0045, 0.0066, 0.0025, 0.0011, 0.0030, 0.0021, 0.0014,
         0.0735, 0.0210, 0.0183, 0.0058, 0.0022, 0.0023, 0.0020, 0.0055, 0.0023,
         0.0026, 0.0067, 0.0037, 0.0025, 0.0028, 0.0065, 0.0014, 0.0025, 0.0017,
         0.0014, 0.0025, 0.0024, 0.0064, 0.0017, 0.0017, 0.0017, 0.0026, 0.0083,
         0.0039, 0.0029, 0.0021, 0.0043, 0.0026, 0.0018, 0.0013, 0.0029, 0.0035,
         0.0028, 0.0021, 0.0027, 0.0022, 0.0033, 0.0030, 0.0019, 0.0019, 0.0019,
         0.0020, 0.0023, 0.0019, 0.0093, 0.0020, 0.0016, 0.0018, 0.0031, 0.0019,
         0.0021, 0.0023, 0.0024, 0.0061, 0.0017, 0.0022, 0.0021, 0.0016, 0.0017,
         0.0066, 0.0040, 0.0057, 0.0025, 0.0020, 0.0024, 0.0022, 0.0015, 0.0025,
         0.0022, 0.0033, 0.0024, 0.0064, 0.0031, 0.0016, 0.0018, 0.0017, 0.0028,
         0.0025, 0.0019, 0.0048, 0.0024, 0.0024, 0.0015, 0.0030, 0.0015, 0.0025,
         0.0016, 0.0023, 0.0029, 0.0025, 0.0029, 0.0034, 0.0019, 0.0053, 0.0025,
         0.0040, 0.0027, 0.0024, 0.0016, 0.0097, 0.0024, 0.0026, 0.0018, 0.0025,
         0.0020, 0.0046, 0.0066, 0.0034, 0.0023, 0.0016, 0.0020, 0.0056, 0.0019,
         0.0030, 0.0072, 0.0024, 0.0024, 0.0019, 0.0045, 0.0026, 0.0042, 0.0018,
         0.0017, 0.0027, 0.0027, 0.0018, 0.0025, 0.0064, 0.0022, 0.0021, 0.0032,
         0.0025, 0.0028, 0.0068, 0.0020, 0.0028, 0.0031, 0.0027, 0.0036, 0.0022,
         0.0020, 0.0015, 0.0033, 0.0076, 0.0019, 0.0035, 0.0030, 0.0026, 0.0028,
         0.0028, 0.0021, 0.0029, 0.0027, 0.0022, 0.0029, 0.0019, 0.0024, 0.0026,
         0.0034, 0.0025, 0.0028, 0.0016, 0.0208, 0.0030, 0.0024, 0.0021, 0.0067,
         0.0032, 0.0025, 0.0027, 0.0033, 0.0033, 0.0037, 0.0021, 0.0017, 0.0028,
         0.0032, 0.0019, 0.0016, 0.0014, 0.0031, 0.0022, 0.0026, 0.0021, 0.0603,
         0.0065, 0.0057, 0.0031, 0.0023, 0.0021, 0.0028, 0.0030, 0.0033, 0.0015,
         0.0037, 0.0130, 0.0025, 0.0025, 0.0027, 0.0019, 0.0021, 0.0036, 0.0033,
         0.0022, 0.0015, 0.0015, 0.0024, 0.0047, 0.0043, 0.0022, 0.0021, 0.0018,
         0.0022, 0.0016, 0.0036, 0.0027, 0.0018, 0.0022, 0.0082, 0.0026, 0.0039,
         0.0056, 0.0023, 0.0044, 0.0043, 0.0027, 0.0021, 0.0021, 0.0032, 0.0026,
         0.0037, 0.0023, 0.0073, 0.0046, 0.0044, 0.0036, 0.0089, 0.0048, 0.0021,
         0.0012, 0.0049, 0.0014, 0.0032, 0.0018, 0.0029, 0.0018, 0.0017, 0.0032,
         0.0017, 0.0026, 0.0015, 0.0026, 0.0046, 0.0027, 0.0029, 0.0032, 0.0034,
         0.0026, 0.0017, 0.0042, 0.0027, 0.0018, 0.0300, 0.0048, 0.0023, 0.0029,
         0.0027, 0.0028, 0.0015, 0.0022, 0.0040, 0.0018, 0.0016, 0.0021, 0.0071,
         0.0045, 0.0029, 0.0034, 0.0051, 0.0021, 0.0016, 0.0019, 0.0041, 0.0023,
         0.0026, 0.0020, 0.0018, 0.0039, 0.0018, 0.0019, 0.0031, 0.0038, 0.0052,
         0.0026, 0.0028, 0.0074, 0.0034, 0.0099, 0.0017, 0.0033, 0.0020, 0.0020,
         0.0018, 0.0026, 0.0018, 0.0349, 0.0045, 0.0113, 0.0052, 0.0041, 0.0043,
         0.0051, 0.0146, 0.0029, 0.0054, 0.0069, 0.0038, 0.0026, 0.0145, 0.0034,
         0.0031, 0.0044, 0.0018, 0.0032, 0.0023, 0.0020, 0.0017, 0.0025, 0.0015,
         0.0038, 0.0022, 0.0025, 0.0028, 0.0026, 0.0033, 0.0031, 0.0042, 0.0030,
         0.0181, 0.0034, 0.0026, 0.0028, 0.0033, 0.0014, 0.0041, 0.0030, 0.0075,
         0.0019, 0.0038, 0.0054, 0.0299, 0.0054, 0.0145, 0.0016, 0.0022, 0.0041,
         0.0018, 0.0031, 0.0015, 0.0042, 0.0026, 0.0020, 0.0051, 0.0030, 0.0025,
         0.0052, 0.0076, 0.0023, 0.0025, 0.0040, 0.0047, 0.0027, 0.0024, 0.0038,
         0.0035, 0.0022, 0.0030, 0.0026, 0.0017, 0.0025, 0.0023, 0.0018, 0.0025,
         0.0017, 0.0019, 0.0022, 0.0027, 0.0022, 0.0076, 0.0021, 0.0036, 0.0029,
         0.0031, 0.0025, 0.0052, 0.0040, 0.0066, 0.0017, 0.0025, 0.0030, 0.0019,
         0.0027, 0.0058, 0.0017, 0.0013, 0.0029, 0.0124, 0.0029, 0.0015, 0.0016,
         0.0029, 0.0050, 0.0026, 0.0032, 0.0018, 0.0037, 0.0018, 0.0048, 0.0017,
         0.0038, 0.0026, 0.0029, 0.0050, 0.0038, 0.0024, 0.0029, 0.0021, 0.0019,
         0.0041, 0.0028, 0.0023, 0.0015, 0.0041, 0.0016, 0.0035, 0.0020, 0.0051,
         0.0479, 0.0030, 0.0062, 0.0023, 0.0014, 0.0023, 0.0025, 0.0016, 0.0065,
         0.0018, 0.0021, 0.0049, 0.0031, 0.0029, 0.0046, 0.0039, 0.0102, 0.0053,
         0.0033, 0.0024, 0.0082, 0.0055, 0.0048, 0.0025, 0.0015, 0.0024, 0.0019,
         0.0008, 0.0016, 0.0019, 0.0045, 0.0016, 0.0043, 0.0011, 0.0025, 0.0008,
         0.0016, 0.0026, 0.0007, 0.0006, 0.0009, 0.0027, 0.0010, 0.0021, 0.0014,
         0.0012, 0.0007, 0.0015, 0.0008, 0.0010, 0.0011, 0.0012, 0.0023, 0.0008,
         0.0008, 0.0012, 0.0009, 0.0010, 0.0014, 0.0024, 0.0008, 0.0013, 0.0025,
         0.0015, 0.0017, 0.0007, 0.0009, 0.0020, 0.0011, 0.0010, 0.0032, 0.0006,
         0.0019, 0.0006, 0.0007, 0.0015, 0.0018, 0.0006, 0.0014, 0.0011, 0.0011,
         0.0014, 0.0011, 0.0035, 0.0006, 0.0006, 0.0021, 0.0017, 0.0006, 0.0008,
         0.0006, 0.0010, 0.0007, 0.0007, 0.0008, 0.0006, 0.0019, 0.0026, 0.0007,
         0.0007, 0.0009, 0.0014, 0.0016, 0.0007, 0.0009, 0.0012, 0.0010, 0.0007,
         0.0011, 0.0017, 0.0016, 0.0011, 0.0008, 0.0017, 0.0015, 0.0007, 0.0022,
         0.0015, 0.0030, 0.0006, 0.0032, 0.0023, 0.0024, 0.0006, 0.0018, 0.0023,
         0.0020, 0.0032, 0.0010, 0.0026, 0.0007, 0.0021, 0.0005, 0.0006, 0.0032,
         0.0005, 0.0010, 0.0017, 0.0008, 0.0006, 0.0017, 0.0050, 0.0016, 0.0009,
         0.0013, 0.0014, 0.0008, 0.0013, 0.0009, 0.0014]], device='cuda:0')
mean_diff: 0.0031035286374390125, max_diff: 0.7594947814941406
mean_diff (with loss): 0.0013847852824255824, max_diff (with loss): 0.009461402893066406 (Please check that mean_diff (with loss) is less than 0.1).
hf_tokens: [25, 12, 4684, 646, 458, 10950, 17847, 429, 151645, 151645, 151645, 15, 11, 14582, 151645, 32, 16, 18, 3265, 15625, 15, 15, 18, 15, 3265, 67561, 700, 25103, 7377, 16359, 53857, 13551, 978, 5810, 20792, 7377, 7439, 76657, 7377, 15, 7377, 17, 749, 18241, 15, 5619, 3460, 14283, 14283, 14283, 14283, 15, 76657, 67561, 17, 1620, 13, 15, 8251, 15, 3753, 3460, 14283, 79, 3460, 17, 708, 12140, 7377, 708, 20169, 20169, 4746, 17, 14283, 12644, 32976, 14283, 323, 76657, 6032, 5651, 5651, 5651, 32775, 5651, 76657, 3460, 3460, 9179, 812, 3460, 817, 15751, 6072, 8413, 7377, 3460, 4004, 8251, 17, 3460, 3460, 4766, 8413, 8251, 8413, 66130, 20, 32775, 7377, 330, 5651, 17071, 20, 3460, 7886, 14283, 69226, 3460, 3265, 3423, 3738, 4004, 15235, 700, 18072, 17, 14283, 18217, 39034, 4727, 12045, 15858, 3579, 8622, 296, 296, 323, 594, 15, 7799, 708, 3460, 11379, 3460, 3460, 1550, 749, 3265, 8413, 72400, 362, 6529, 40659, 3460, 1293, 3460, 4628, 11832, 323, 76657, 15, 40857, 5651, 5383, 67590, 5651, 31654, 3579, 40659, 3460, 16, 330, 67561, 11379, 2168, 2115, 14791, 4004, 3330, 67561, 14283, 14283, 3015, 448, 5651, 36099, 5651, 5651, 54688, 29214, 5651, 5651, 3460, 40659, 3108, 4766, 3460, 20332, 1290, 6869, 100377, 2745, 708, 7329, 438, 15186, 15, 14283, 1293, 14283, 29214, 8770, 15, 3265, 7377, 5651, 45542, 5651, 323, 76657, 3460, 15430, 4158, 9179, 12756, 4766, 3460, 4004, 2341, 708, 264, 7329, 3330, 15, 1968, 17, 16, 51254, 40659, 5651, 8251, 5651, 15, 3579, 64072, 5651, 11832, 7377, 17, 18575, 3460, 304, 8251, 1459, 7002, 59021, 3265, 27854, 8251, 24243, 96306, 3330, 1232, 12644, 12644, 15, 1290, 6303, 323, 323, 10876, 3579, 17, 3460, 3931, 17, 78265, 14283, 3330, 3265, 6869, 4628, 72400, 72400, 8380, 8115, 67561, 76657, 3265, 117097, 15, 4124, 3460, 13, 11, 3460, 8622, 7377, 7548, 13, 3460, 3460, 3460, 264, 5651, 5651, 76657, 3265, 749, 708, 7329, 6422, 20446, 4065, 3330, 700, 198, 330, 11, 220, 11, 10014, 3330, 36774, 3460, 15, 11, 3691, 12, 3460, 11, 67590, 5651, 3460, 48492, 7377, 54054, 708, 7329, 3931, 70, 20748, 8770, 5651, 8251, 5651, 3460, 12, 3460, 10380, 4065, 4158, 3265, 3460, 11, 448, 323, 323, 11379, 36099, 87, 6869, 4937, 7329, 35396, 4004, 30249, 6414, 8251, 8770, 18824, 5651, 18, 12140, 3460, 19, 3579, 19142, 18217, 32976, 2118, 56211, 330, 54688, 7548, 67590, 76657, 45297, 14283, 18, 2168, 15, 3265, 3460, 3460, 3330, 8413, 1293, 323, 5651, 76657, 342, 13673, 4131, 3691, 13876, 3691, 3579, 12728, 36774, 76657, 323, 20792, 10876, 8251, 3460, 4628, 411, 708, 7201, 3460, 1290, 11699, 2487, 7139, 19423, 40659, 323, 2613, 40659, 10078, 1459, 1378, 10078, 40659, 14283, 323, 4158, 67561, 1293, 39034, 323, 72400, 1290, 17716, 708, 53317, 8251, 1947, 5651, 4004, 3265, 40659, 4722, 99338, 76657, 35396, 10780, 10078, 10078, 99742, 10078, 56211, 1639, 99338, 14283, 45297, 17, 99538, 11699, 7329, 708, 20446, 3265, 7377, 16, 32976, 4065, 40659, 10887, 29214, 40659, 14283, 3460, 35396, 69226, 1212, 4131, 3931, 56211, 40659, 67561, 76657, 3330, 323, 32976, 17, 18241, 12822, 31592, 4004, 2030, 67590, 7377, 3738, 2487, 11900, 40659, 5651, 40659, 3460, 3579, 3265, 7548, 10507, 15, 15, 64072, 4509, 7678, 15, 101823, 39034, 16, 7377, 72400, 4004, 79945, 11379, 15625, 2024, 8251, 3460, 14283, 15430, 14283, 40659, 3565, 323, 15138, 12756, 15, 67561, 7225, 4126, 6176, 14283, 12728, 15, 4004, 323, 5810, 7329, 151645, 8413, 8413, 3691, 518, 5651, 67590, 8251, 3460, 323, 3842, 40659, 22208, 67561, 12975, 5651, 323, 3330, 3460, 2385, 7377, 18241, 3265, 11699, 18, 7824, 4004, 15, 4004, 4628, 23, 5651, 5651, 14283, 29481, 323, 3460, 2770, 3460, 36774, 15, 15138, 17, 3460, 95431, 6414, 16, 323, 16, 18241, 708, 15, 7329, 12822, 27854, 6742, 471, 16, 16202, 67590, 67590, 3753, 15, 76657, 3460, 3931, 3579, 108996, 4004, 8416, 15138, 15138, 8770, 2613, 12880, 323, 15, 56211, 15, 39020, 2168, 4145, 749, 10078, 11699, 15, 19423, 12140, 71734, 5651, 11699, 4065, 15, 1290, 19142, 8413, 15138, 35512, 76657, 76657, 15, 15, 67590, 6176, 7377, 4004, 14697, 8413, 8413, 100237, 20446, 32009, 4722, 323, 323, 20792, 5651, 3460, 5312, 111056, 54688, 4158, 73017, 14762, 8251, 5810, 15, 15, 5651, 5651, 10876, 8413, 72400, 4004, 8205, 749, 5810, 15, 8251, 15, 6884, 11699, 30367, 5651, 14201, 24036, 11699, 11379, 3460, 3460, 3460, 5810, 11699, 54688, 16202, 16202, 80147, 18824, 8413, 8413, 4004, 33542, 151645, 7329, 8413, 7548, 3460, 4004, 6078, 13673, 48674, 14201, 17, 14283, 7329, 3460, 21239, 15, 8251, 15, 3460, 10078, 7824, 6742, 27854, 15, 7377, 6303, 10078, 16, 279, 2168, 151645, 151645, 11, 785, 151645, 785, 785, 2168, 4933, 264, 3265, 5239, 315, 264, 76657, 448, 21239, 6303, 13, 576, 76657, 702, 264, 8413, 323, 19780, 67590, 448, 12460, 3691, 54688, 13, 7086, 42326, 389, 1181, 3579, 323, 24230, 13, 11445, 6414, 525, 3460, 323, 77123, 11, 448, 264, 85182, 6303, 1894, 429, 13352, 700, 2348, 1181, 18241, 18241, 13, 1105, 13, 576, 76657, 594, 24230, 374, 2613, 323, 18217, 11, 323, 1181, 702, 264, 11, 8413, 40659, 388, 429, 504, 1181, 3108, 315, 1181, 3579, 13, 576, 24230, 374, 72400, 11, 892, 5244, 311, 279, 76657, 594, 3579, 323, 80903, 432, 279, 41099, 1459, 315, 279, 2168, 13, 576, 8084, 20792, 374, 825, 315, 56211, 23709, 323, 56211, 11, 151645, 151645, 151645]
mg_tokens: [25, 12, 4684, 646, 458, 10950, 17847, 429, 151645, 151645, 151645, 15, 11, 14582, 151645, 32, 16, 18, 3265, 15625, 15, 15, 18, 15, 3265, 67561, 700, 25103, 7377, 16359, 53857, 13551, 978, 5810, 20792, 7377, 7439, 76657, 7377, 15, 7377, 17, 749, 18241, 15, 5619, 3460, 14283, 14283, 14283, 14283, 15, 76657, 67561, 17, 1620, 13, 15, 8251, 15, 3753, 3460, 14283, 79, 3460, 17, 708, 12140, 7377, 708, 20169, 20169, 4746, 17, 14283, 12644, 32976, 14283, 323, 76657, 6032, 5651, 5651, 5651, 32775, 5651, 76657, 3460, 3460, 9179, 812, 3460, 817, 15751, 6072, 8413, 7377, 3460, 4004, 8251, 17, 3460, 3460, 4766, 8413, 8251, 8413, 66130, 20, 32775, 7377, 330, 5651, 17071, 20, 3460, 7886, 14283, 69226, 3460, 3265, 3423, 3738, 4004, 15235, 700, 18072, 17, 14283, 18217, 39034, 4727, 12045, 15858, 3579, 8622, 296, 296, 323, 594, 15, 7799, 708, 3460, 11379, 3460, 3460, 1550, 749, 3265, 8413, 72400, 362, 6529, 40659, 3460, 1293, 3460, 4628, 11832, 323, 76657, 15, 40857, 5651, 5383, 67590, 5651, 31654, 3579, 40659, 3460, 16, 330, 67561, 11379, 2168, 2115, 14791, 4004, 3330, 67561, 14283, 14283, 3015, 448, 5651, 36099, 5651, 5651, 54688, 29214, 5651, 5651, 3460, 40659, 3108, 4766, 3460, 20332, 1290, 6869, 100377, 2745, 708, 7329, 438, 15186, 15, 14283, 1293, 14283, 29214, 8770, 15, 3265, 7377, 5651, 45542, 5651, 323, 76657, 3460, 15430, 4158, 9179, 12756, 4766, 3460, 4004, 2341, 708, 264, 7329, 3330, 15, 1968, 17, 16, 51254, 40659, 5651, 8251, 5651, 15, 3579, 64072, 5651, 11832, 7377, 17, 18575, 3460, 304, 8251, 1459, 7002, 59021, 3265, 27854, 8251, 24243, 96306, 3330, 1232, 12644, 12644, 15, 1290, 6303, 323, 323, 10876, 3579, 17, 3460, 3931, 17, 78265, 14283, 3330, 3265, 6869, 4628, 72400, 72400, 8380, 8115, 67561, 76657, 3265, 117097, 15, 4124, 3460, 13, 11, 3460, 8622, 7377, 7548, 13, 3460, 3460, 3460, 264, 5651, 5651, 76657, 3265, 749, 708, 7329, 6422, 20446, 4065, 3330, 700, 198, 330, 11, 220, 11, 10014, 3330, 36774, 3460, 15, 11, 3691, 12, 3460, 11, 67590, 5651, 3460, 48492, 7377, 54054, 708, 7329, 3931, 70, 20748, 8770, 5651, 8251, 5651, 3460, 12, 3460, 10380, 4065, 4158, 3265, 3460, 11, 448, 323, 323, 11379, 36099, 87, 6869, 4937, 7329, 35396, 4004, 30249, 6414, 8251, 8770, 18824, 5651, 18, 12140, 3460, 19, 3579, 19142, 18217, 32976, 2118, 56211, 330, 54688, 7548, 67590, 76657, 45297, 14283, 18, 2168, 15, 3265, 3460, 3460, 3330, 8413, 1293, 323, 5651, 76657, 342, 13673, 4131, 3691, 13876, 3691, 3579, 12728, 36774, 76657, 323, 20792, 10876, 8251, 3460, 4628, 411, 708, 7201, 3460, 1290, 11699, 2487, 7139, 19423, 40659, 323, 2613, 40659, 10078, 1459, 1378, 10078, 40659, 14283, 323, 4158, 67561, 1293, 39034, 323, 72400, 1290, 17716, 708, 53317, 8251, 1947, 5651, 4004, 3265, 40659, 4722, 99338, 76657, 35396, 10780, 10078, 10078, 99742, 10078, 56211, 1639, 99338, 14283, 45297, 17, 99538, 11699, 7329, 708, 20446, 3265, 7377, 16, 32976, 4065, 40659, 10887, 29214, 40659, 14283, 3460, 35396, 69226, 1212, 4131, 3931, 56211, 40659, 67561, 76657, 3330, 323, 32976, 17, 18241, 12822, 31592, 4004, 2030, 67590, 7377, 3738, 2487, 11900, 40659, 5651, 40659, 3460, 3579, 3265, 7548, 10507, 15, 15, 64072, 4509, 7678, 15, 101823, 39034, 16, 7377, 72400, 4004, 79945, 11379, 15625, 2024, 8251, 3460, 14283, 15430, 14283, 40659, 3565, 323, 15138, 12756, 15, 67561, 7225, 4126, 6176, 14283, 12728, 15, 4004, 323, 5810, 7329, 151645, 8413, 8413, 3691, 518, 5651, 67590, 8251, 3460, 323, 3842, 40659, 22208, 67561, 12975, 5651, 323, 3330, 3460, 2385, 7377, 18241, 3265, 11699, 18, 7824, 4004, 15, 4004, 4628, 23, 5651, 5651, 14283, 29481, 323, 3460, 2770, 3460, 36774, 15, 15138, 17, 3460, 95431, 6414, 16, 323, 16, 18241, 708, 15, 7329, 12822, 27854, 6742, 471, 16, 16202, 67590, 67590, 5651, 15, 76657, 3460, 3931, 3579, 108996, 4004, 8416, 15138, 15138, 8770, 2613, 12880, 323, 15, 56211, 15, 39020, 2168, 4145, 749, 10078, 11699, 15, 19423, 12140, 71734, 5651, 11699, 4065, 15, 1290, 19142, 8413, 15138, 35512, 76657, 76657, 15, 15, 67590, 6176, 7377, 4004, 14697, 8413, 8413, 100237, 20446, 32009, 4722, 323, 323, 20792, 5651, 3460, 5312, 111056, 54688, 4158, 73017, 14762, 8251, 5810, 15, 15, 5651, 5651, 10876, 8413, 72400, 4004, 10078, 749, 5810, 15, 8251, 15, 6884, 11699, 30367, 5651, 14201, 24036, 11699, 11379, 3460, 3460, 3460, 5810, 11699, 54688, 16202, 16202, 80147, 18824, 8413, 8413, 4004, 33542, 151645, 7329, 8413, 7548, 3460, 4004, 6078, 13673, 48674, 14201, 17, 14283, 7329, 3460, 21239, 15, 8251, 15, 3460, 10078, 7824, 6742, 27854, 15, 7377, 6303, 10078, 16, 279, 2168, 151645, 151645, 11, 785, 151645, 785, 785, 2168, 4933, 264, 3265, 5239, 315, 264, 76657, 448, 21239, 6303, 13, 576, 76657, 702, 264, 8413, 323, 19780, 67590, 448, 12460, 3691, 54688, 13, 7086, 42326, 389, 1181, 3579, 323, 24230, 13, 11445, 6414, 525, 3460, 323, 77123, 11, 448, 264, 85182, 6303, 1894, 429, 13352, 700, 2348, 1181, 18241, 18241, 13, 1105, 13, 576, 76657, 594, 24230, 374, 2613, 323, 18217, 11, 323, 1181, 702, 264, 11, 8413, 40659, 388, 429, 504, 1181, 3108, 315, 1181, 3579, 13, 576, 24230, 374, 72400, 11, 892, 5244, 311, 279, 76657, 594, 3579, 323, 80903, 432, 279, 41099, 1459, 315, 279, 2168, 13, 576, 8084, 20792, 374, 825, 315, 56211, 23709, 323, 56211, 11, 151645, 151645, 151645]
token_diff: 2
token_diff (with loss): 0
[INFO:swift] End time of running main: 2026-04-28 18:43:33.112564
[rank0]:[W428 18:43:35.870117771 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

[gemini-code-assist]

Code Review

This pull request adds support for the llava_onevision1_5 model, including its registration, constants, and a new implementation file. A typo was found in the configuration parser where the model string was missing an underscore, which would prevent the qk_layernorm setting from being correctly applied to this model.

[gemini-code-assist]

The model type string 'llavaonevision1_5' appears to have a typo (missing underscore). It should be 'llava_onevision1_5' to be consistent with the definition in src/mcore_bridge/model/constant.py and the registration in src/mcore_bridge/model/mm_gpts/llava.py. Without the underscore, the qk_layernorm configuration will not be correctly applied to this model.

Suggested change

	'qwen3_omni_moe', 'qwen3_omni', 'qwen3_vl', 'qwen3_vl_moe', 'qwen3_5', 'qwen3_5_moe', 'llavaonevision1_5'
	'qwen3_omni_moe', 'qwen3_omni', 'qwen3_vl', 'qwen3_vl_moe', 'qwen3_5', 'qwen3_5_moe', 'llava_onevision1_5'

[randydl]

The expected value is "llavaonevision1_5" (see https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/blob/main/configuration_llavaonevision1_5.py#:~:text=model_type%20%3D%20%22-,llavaonevision1_5,-%22), but note that in Swift it is referred to as "llava_onevision1_5".

[Jintao-Huang]

thanks!