Skip to content

MiniCPM-V 4.6训练是否支持data packing? #9647

Description

@randydl

Checklist / 检查清单

  • I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues,确认这是一个新的问题与讨论。

Question Description / 问题描述

您好,我在查看 ms-swift 源码时,注意到 _patch_qwen3_5_linear_attention_sequence_parallel 函数的实现中已支持 Qwen3.5 模型的 data packing 功能。由于 MiniCPM-V 4.6 的 text model 本质上就是 Qwen3.5 的 text model,因此想请教一下:这是否意味着 MiniCPM-V 4.6 在训练时也天然支持 data packing?

如果当前并不支持,那么通过添加如下代码是否可以实现该功能?

class MiniCPMV4_6Loader(ModelLoader):
    def get_model(self, *args, **kwargs) -> PreTrainedModel:
        from transformers import AutoModelForImageTextToText
        self.auto_model_cls = self.auto_model_cls or AutoModelForImageTextToText
        
        # 参照 Qwen3.5 的实现,添加以下代码以支持 data packing
        from .qwen import _patch_qwen3_5_linear_attention_sequence_parallel
        _patch_qwen3_5_linear_attention_sequence_parallel()
        
        return super().get_model(*args, **kwargs)

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions