Pure add internlm2

[learncat163]

当前PR计划废弃，整合到 #4131 中

因为 internlm2 和 nternlm2.5 相似度特别高，在2个分支里分别处理。不好处理公共代码部分，容易造成后续的合并冲突

PR types

New features

PR changes

• Add model: InternLM2
• Add model tests: InternLM2

Description

Adds support for the InternLM2 large language model (LLM), enabling inference with the internlm/internlm2-7b checkpoint.
Supports loading InternLM2 weights from both PyTorch and PaddlePaddle formats.
Some code is copied from PaddlePaddle/PaddleMIX.

Loss Alignment

test python file tests/transformers/intern_lm2/test_modeling.py

use function test_inference_with_torch_model and test_inference_with_paddle_model ,

Testing the same fixed input with these two methods shows that the loss margin is within 1e-6.

[paddle-bot]

Thanks for your contribution!

[learncat163]

/re-run all-failed

[a31413510]

/re-run all-failed

[a31413510]

/re-run all-failed

[codecov-commenter]

Codecov Report

❌ Patch coverage is 61.45695% with 291 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@a10400c). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
paddleformers/transformers/intern_lm2/modeling.py	61.14%	204 Missing ⚠️
paddleformers/transformers/intern_lm2/tokenizer.py	64.78%	50 Missing ⚠️
...eformers/transformers/intern/bert_padding_delte.py	39.13%	28 Missing ⚠️
...leformers/transformers/intern_lm2/configuration.py	75.67%	9 Missing ⚠️

❌ Your patch status has failed because the patch coverage (61.45%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #4018   +/-   ##
==========================================
  Coverage           ?   34.60%           
==========================================
  Files              ?      457           
  Lines              ?    86856           
  Branches           ?        0           
==========================================
  Hits               ?    30055           
  Misses             ?    56801           
  Partials           ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[learncat163]

/re-run all-failed

[learncat163]

/re-run all-failed

[a31413510]

PaddleFormers 的 from_pretrained 默认走 flex_checkpoint 加载路径，而这条路径强制要求模型实现_gen_aoa_config
添加_gen_aoa_config可参考：
class InternLM2PretrainedModel(PretrainedModel):
...
@classmethod
def _gen_aoa_config(cls, config):
# HF keys: model.layers.$LAYER_ID.attention.wqkv.weight, model.tok_embeddings.weight, output.weight
# PF keys: model.layers.$LAYER_ID.attention.wqkv.weight, model.tok_embeddings.weight, output.weight
# Linear weights need ^T (HF: [out, in], PF: [in, out])
aoa_config = {
"aoa_statements": [
"model.tok_embeddings.weight -> model.tok_embeddings.weight",
"model.norm.weight -> model.norm.weight",
"model.layers.$LAYER_ID.attention_norm.weight -> model.layers.$LAYER_ID.attention_norm.weight",
"model.layers.$LAYER_ID.ffn_norm.weight -> model.layers.$LAYER_ID.ffn_norm.weight",
"model.layers.$LAYER_ID.attention.wqkv.weight^T -> model.layers.$LAYER_ID.attention.wqkv.weight",
"model.layers.$LAYER_ID.attention.wo.weight^T -> model.layers.$LAYER_ID.attention.wo.weight",
"model.layers.$LAYER_ID.feed_forward.w1.weight^T -> model.layers.$LAYER_ID.feed_forward.w1.weight",
"model.layers.$LAYER_ID.feed_forward.w2.weight^T -> model.layers.$LAYER_ID.feed_forward.w2.weight",
"model.layers.$LAYER_ID.feed_forward.w3.weight^T -> model.layers.$LAYER_ID.feed_forward.w3.weight",
"output.weight^T -> output.weight",
]
}
return aoa_config
以上代码已跑通

[a31413510]

问题1：AutoConfig 找不到 internlm2 的 model_type

现象：KeyError: 'internlm2'

原因：模型的 config.json 里 model_type="internlm2"，但 PaddleFormers 的 CONFIG_MAPPING_NAMES 里只注册了 "intern_lm2"（带下划线）。

修改：auto/configuration.py

• CONFIG_MAPPING_NAMES 增加 ("internlm2", "InternLM2Config")
• SPECIAL_MODEL_TYPE_TO_MODULE_NAME 增加 ("internlm2", "intern_lm2")

问题2：Tokenizer 缺少 vocab_files_names 属性

现象：AttributeError: type object 'InternLM2Tokenizer' has no attribute 'vocab_files_names'

原因：PF 的 tokenizer_utils.py 读取 cls.vocab_files_names，但 InternLM2Tokenizer 只定义了 resource_files_names。

修改：intern_lm2/tokenizer.py

• 增加 vocab_files_names = VOCAB_FILES_NAMES

问题3：缺少模版，参考：
register_template(
name="internlm2",
format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),
format_assistant=StringFormatter(slots=["{{content}}"]),
format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),
format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
default_system=(
"You are an AI assistant whose name is InternLM (书生·浦语).\n"
"- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory "
"(上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n"
"- InternLM (书生·浦语) can understand and communicate fluently in the language chosen by the user "
"such as English and 中文."
),
chat_sep="<|im_end|>\n",
suffix=["<|im_end|>\n"],
)

问题4：encode() 返回 dict 而非 list

现象：模板处理时报错 too many values to unpack，token 数量只有 7-9 个（正常应有 190 个）

原因：PF 基类 encode() 返回 BatchEncoding 字典，而模板代码做 token_ids += tokenizer.encode(...) 期望得到列表。另外初始实现用了 _tokenize()，直接调用 SPM 把 <|im_start|>
分成 6 个子词片段。

修改：intern_lm2/tokenizer.py

• 重写 encode() 方法，返回 List[int]
• 使用 self.tokenize()（而非 self._tokenize()），前者会查 added_tokens_encoder，把 <|im_start|> 正确识别为单个 token（id=92543）

---

问题5：_expand_mask 不支持 4D attention mask

现象：ValueError: too many values to unpack (expected 2)

原因：Trainer 传入了预先展开的 4D mask，但 _expand_mask 里写死 bsz, src_len = mask.shape，只支持 2D。

修改：intern_lm2/modeling.py
if mask.ndim > 2:
return mask.astype(dtype) # 已经是4D，直接返回

---

问题6（核心）：训练 loss 系统性偏高约 0.85

现象：PF 的 loss 比 Swift 始终高 ~0.85，100步内全为正偏差，测试 FAIL。

（主因）：label 被双重 shift

• PF 的 SFTDataset 已经对 labels 做了预移位：labels = labels[1:] + [-100]，即 labels[i] = token[i+1]
• InternLM2 的 forward() 里又做了一次移位：shift_labels = labels[..., 1:]
• 两次叠加后，position i 的 logit 对应的 label 变成了 token[i+2]，预测难度更高，loss 系统性偏大
• 其他 PF 模型（Qwen3 等）用的是 CriterionLayer，不会在 forward 里再做 shift，所以没有这个问题

修改：intern_lm2/modeling.py，去掉 forward 里的 shift：

修改前（错误）

shift_logits = logits[..., :-1, :]
shift_labels = labels[..., 1:]
loss = CrossEntropyLoss()(shift_logits, shift_labels)

修改后（正确）

loss = CrossEntropyLoss()(logits.reshape([-1, vocab_size]), labels.reshape([-1]))

[liuhao2638]

已完成初次 review。发现几处会阻断 InternLM2 自动加载和训练 mask 路径的问题，具体建议已放在行级评论中；建议修复后再合入。CI 当前显示通过。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

[liuhao2638]

优先级：P1

这里无条件调用 _expand_mask()，而 _expand_mask() 里面直接执行 bsz, src_len = mask.shape，只能处理 2D padding mask。PaddleFormers 的 packed/多段样本 collator 会生成形状为 [B, 1, S, S] 的 dense attention mask（例如 gen_self_attn_mask()），其他 decoder 的 _prepare_decoder_attention_mask 也都支持 3D/4D 直接传入；InternLM2 现在遇到这种训练 batch 会抛 ValueError: too many values to unpack。请在准备 mask 时区分 2D/3D/4D，并把可见性 mask 转成后续 attn_weights + attention_mask 需要的 additive mask。

可直接替换这段 mask 准备逻辑：

Suggested change

	if attention_mask is not None:
	# [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
	expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])
	combined_attention_mask = (
	expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
	)
	if attention_mask is not None:
	if len(attention_mask.shape) == 2:
	expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])
	elif len(attention_mask.shape) == 3:
	expanded_attn_mask = attention_mask.unsqueeze(1)
	expanded_attn_mask = paddle.where(
	expanded_attn_mask.astype("bool"), 0.0, paddle.finfo(inputs_embeds.dtype).min
	).astype(inputs_embeds.dtype)
	else:
	expanded_attn_mask = paddle.where(
	attention_mask.astype("bool"), 0.0, paddle.finfo(inputs_embeds.dtype).min
	).astype(inputs_embeds.dtype)
	combined_attention_mask = (
	expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
	)

[liuhao2638]

优先级：P1

这里把 HF 配置里的 model_type="internlm2" 加进了 CONFIG_MAPPING_NAMES，但下面的 model_type_to_module_name() 只会把 key 里的 - 换成 _，不会把 internlm2 映射到实际新增的 intern_lm2 包。因此 AutoConfig.from_pretrained() 遇到 InternLM2 的标准 config.json 时会尝试导入 paddleformers.transformers.internlm2.configuration，模块不存在，PR 描述里的自动加载路径会在配置阶段失败。

请同时把该 model_type 加到 SPECIAL_MODEL_TYPE_TO_MODULE_NAME，例如：

        ("qwen3_vl_moe_text", "qwen3_vl_moe"),
        ("internlm2", "intern_lm2"),

[liuhao2638]

优先级：P1

InternLM2Tokenizer 继承的是当前的 PretrainedTokenizer，其 from_pretrained() 在解析文件列表时直接读取 cls.vocab_files_names。这里只定义了 resource_files_names，所以 InternLM2Tokenizer.from_pretrained(...) 会在下载/本地加载 tokenizer 前触发 AttributeError: type object 'InternLM2Tokenizer' has no attribute 'vocab_files_names'，这会阻断 PR 测试和说明里的 ModelScope/HF tokenizer 加载路径。

请补上兼容属性：

Suggested change

	resource_files_names = VOCAB_FILES_NAMES # vocab_files_names in torch
	resource_files_names = VOCAB_FILES_NAMES # vocab_files_names in torch
	vocab_files_names = VOCAB_FILES_NAMES