Skip to content

[cherry-pick][fix] internal_medicine_monitor_interval default to 0 (disabled)#4751

Merged
zjjlivein merged 1 commit into
PaddlePaddle:release/1.2from
yuweivvv:yw/fix_im_interval_default_release_1.2
Jul 8, 2026
Merged

[cherry-pick][fix] internal_medicine_monitor_interval default to 0 (disabled)#4751
zjjlivein merged 1 commit into
PaddlePaddle:release/1.2from
yuweivvv:yw/fix_im_interval_default_release_1.2

Conversation

@yuweivvv

@yuweivvv yuweivvv commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Merged in dev: #4746

Backport of #4746 to release/1.2.

Manually resolved conflicts in:

  • paddleformers/cli/hparams/finetuning_args.py
  • paddleformers/trainer/trainer.py
  • paddleformers/trainer/trainer_callback.py

@Paddle-CI-Bot

Paddle-CI-Bot commented Jul 7, 2026

Copy link
Copy Markdown

PaddleFormers Log Analysis

Run #28878578015 · Attempt 1

日志分析报告

流水线名称 问题标签 修复建议 日志片段
Fleet Model Test — GLM4.5 sft Loss Diff 在 develop 更新 glm45_sft_multi_card_gt_loss.txt 精度基准后 rerun 报错代码
Fleet Model Test — GLM4.5 lora Loss Diff 在 develop 更新 glm45_lora_multi_card_gt_loss.txt 精度基准后 rerun 报错代码
Fleet Model Test — GLM4.5 dpo Loss Diff 在 develop 更新 glm45_dpo_multi_card_gt_loss.txt 精度基准后 rerun 报错代码

失败的测试case:

GLM4.5 sft    Step 10: Log Loss=0.06036192, GT Loss=0.05900278, abs_diff=0.00135914, rel_diff=2.30%
GLM4.5 lora   Step 10: Log Loss=0.00465447, GT Loss=0.00569033, abs_diff=0.00103586, rel_diff=18.20%
GLM4.5 dpo    Step 10: Log Loss=0.60607970, GT Loss=0.60747975, abs_diff=0.00140005, rel_diff=0.23%

根本原因分析:
PR #4751trainer.pyinternal_medicine_monitor_interval gate 逻辑做了修改(仅当 interval>0 时才注册 callback),该改动影响了 SFT/LoRA/DPO 训练路径的数值行为,导致 3 个场景的 step-10 loss 均与 release/1.2 分支上的旧基准值不匹配;CI 脚本在非 develop 分支时明确打印 please update precision in develop and rerun this workflow 并 exit 1。

修复建议:

  1. 在 develop 分支上分别运行 GLM4.5 sft / lora / dpo multi-card 任务,用新产出的 loss 值更新 BOS 上的三个基准文件:
    • glm45_sft_multi_card_gt_loss.txt
    • glm45_lora_multi_card_gt_loss.txt
    • glm45_dpo_multi_card_gt_loss.txt
  2. 基准更新后,对本 PR (release/1.2) 重新触发 Fleet Model Test workflow。

🔍 准确性记录:请点击评论底部 😊 图标,选择 👍(准确)或 👎(有误),将自动记录到 CI 监控系统

🔄 每次 Re-run 后自动更新

Backport of the trainer.py gate change from PaddlePaddle#4746 to release/1.2.
Currently the callback is registered whenever internal_medicine_monitors
is non-empty, ignoring internal_medicine_monitor_interval. When callers
set monitors but leave interval at 0 (the "disable monitoring" default),
the callback still runs and later crashes in _should_monitor with a
ZeroDivisionError.

Only the trainer gate is changed. Callback constructor is unchanged on
release/1.2 (feature scaffolding differs from develop), so extra kwargs
like qk_row_stride / log_dir are intentionally not backported here.

Merged in dev: PaddlePaddle#4746
@yuweivvv yuweivvv force-pushed the yw/fix_im_interval_default_release_1.2 branch from 9f8c9e6 to eaa3c3b Compare July 7, 2026 14:55

@zjjlivein zjjlivein left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm : CI精度diff 是已知问题

@zjjlivein zjjlivein merged commit b0e2106 into PaddlePaddle:release/1.2 Jul 8, 2026
17 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants