Skip to content

[fix] Rotate pre-existing internal_medicine.jsonl on fresh start#4755

Merged
From00 merged 2 commits into
PaddlePaddle:developfrom
yuweivvv:yw/im_rotate_on_fresh_start
Jul 9, 2026
Merged

[fix] Rotate pre-existing internal_medicine.jsonl on fresh start#4755
From00 merged 2 commits into
PaddlePaddle:developfrom
yuweivvv:yw/im_rotate_on_fresh_start

Conversation

@yuweivvv

@yuweivvv yuweivvv commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

PR types

Bug fixes

PR changes

APIs

Description

问题

InternalMedicineCallback._maybe_truncate_on_resume 目前只处理 resume_from_checkpoint 场景(state.global_step > 0)。当 fresh restart(state.global_step == 0)且已有前一次 crash / Ctrl+C 中断留下的老 jsonl 时,新 run 会直接在老 step-58 行之后 append 新 step-1 行,导致 global_step 轴出现非单调,viewer 画图时曲线错乱:

L1-58:  first run,  global_step 1 → 58   (Ctrl+C 中断)
L59+:   second run, global_step 1 → ...  (fresh 起,无 checkpoint)

修复

resume_step == 0 时,若已存在老 jsonl,则将其 rotate 为 <log_path>.bak.<YYYYMMDD_HHMMSS>,新 run 从空文件开始写;resume_step > 0(真正的 checkpoint resume)路径不变,仍然截尾保留 global_step <= resume_step 的行。

老数据不删除,只是改名备份到 .bak.* 文件;用户仍可从这些文件里查看历史数据。

行为矩阵

resume_step 老 jsonl 是否存在 新行为
0(fresh) 无操作
0(fresh) rotate 老文件到 .bak.<ts>,新 run 从空文件开始
> 0(ckpt) 无操作
> 0(ckpt) 截尾保留 global_step <= resume_step 的行(原有逻辑)

Before: _maybe_truncate_on_resume only handled resume_from_checkpoint
(state.global_step > 0). On a fresh restart (state.global_step == 0)
with a leftover jsonl from a previous crashed / Ctrl+C'd run, the new
run just appended after step-58 rows with step-1 rows, producing a
non-monotonic global_step axis that confuses the viewer.

After: On resume_step == 0, if a leftover jsonl exists, rotate it to
<log_path>.bak.<YYYYMMDD_HHMMSS> and start fresh. Existing checkpoint
resume path (resume_step > 0) is unchanged.

Old runs are not deleted, just moved aside; users can inspect the .bak
files if they want the historical data.

@risemeup1111 risemeup1111 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已完成首轮 review,未发现需要阻塞合入的问题。一个非阻塞的边界场景已放在行级评论中;当前仍有部分 CI 任务运行中,建议等 CI 全部结束后再合入。

Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.

Comment thread paddleformers/trainer/trainer_callback.py Outdated
Co-authored-by: risemeup1111 <13125134909@163.com>

@risemeup1111 risemeup1111 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已复查新提交,上一轮关于备份文件名冲突的意见已在当前代码中修复,未发现新的需要阻塞合入的问题。当前还有部分 CI 任务未完成,建议等 CI 全部结束后再合入。

Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.

@From00 From00 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@From00 From00 merged commit 0edaec0 into PaddlePaddle:develop Jul 9, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants