Skip to content

fix rerun ci#4757

Merged
zjjlivein merged 1 commit into
PaddlePaddle:developfrom
zjjlivein:fix_rerun_ci
Jul 9, 2026
Merged

fix rerun ci#4757
zjjlivein merged 1 commit into
PaddlePaddle:developfrom
zjjlivein:fix_rerun_ci

Conversation

@zjjlivein

Copy link
Copy Markdown
Collaborator

Before submitting

  • Lint code. If there are lint issues, please format the code first.
# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install

# Process previous code files separately
pre-commit run --file XXXX.py
  • Add test cases into tests folder. If there are codecov issues, please add tests cases first.

PR types

PR changes

Description

@risemeup1111 risemeup1111 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

代码层面看,这次只把 rerun workflow 的 runner 从 ernie-cpu 调整到 APPROVAL runner group;仓库里已有同类用法,未发现需要阻塞合入的代码问题。当前还有部分 CI checks 正在运行,合入前请以最终结果为准。

  • P3 优先级:P3
    非行级:PR 标题/描述不在 diff 行内。当前标题 fix rerun ci 和描述仍是模板,缺少这次变更的动机、影响范围和验证方式。建议把标题改成类似 Use APPROVAL runner group for rerun workflow,并在描述中补充:为什么 rerun job 需要切到 APPROVAL、是否影响 /re-run 评论触发流程、以及本次验证结果/待 CI 通过。
    处理要求:请针对该评论进行回复(同意并已修改请回复 Done,不同意请说明理由)。
Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.

@zjjlivein

Copy link
Copy Markdown
Collaborator Author

/re-run all-failed

@zjjlivein zjjlivein merged commit 2fd3d0d into PaddlePaddle:develop Jul 9, 2026
13 of 15 checks passed
@Paddle-CI-Bot

Copy link
Copy Markdown

PaddleFormers Log Analysis

Run #29006299429 · Attempt 1

日志分析报告

流水线名称 问题标签 修复建议 日志片段
Integration test (H20, multi-card) — GLM4.5 dpo 124超时 GLM4.5 dpo 训练在 ***** Running training ***** 后挂起超 4 分钟无任何 iteration 输出,触发 5min timeout,rerun 验证;若复现则用 py-spy dump 定位 PP+EP 初始化卡点 报错代码

失败的测试case:

Integration test (H20, multi-card) / GLM4.5 dpo
  - glm45_dpo.sh: exit code 124 (timeout 5m)
  - 训练进程在 "Running training" 打印后 4m7s 内无任何 step 输出

根本原因分析:
本 PR 仅修改了 .github/workflows/rerun.ymlruns-onernie-cpu 改为 group: APPROVAL),与 GLM4.5 dpo 训练逻辑无关。超时原因是 GLM4.5 dpo 在 PP=2、EP=4、MP=4 配置下第一个 step 数据加载或 pipeline 通信初始化卡死,属偶发性 hang,与本 PR 改动无关。

修复建议:

  1. 直接 rerun 该 job,验证是否为偶发 hang。
  2. 若 rerun 复现,在容器内对 paddleformers-cli train 进程用 py-spy dump --pid <PID> 抓取 8 卡堆栈,重点排查 pipeline_parallel.py start broadcast mp parameters 之后的 barrier/allgather 是否有卡进程。
  3. 排查 glm45_dpo.yamldata_load_time 是否因 DPO 双样本拼接导致首个 microbatch 数据准备耗时远超 sft,必要时适当放宽 CI timeout 至 8min。

🔍 准确性记录:请点击评论底部 😊 图标,选择 👍(准确)或 👎(有误),将自动记录到 CI 监控系统

🔄 每次 Re-run 后自动更新

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants