Skip to content

Update check_loss utility for loss extraction#4759

Merged
zjjlivein merged 1 commit into
PaddlePaddle:release/1.2from
zjjlivein:fix/check-loss-release12
Jul 10, 2026
Merged

Update check_loss utility for loss extraction#4759
zjjlivein merged 1 commit into
PaddlePaddle:release/1.2from
zjjlivein:fix/check-loss-release12

Conversation

@zjjlivein

Copy link
Copy Markdown
Collaborator

改动内容

更新 tests/integration_test/check_loss.py:

  • 新增 write_loss_file 辅助函数,统一写出 loss 文件的逻辑
  • 新增 --extract_loss_only 参数:仅从日志中提取 loss 值并写出,不依赖 ground truth 文件
  • --gt_file 改为非必填(--extract_loss_only 模式下可不传)
  • 简化 loss_pattern 正则

测试

本地 python -m py_compile 通过,未做行为破坏性改动,其余分支逻辑保持一致。


🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zjjlivein zjjlivein merged commit f8320e0 into PaddlePaddle:release/1.2 Jul 10, 2026
16 of 18 checks passed
@Paddle-CI-Bot

Copy link
Copy Markdown

PaddleFormers Log Analysis

Run #29084417214 · Attempt 1

日志分析报告

流水线名称 问题标签 修复建议 日志片段
Fleet Model Test - Integration test (H20, multi-card) nvidia-container-cli 初始化超时 与本PR无关,H20 runner GPU 驱动 RPC 不通,CI 维护人员检查机器 nvidia-container-runtime 状态,rerun 报错代码

失败的测试case:

Integration test (H20, multi-card) — docker 启动失败,未进入测试阶段,无具体 test case 失败

根本原因分析:
nvidia-container-cli 在 H20 multi-card runner (yqlcc01-bbc-yqonlinea-com-1567435) 上 GPU 驱动 RPC 超时,docker 容器以 exit code 125 启动失败,与本 PR (fix/check-loss-release12) 代码变更无关。

修复建议:
CI 维护人员登录 runner yqlcc01-bbc-yqonlinea-com-1567435 检查 nvidia-container-runtime 及 GPU 驱动状态(nvidia-smisystemctl status nvidia-persistenced),恢复后对 PR #4759 重跑该 job 即可。


🔍 准确性记录:请点击评论底部 😊 图标,选择 👍(准确)或 👎(有误),将自动记录到 CI 监控系统

🔄 每次 Re-run 后自动更新

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants