Test ce#4732
Conversation
|
root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。新增 workflow 的 checkout 问题已放在行级评论中,需要先修复;另外还有以下非行级问题:
优先级:P1 非行级:GitHub diff 未提供
paddleformers/fleet/transformer/moe/moe_layer.py和paddleformers/fleet/transformer/moe/vmm_utils.py的可附着 hunk。当前 lint job 报F821 undefined name 'paddlefleet_ops';本地代码也只导入了paddleformers.fleet as fleet_ops,但实际调用的是paddlefleet_ops,会阻塞 lint 并在相关路径运行时报NameError。请改为导入实际使用的包并移除未使用别名,例如:
import paddlefleet_ops优先级:P3 非行级:PR 标题和描述不在 diff 中。当前标题
Test ce、正文仍是模板,无法说明这批大范围变更的目的、测试计划和风险。建议改成更具体的标题,并在描述中补充变更范围、已跑/失败 CI、兼容风险以及是否计划拆分。
| git clone https://github.com/PaddlePaddle/PaddleFormers.git | ||
| git config --global --add safe.directory /paddle/PaddleFormers | ||
| cd PaddleFormers | ||
| git fetch origin pull/4731/head:PR_4731 && git checkout PR_4731 |
There was a problem hiding this comment.
这里把待测代码固定成 pull/4731/head,但该 workflow 会在当前 PR 上触发;PR #4732 运行时会 checkout PR #4731,CE 结果验证的不是本次提交。同样的硬编码还出现在本文件的 197、336、510、757、1259、1602 行。请把这些 checkout 都改为根据当前事件选择当前 PR head 或调度分支,例如:
if [ "${PR_ID}" != "0" ]; then
git fetch origin pull/${PR_ID}/head:PR_${PR_ID}
git checkout PR_${PR_ID}
else
git fetch origin "${BRANCH}"
git checkout "${COMMIT_ID}"
fi
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。新增的安装顺序问题已放在行级评论中;之前的 CE workflow 固定 checkout 和 paddlefleet_ops lint 阻塞问题仍未解决,继续需要修复后再合入。
| #formers | ||
| python -m pip install uv | ||
| uv build --wheel --out-dir dist --clear -vv | ||
| python -m pip install "$(ls -t dist/paddleformers-*.whl | head -1)" | ||
| #paddle develop | ||
| python -m pip uninstall paddlepaddle-gpu -y | ||
| wget -q https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuAll-LinuxCentos-Gcc11-Cuda132-Cudnn920-Trt1016-Py312-Compile/latest/paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl | ||
| python -m pip install paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu132 | ||
| #formers | ||
| python setup.py bdist_wheel > /dev/null | ||
| python -m pip install ./dist/*.whl | ||
| elif [[ "$ce_branch" == "CE_Develop_cu130_py313" ]]; then # nightly regerssion | ||
| #fleet develop | ||
| python -m pip install --pre paddlefleet --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu130/ --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu130/ -i https://pypi.org/simple | ||
| python -m pip install paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu132 |
There was a problem hiding this comment.
这里先安装本地 paddleformers wheel,再到后面安装分支指定的 Paddle wheel;但 pyproject.toml 里 paddleformers 依赖了精确的 paddlepaddle-gpu==3.4.0.post20260626+93ecba8dc6c,而本函数开头已经卸载了 Paddle,且这里没有传 Paddle nightly extra-index。结果是 pip install paddleformers-*.whl 会先尝试解析/安装自己的 Paddle 依赖,可能直接找不到包失败,或者装上与本 CE 分支不一致的 Paddle;同样的顺序问题还在 49、60 行。请先安装该 CE 分支的 Paddle,再安装 PaddleFormers wheel,并避免依赖解析覆盖已选定的 Paddle。
| #formers | |
| python -m pip install uv | |
| uv build --wheel --out-dir dist --clear -vv | |
| python -m pip install "$(ls -t dist/paddleformers-*.whl | head -1)" | |
| #paddle develop | |
| python -m pip uninstall paddlepaddle-gpu -y | |
| wget -q https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuAll-LinuxCentos-Gcc11-Cuda132-Cudnn920-Trt1016-Py312-Compile/latest/paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl | |
| python -m pip install paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu132 | |
| #formers | |
| python setup.py bdist_wheel > /dev/null | |
| python -m pip install ./dist/*.whl | |
| elif [[ "$ce_branch" == "CE_Develop_cu130_py313" ]]; then # nightly regerssion | |
| #fleet develop | |
| python -m pip install --pre paddlefleet --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu130/ --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu130/ -i https://pypi.org/simple | |
| python -m pip install paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu132 | |
| #paddle develop | |
| python -m pip uninstall paddlepaddle-gpu -y | |
| wget -q https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuAll-LinuxCentos-Gcc11-Cuda132-Cudnn920-Trt1016-Py312-Compile/latest/paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl | |
| python -m pip install paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu132 | |
| #formers | |
| python -m pip install uv | |
| uv build --wheel --out-dir dist --clear -vv | |
| python -m pip install --no-deps "$(ls -t dist/paddleformers-*.whl | head -1)" |
There was a problem hiding this comment.
这次只把第一处安装命令改成了 uv pip install,但它仍然发生在分支指定的 Paddle wheel 安装之前,而且没有加 --no-deps;uv pip 同样会解析 paddleformers wheel 里的 paddlepaddle-gpu==3.4.0.post20260626+93ecba8dc6c 依赖,所以原问题仍未解决。并且 49、60 行两个 develop 分支仍是先安装 PaddleFormers、后安装 Paddle。请把三个 develop 分支都调整为先安装对应 Paddle wheel,再用不解析依赖的方式安装 PaddleFormers,例如:
# paddle develop
python -m pip uninstall paddlepaddle-gpu -y
wget -q <branch-specific-paddle-wheel>
python -m pip install <branch-specific-paddle-wheel> --extra-index-url <branch-specific-index>
# formers
python -m pip install uv
uv build --wheel --out-dir dist --clear -vv
python -m pip install --no-deps "$(ls -t dist/paddleformers-*.whl | head -1)"There was a problem hiding this comment.
更新一下当前提交的状态:现在 38、49、60 行都改成了 uv pip install,但三处仍然都在对应的 Paddle wheel 安装之前执行,而且仍未使用 --no-deps。因此 uv 仍会解析 paddleformers wheel 中的 paddlepaddle-gpu==3.4.0.post20260626+93ecba8dc6c 依赖,可能先装上错误/不可用的 Paddle,原安装顺序问题仍未修复。请把三个 develop 分支统一改成先安装对应 Paddle wheel,再用不解析依赖的方式安装 PaddleFormers,例如:
# paddle develop
python -m pip uninstall paddlepaddle-gpu -y
wget -q <branch-specific-paddle-wheel>
python -m pip install <branch-specific-paddle-wheel> --extra-index-url <branch-specific-index>
# formers
python -m pip install uv
uv build --wheel --out-dir dist --clear -vv
python -m pip install --no-deps "$(ls -t dist/paddleformers-*.whl | head -1)"
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。安装脚本的修复仍不完整,已在既有行级线程补充当前提交下的证据;之前的 CE workflow 固定 checkout 和 paddlefleet_ops lint 阻塞问题也仍未解决。请先修复这些阻塞项后再合入。
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。新增的单测脚本问题已放在行级评论中;之前的 CE workflow 固定 checkout、paddlefleet_ops lint 和回归安装顺序问题仍需继续按既有线程处理。
| # get_diff_TO_case(){ | ||
| # export FLAGS_enable_CI=false | ||
| # if [ -z "${AGILE_COMPILE_BRANCH}" ]; then | ||
| # # Scheduled Regression Test | ||
| # FLAGS_enable_CI=true | ||
| # else | ||
| # for file_name in `git diff --numstat ${AGILE_COMPILE_BRANCH} -- |awk '{print $NF}'`;do | ||
| # ext="${file_name##*.}" | ||
| # echo "file_name: ${file_name}, ext: ${file_name##*.}" | ||
| # [[ -f "$file_name" ]] || continue | ||
| # if [[ "$ext" == "py" ]] || [[ "$ext" == "yml" ]] || [[ "$file_name" == "requirements.txt" ]] || [[ "$file_name" == packages/* ]]; then | ||
| # FLAGS_enable_CI=true | ||
| # break | ||
| # fi | ||
| # done | ||
| # fi | ||
| # } | ||
|
|
||
| get_diff_TO_case | ||
| # get_diff_TO_case |
There was a problem hiding this comment.
这里把 get_diff_TO_case 和调用一起注释掉后,FLAGS_enable_CI 不再在脚本内初始化;但 unittest-gpu.yml 仍以 bash -x scripts/unit_test/ci_unittest.sh ${paddle_whl} false ... 调用,第二个参数会让 FLAGS_enable_CE=false,也没有额外传入 FLAGS_enable_CI。结果下面的判断会直接进入 Changed Not CI case, Skips,非 CE 单测 workflow 会成功跳过全部 pytest。请恢复 diff gate,或显式为 CI 调用设置 FLAGS_enable_CI=true,确保 PR 单测仍会按改动执行。
| # get_diff_TO_case(){ | |
| # export FLAGS_enable_CI=false | |
| # if [ -z "${AGILE_COMPILE_BRANCH}" ]; then | |
| # # Scheduled Regression Test | |
| # FLAGS_enable_CI=true | |
| # else | |
| # for file_name in `git diff --numstat ${AGILE_COMPILE_BRANCH} -- |awk '{print $NF}'`;do | |
| # ext="${file_name##*.}" | |
| # echo "file_name: ${file_name}, ext: ${file_name##*.}" | |
| # [[ -f "$file_name" ]] || continue | |
| # if [[ "$ext" == "py" ]] || [[ "$ext" == "yml" ]] || [[ "$file_name" == "requirements.txt" ]] || [[ "$file_name" == packages/* ]]; then | |
| # FLAGS_enable_CI=true | |
| # break | |
| # fi | |
| # done | |
| # fi | |
| # } | |
| get_diff_TO_case | |
| # get_diff_TO_case | |
| get_diff_TO_case(){ | |
| export FLAGS_enable_CI=false | |
| if [ -z "${AGILE_COMPILE_BRANCH}" ]; then | |
| # Scheduled Regression Test | |
| FLAGS_enable_CI=true | |
| else | |
| for file_name in `git diff --numstat ${AGILE_COMPILE_BRANCH} -- |awk '{print $NF}'`;do | |
| ext="${file_name##*.}" | |
| echo "file_name: ${file_name}, ext: ${file_name##*.}" | |
| [[ -f "$file_name" ]] || continue | |
| if [[ "$ext" == "py" ]] || [[ "$ext" == "yml" ]] || [[ "$file_name" == "requirements.txt" ]] || [[ "$file_name" == packages/* ]]; then | |
| FLAGS_enable_CI=true | |
| break | |
| fi | |
| done | |
| fi | |
| } | |
| get_diff_TO_case |
| else | ||
| python -m pip install uv | ||
| uv build --wheel --out-dir dist --clear -vv | ||
| python -m pip install "$(ls -t dist/paddleformers-*.whl | head -1)" -i https://pypi.org/simple --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu129/ --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu129/ |
There was a problem hiding this comment.
unittest-gpu-ce.yml 现在会在 PR 上触发并以 FLAGS_enable_CE=true 调用这个脚本;但这里先安装 paddleformers wheel,后面 59-61 行才卸载并安装 CE 指定的 Paddle wheel。由于 pyproject.toml 中 paddleformers 依赖了精确的 paddlepaddle-gpu==3.4.0.post20260626+93ecba8dc6c,这里会先解析/安装 wheel 自带的 Paddle 依赖,可能失败或装入与 CE 指定 wheel 不一致的 Paddle,之后又覆盖掉已安装依赖,导致单测环境不可信。请在 CE 分支先安装 $paddle 对应的 Paddle wheel,再用不解析依赖的方式安装 PaddleFormers,并随后构建/安装 paddlefleet_ops,例如:
python -m pip install uv
uv build --wheel --out-dir dist --clear -vv
if [ "${FLAGS_enable_CE}" == "true" ]; then
python -m pip uninstall paddlepaddle-gpu -y
wget -q "$paddle"
python -m pip install paddlepaddle_gpu-0.0.0-cp312-cp312-linux_x86_64.whl --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu129/
fi
python -m pip install --no-deps "$(ls -t dist/paddleformers-*.whl | head -1)"
bash scripts/install_or_build_ops_wheel.sh ${AGILE_COMPILE_BRANCH:-HEAD^} -i https://pypi.org/simple --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu129/ --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu129/
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #4732 +/- ##
==========================================
Coverage ? 14.14%
==========================================
Files ? 631
Lines ? 112171
Branches ? 19797
==========================================
Hits ? 15864
Misses ? 94634
Partials ? 1673
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
liuhao2638
left a comment
There was a problem hiding this comment.
已复查最新提交。ci_unittest.sh 里此前的 diff gate 跳过问题已经恢复;新引入的 pr_update_ops.yml 仍有会阻断 workflow 解析/产物上传的问题,细节已放在行级评论中。此前 CE workflow 固定 checkout、安装顺序和 paddlefleet_ops lint 问题仍按既有讨论继续处理。
优先级:P3 非行级:PR 标题和描述仍是
Test ce 与模板内容。建议改成能说明变更范围的标题,例如 Add PaddleFormers CE and paddlefleet_ops wheel workflows,并在描述里补充主要 workflow/脚本变更、已知失败 CI、测试计划和风险。
| uses: ./.github/workflows/_clone.yml | ||
| with: | ||
| clone_dir: $BRANCH | ||
| ref: ${{ github.event.pull_request.base.ref }} |
There was a problem hiding this comment.
这里调用了 ./.github/workflows/_clone.yml,但当前仓库只有 .github/workflows/_clone_linux.yml;同时 _clone_linux.yml 也不接受下面的 clone_dir/ref 输入。这个 workflow 在 develop/release push 且 packages/ 有变更时会在复用 workflow 解析阶段失败,后续 ops wheel 构建/上传不会执行。请改为现有的 reusable workflow,并让下载步骤使用 clone job 的输出 URL,避免继续拼不存在的 PaddleFormers/${BRANCH}/${PR_ID}/${COMMIT_ID} 路径,例如:
clone:
uses: ./.github/workflows/_clone_linux.yml
# downstream jobs
wget --tries=5 --no-proxy --no-check-certificate "${{ needs.clone.outputs.repo_archive_url }}" -O PaddleFormers.tar.gz
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。新增的 pull_request 触发会让 ops 发布 workflow 在 PR 上运行并失败,细节已放在行级评论中;此前 risemeup1111 提出的 CE 固定 checkout、安装顺序和 lint 阻塞问题仍未完全解决,请继续按既有讨论处理。
另外,PR 标题仍是 Test ce、正文仍是模板。建议改成能说明变更范围的标题,例如 Add PaddleFormers CE and paddlefleet_ops wheel workflows,并在描述中补充主要 workflow/脚本变更、已知失败 CI、测试计划和风险。
| pull_request: | ||
| branches: | ||
| - "develop" | ||
| push: |
There was a problem hiding this comment.
pr_update_ops.yml 是发布 ops wheel 的 workflow,后续 job 会使用 BOS 凭据并上传产物;这里新增 pull_request 后,它会在每次 PR 更新时运行。当前 head 已产生 .github/workflows/pr_update_ops.yml 的 pull_request run,且没有任何 job 就直接失败:https://github.com/PaddlePaddle/PaddleFormers/actions/runs/28362051884 。这会把 PR 检查变成失败状态,也不应该在 PR 事件执行发布链路。请移除该触发器;如需要 PR 侧验证,建议拆一个只构建、不上传的独立 workflow。
| pull_request: | |
| branches: | |
| - "develop" | |
| push: | |
| push: |
There was a problem hiding this comment.
更新当前提交下的证据:路径变量调整后,pr_update_ops.yml 的 PR 运行已经不再停在归档下载;例如 https://github.com/PaddlePaddle/PaddleFormers/actions/runs/28364256076 中 Clone PaddleFormers 成功,多个 PR merge update ops ... job 已进入 Download PaddleFormers/Build wheel,并在构建阶段失败。这说明该 workflow 仍会在 PR 上启动 ops 发布矩阵并把 PR 检查置为失败,原阻塞问题没有解除。请继续移除这里的 pull_request 触发;需要 PR 验证时请拆成不上传、不使用发布链路的 build-only workflow。
There was a problem hiding this comment.
更新当前提交下的证据:这次新增 is_pr 后,pr_update_ops.yml 的 Clone PaddleFormers / Clone PaddleFormers 已经成功,但同一个 PR 运行随后启动了完整的 PR merge update ops ... 与 PR merge update ops on paddle-dev ... 矩阵;运行 https://github.com/PaddlePaddle/PaddleFormers/actions/runs/28367582681 中已有多项矩阵 job 进入构建并失败。说明问题不是 clone 参数,而是该发布 workflow 仍由 PR 事件触发。请移除这里的 pull_request 触发;如果需要 PR 验证,请拆出不上传、不使用发布链路的 build-only workflow。
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。pr_update_ops.yml 仍在 PR 上启动 ops 发布矩阵;最新运行已经进入下载/构建阶段并失败,新的证据已补充到既有行级线程中。此前 CE 固定 checkout、安装顺序和 lint 阻塞问题也仍按既有讨论处理。
优先级:P3 非行级:PR 标题和描述不在 diff 中。当前标题仍是
Test ce、正文仍是模板。建议改成能说明变更范围的标题,例如Add PaddleFormers CE and paddlefleet_ops wheel workflows,并在描述中补充主要 workflow/脚本变更、已知失败 CI、测试计划和风险。
| export UV_PYTHON=${{ matrix.python_version }} | ||
| source /root/proxy | ||
| uv cache clean paddlepaddle-gpu | ||
| pip install packaging "setuptools>=66.1.0" wheel "ninja==1.11.1.1" "pybind11[global]>=2.13,<3" --extra-index-url=https://www.paddlepaddle.org.cn/packages/nightly/${{ matrix.cuda.bos_CUDA }}/ |
There was a problem hiding this comment.
publish-ops-wheel-paddle-dev 现在直接执行 uv build --package paddlefleet-ops --no-build-isolation,但这个 build 后端最终会在 packages/paddlefleet_ops/setup.py 中 from paddle.utils.cpp_extension import CUDAExtension, setup,而这个 job 的下载阶段只装了 uv/bce-python-sdk,这里也没有再安装 paddlepaddle-gpu。当前运行里所有 PR merge update ops on paddle-dev ... 矩阵都已经在 Build wheel 阶段失败(例如 https://github.com/PaddlePaddle/PaddleFormers/actions/runs/28368264139/job/84039988486 )。请先安装对应 CUDA 源的 Paddle,再构建 ops wheel。
| pip install packaging "setuptools>=66.1.0" wheel "ninja==1.11.1.1" "pybind11[global]>=2.13,<3" --extra-index-url=https://www.paddlepaddle.org.cn/packages/nightly/${{ matrix.cuda.bos_CUDA }}/ | |
| python -m pip install --pre paddlepaddle-gpu -i https://www.paddlepaddle.org.cn/packages/nightly/${{ matrix.cuda.bos_CUDA }}/ --extra-index-url=https://www.paddlepaddle.org.cn/packages/stable/${{ matrix.cuda.bos_CUDA }}/ | |
| pip install packaging "setuptools>=66.1.0" wheel "ninja==1.11.1.1" "pybind11[global]>=2.13,<3" --extra-index-url=https://www.paddlepaddle.org.cn/packages/nightly/${{ matrix.cuda.bos_CUDA }}/ |
risemeup1111
left a comment
There was a problem hiding this comment.
已复查最新提交。pr_update_ops.yml 仍在 PR 上触发 ops 发布 workflow,已有线程继续作为该阻塞问题的讨论位置;本次另发现 paddle-dev ops 构建链路少装 Paddle,细节已放在新的行级评论中。
此前 CE 固定 checkout、安装顺序和 lint/PR 描述等问题仍按既有讨论处理。
Before submitting
testsfolder. If there are codecov issues, please add tests cases first.PR types
PR changes
Description