merge paddlefleet to paddleformers

[risemeup1]

Before submitting

• Lint code. If there are lint issues, please format the code first.

# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install

# Process previous code files separately
pre-commit run --file XXXX.py

• Add test cases into tests folder. If there are codecov issues, please add tests cases first.

PR types

PR changes

Description

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 5 committers have signed the CLA.

✅ zjjlivein
✅ XieYunshen
✅ risemeup1
❌ root
❌ LLLLLLLLLLLLe

---

root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Paddle-CI-Bot]

PaddleFormers Log Analysis

Run #27564247607 · Attempt 1

日志分析报告

流水线名称	问题标签	修复建议	日志片段
Fleet Model Test (H20 single-card / H20 multi-card / A100) - Install PaddleFormers	依赖冲突（scikit-learn 找不到）	PR 新增了 scikit-learn 依赖但 CI 环境无法解析，在 requirements.txt / pyproject.toml 中将 scikit-learn 版本固定到可用版本，或改用 scikit-learn; extra == "optional" 条件依赖	报错代码
CI_XPU - run_xpu_cases	依赖缺失（paddlefleet_ops 未安装）	XPU 环境缺少 paddlefleet_ops，transformer_layer.py 顶层 from paddlefleet_ops import is_deep_ep_available 导致 import chain 崩溃；需在 XPU CI 安装步骤中补充 uv pip install -e packages/paddlefleet_ops，或将该 import 改为延迟/条件 import	报错代码
Unittest GPU CI - Test	依赖缺失（build 包未安装）	python -m build --wheel 报 No module named build，需在 CI 安装步骤加 pip install build	报错代码
Unittest GPU CI - upload-coverage	runner GLIBC 版本不兼容	Node.js 24 要求 GLIBC ≥ 2.27，而 runner_04 系统不满足；与本PR无关，CI 维护人员升级 runner_04 系统或固定 actions/checkout 为 Node.js 20 版本	报错代码
Model Unittest GPU CI - Test	依赖缺失（paddle 未安装）	import paddle 报 No module named 'paddle'，paddle whl 未能成功安装到 py_3.12 环境，需检查安装步骤日志确认 whl 是否下载/安装成功	报错代码

失败的测试case:

# CI_XPU
scripts/xpu_ci/test_ernie_21b_sft.py::test_ernie_21b_sft_training
scripts/xpu_ci/test_ernie_28b_thinking_sft.py::test_ernie_28b_thinking_sft_training

# Model Unittest GPU CI
ERROR scripts/regression/test_models.py  (collection failed - ModuleNotFoundError: No module named 'paddle')

# Fleet Model Test
Integration test (H20, single card) - Install PaddleFormers: exit 1
Integration test (H20, multi-card) - Install PaddleFormers: exit 1
Integration test (A100) - Install PaddleFormers: exit 1

根本原因分析:
PR #4670（merge_paddlefleet_to_paddleformers）将 paddlefleet_ops 从独立包合入 PaddleFormers 主仓，同时在 pyproject.toml/requirements.txt 中新增了 scikit-learn 依赖，但 CI 的 uv 解析环境中 scikit-learn 无可用版本，导致三条 Fleet 流水线安装 paddleformers 全部失败；同时 transformer_layer.py 顶层直接 import paddlefleet_ops 在未安装该包的 XPU 环境中造成全链路 import 崩溃，build 包未预装导致 Unittest GPU CI wheel 构建失败。

修复建议:

1. scikit-learn 依赖：确认 scikit-learn 是否为必需运行时依赖，若非必须则移除或改为 optional；若必须，则固定一个在 PaddlePaddle PyPI 镜像中可解析的版本（如 scikit-learn>=1.3），并在本地 uv pip install 验证可解析后再提交。
2. paddlefleet_ops XPU 环境：在 XPU CI 的安装脚本（scripts/xpu_ci/ 对应 workflow）中补充 pip install -e packages/paddlefleet_ops --no-build-isolation 步骤；或将 transformer_layer.py:33 的顶层 import 改为 try/except ImportError 条件导入，避免 XPU 环境无该包时整个 import chain 崩溃。
3. build 包缺失：在 Unittest GPU CI 的 install_requirements 阶段补充 pip install build。
4. Model Unittest paddle 安装失败：确认 paddle whl 安装日志，可能是 build 包缺失导致安装提前中断，修复第3条后验证。
5. upload-coverage GLIBC：与本PR无关，CI 维护人员处理 runner_04 Node.js 24 兼容性问题。

---

🔍 准确性记录：请点击评论底部 😊 图标，选择 👍（准确）或 👎（有误），将自动记录到 CI 监控系统

_{🔄 每次 Re-run 后自动更新}

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@a81a6ff). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff            @@
##             develop   #4670   +/-   ##
=========================================
  Coverage           ?       0           
=========================================
  Files              ?       0           
  Lines              ?       0           
  Branches           ?       0           
=========================================
  Hits               ?       0           
  Misses             ?       0           
  Partials           ?       0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[swgu98]

这里丢了id: formers_install

[risemeup1111]

已复查最新提交，默认依赖中的 Paddle GPU nightly 固定版本会让只从 PyPI/镜像安装依赖的环境直接失败；具体证据和建议已放在行级评论中。该问题会阻断当前 XPU/回归安装流程，建议修复后再继续合入。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

[risemeup1111]

优先级：P1

这里把 GPU nightly 版 paddlepaddle-gpu==3.4.0.post20260623+7bd6c765854 放进默认依赖后，任何只从 PyPI/镜像安装 requirements.txt 或 wheel 默认依赖的环境都会在安装阶段失败。当前 XPU job 已在 python -m pip install -r requirements.txt 处报 No matching distribution found for paddlepaddle-gpu==3.4.0.post20260623+7bd6c765854，本地查询 PyPI 也只列到 2.6.2；后续脚本中按 CUDA/XPU 分支安装专用 Paddle wheel 的逻辑还没执行就会退出。

请不要把硬件/索引相关的 GPU nightly wheel 固定在默认依赖里；改为在各 CI/部署脚本按目标硬件显式安装对应 Paddle 包，或移到明确的 GPU extra/constraint，并同步移除 requirements.txt 和 pyproject.toml 默认依赖中的这项。建议形态如下：

# pyproject.toml
# dependencies 中不放 paddlepaddle-gpu；GPU CI 脚本用对应 index/wheel 单独安装。
dependencies = [
    "colorlog>=6.10.1",
    "tilelang-paddle",
    # ...
]

[risemeup1111]

已复查最新提交。此前默认依赖中的 Paddle GPU nightly 固定版本仍未移除，继续沿用已有行级讨论；这次新增的构建后端还会让无 .git 的源码归档构建失败，具体证据和建议已放在新的行级评论中。

^{Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.}

[risemeup1111]

优先级：P1

这里在检查 version_py.exists() and not is_git_repo() 之前就执行 git rev-parse HEAD，导致没有 .git 的源码归档/发布包无法进入兜底分支，PEP 517 后端导入时直接失败。我用当前 head 的 git archive 解包后执行 /usr/local/bin/python -m pip wheel . --no-deps，在 get_requires_for_build_wheel 阶段复现了 CalledProcessError: git rev-parse HEAD；这会影响从 GitHub tarball、sdist 或其他无 Git 元数据目录构建 PaddleFormers。

请先处理非 Git 场景，再调用任何 git 命令；若已有生成的 _version.py 就直接复用，否则至少用环境变量/unknown commit 生成可构建的版本信息。示例修复形态：

    version_py = _pkg_root / "paddleformers" / "_version.py"
    if not is_git_repo():
        if version_py.exists():
            logger.info("_version.py already exists (not in git repo), keeping it")
            return base_version
        final_version = os.environ.get("PADDLEFORMERS_VERSION", base_version)
        git_commit_hash = os.environ.get("PADDLEFORMERS_COMMIT", "unknown")
    else:
        git_commit_hash = get_git_commit_hash(_pkg_root)
        # existing git-based version/date/branch logic