[Cherry-Pick] [Optimization] TopP=1.0 using _random_sample (#7892) and Triton SamplerBackend (#7639) by ckl117 · Pull Request #7910 · PaddlePaddle/FastDeploy

ckl117 · 2026-05-25T05:06:56Z

Motivation

支持triton采样后端 #7639
优化top_p=1.0采样 #7892
新增环境变量FD_ENABLE_TOP_P_ONE_OPT=1，方便RL验证，待验证通过后删掉环境变量。

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick，PR标题需遵循格式，在最开始加上[Cherry-Pick]标签，以及最后面加上原PR ID，例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

Modifications

Usage or Command

Accuracy Tests

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

)

paddle-bot · 2026-05-25T05:07:02Z

Thanks for your contribution!

PaddlePaddle-bot · 2026-05-25T06:03:08Z

🤖 Paddle-CI-Agent | ci_status_monitor | 2026-05-26 17:45:07

CI报告基于以下代码生成（30分钟更新一次）:

PR commit: 118584d
Merge base: 85399db (branch: release/2.6)
查看完整 Diff
CI 详情

1 任务总览

Required 任务当前 8/10 通过；仍有 2 个 Required 失败（其中 Approval 需要人工审批），需优先处理后再合入。Optional 失败 3 个，仅供参考，不阻塞合并。

总执行（rerun次数）	总任务	✅ 通过	❌ 失败	⏳ 运行中	⏸️ 等待中	跳过
36(0)	36	31	5	0	0	0

2 任务状态汇总

2.1 Required任务 : 8/10 通过

必选任务阻塞合并，失败需优先处理。

状态	任务	耗时	根因	修复建议	日志	重跑
❌	`Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage`	1h14m	PR问题：Diff覆盖率20%，Triton采样未覆盖	补测Triton采样分支或豁免JIT代码	Job	-
❌	`Approval`	19s	需要 Approval	请通过人工审批	Job	-
✅	其余 8 个必选任务通过	-	-	-	-	-

2.2 可选任务 — 23/26 通过

可选任务不阻塞合并，失败仅供参考。

状态	任务	耗时	日志	重跑
❌	`Run iluvatar Tests / run_iluvatar_cases`	1m44s	Job	-
❌	`Check PR Template`	22s	Job	-
❌	`CI_HPU`	2h23m	Job	-
✅	其余 23 个可选任务通过	-	-	-

3 失败详情（仅 required）

Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage — 覆盖率未达标（置信度: 高）

Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage

状态: ❌ 失败
错误类型: 覆盖率未达标
置信度: 高
根因摘要: Diff覆盖率仅20%，新增Triton采样代码未覆盖
分析器: ci_analyze_unittest_fastdeploy

失败用例: 无。日志显示 TEST_EXIT_CODE=0，单测已全部通过；失败发生在覆盖率阈值校验步骤。

根因详情:
CI 在 Verify Code Coverage Threshold (80%) 步骤失败，COVERAGE_EXIT_CODE=9。diff_coverage.json 显示本 PR diff 总覆盖率为 20%（635 行变更中 506 行未覆盖），低于 80% 阈值。主要缺口来自新增/修改的采样代码：fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py 覆盖率 11.5%，同时 top_k_top_p_sampling.py、sampler.py、input_batch.py 仍有未覆盖行。

关键日志:

All tests passed
Coverage generation failed (exit code 9)
fastdeploy/model_executor/layers/sample/ops/top_k_top_p_sampling.py (70.0%): Missing lines 40,45-46
fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py (11.5%): Missing lines 105-112,133-138,...,948-957
fastdeploy/model_executor/layers/sample/sampler.py (85.5%): Missing lines 125,365,822,862,1132,1233,1337-1338,1347
fastdeploy/worker/input_batch.py (66.7%): Missing lines 552
Total:   635 lines
Missing: 506 lines
Coverage: 20%
Process completed with exit code 9.

修复建议:

在 tests/layers/test_triton_sampler.py 补充可执行 fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py 新增分支的测试，重点覆盖 lines 105-112、133-827、865-868、880、893、948-957；如 Triton JIT kernel 主体在当前单测环境不可执行，建议抽出可测 host-side 逻辑或按仓库规范申请覆盖率豁免。
在 tests/layers/test_sampler.py 或 tests/layers/test_triton_sampler.py 补充 FD_SAMPLING_CLASS="triton" + VerifyStrategy.TARGET_MATCH 场景，覆盖 sampler.py:1131-1132 的 _random_sample 路径及 top_p=1.0 优化分支。
在 worker 相关测试中增加 reset_share_inputs 断言，覆盖 fastdeploy/worker/input_batch.py:552 对 top_p_list 的重置逻辑。

修复建议摘要: 补测Triton采样新增分支或豁免JIT代码

关联变更: fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py, fastdeploy/model_executor/layers/sample/ops/top_k_top_p_sampling.py, fastdeploy/model_executor/layers/sample/sampler.py, fastdeploy/worker/input_batch.py, tests/layers/test_triton_sampler.py

链接: 查看日志

Approval — 需要人工审批（置信度: 高）

该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。

codecov-commenter · 2026-05-25T08:05:15Z

Codecov Report

❌ Patch coverage is 18.42520% with 518 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release/2.6@85399db). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...l_executor/layers/sample/ops/top_k_top_p_triton.py	10.95%	493 Missing and 3 partials ⚠️
fastdeploy/model_executor/layers/sample/sampler.py	72.58%	9 Missing and 8 partials ⚠️
...executor/layers/sample/ops/top_k_top_p_sampling.py	60.00%	3 Missing and 1 partial ⚠️
fastdeploy/worker/input_batch.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##             release/2.6    #7910   +/-   ##
==============================================
  Coverage               ?   71.86%           
==============================================
  Files                  ?      383           
  Lines                  ?    55075           
  Branches               ?     8621           
==============================================
  Hits                   ?    39579           
  Misses                 ?    12718           
  Partials               ?     2778

Flag	Coverage Δ
GPU	`71.86% <18.42%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

into 26_topp1

PaddlePaddle-bot

🤖 Paddle-CI-Agent | pr_review | 2026-05-26 23:33:18

📋 Review 摘要

PR 概述：Cherry-pick 两项优化：Triton 采样后端（#7639）和 top_p=1.0 采样路径优化（#7892）
变更范围：model_executor/layers/sample/、worker/gpu_model_runner.py、envs.py
影响面 Tag：[OP] [Optimization]

问题

级别	文件	概述
🟡 建议	`top_k_top_p_triton.py:44`	`_TRITON_BUFFER_CACHE` 模块级缓存无界增长
🟡 建议	`gpu_model_runner.py`	A6：通用路径变更需确认多硬件 ModelRunner 是否同步

历史 Findings 修复情况

Finding	问题	状态
F1	`FD_ENABLE_TOP_P_ONE_OPT` 默认值为 `"1"`（启用）	⚠️ 仍存在
F2	移除 `topp_seed` 设备拷贝可能导致运行时错误	⚠️ 仍存在
F3	`top_p_list` 类型标注过宽	⚠️ 仍存在
F4	模块级全局 warning 抑制范围过广	⚠️ 仍存在

📝 PR 规范检查

标题格式存在小问题：[Cherry-Pick] 与 [Optimization] 之间有多余空格，标准格式应为 [Cherry-Pick][Tag]（无空格）。## Modifications、## Usage or Command、## Accuracy Tests 三个 section 内容为空，需补充。

标题建议（可直接复制）：

[Cherry-Pick][Optimization] TopP=1.0 using _random_sample (#7892) and Triton SamplerBackend (#7639)

PR 描述建议（点击展开，可直接复制）

## Motivation
1. 支持 Triton 采样后端，基于 Qrita 论文实现高性能 top-k/top-p Triton kernel（#7639）
2. 优化 top_p=1.0 场景的采样路径（#7892）
3. 新增环境变量 `FD_ENABLE_TOP_P_ONE_OPT`，方便 RL 场景验证，待验证通过后移除

## Modifications
- `fastdeploy/envs.py`：新增 `FD_ENABLE_TOP_P_ONE_OPT` 环境变量；更新 `FD_SAMPLING_CLASS` 注释，新增 `"triton"` 选项
- `fastdeploy/model_executor/layers/sample/meta_data.py`：`SamplingMetadata` 新增 `top_p_list` 字段
- `fastdeploy/model_executor/layers/sample/ops/top_k_top_p_sampling.py`：提取 `dispatch_top_k_renorm_probs` 公共函数；移除 `topp_seed` 的显式设备拷贝
- `fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py`：新增基于 Qrita 论文的 Triton top-k/top-p 采样 kernel（993 行）
- `fastdeploy/model_executor/layers/sample/sampler.py`：集成 Triton 采样后端分支
- `fastdeploy/worker/gpu_model_runner.py`：适配新采样逻辑
- `fastdeploy/worker/input_batch.py`：适配 `top_p_list` 字段
- `tests/layers/test_triton_sampler.py`：新增 Triton 采样器单测

## Usage or Command
```bash
# 启用 Triton 采样后端
FD_SAMPLING_CLASS=triton python ...

# 启用 top_p=1.0 优化（默认已启用，可通过设为 0 关闭）
FD_ENABLE_TOP_P_ONE_OPT=1 python ...
```

## Accuracy Tests
N/A

## Checklist

- [x] Add at least a tag in the PR title.
  - Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
  - You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [x] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [x] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.

总体评价

Triton 采样后端实现完整，新增了单测。历史 F1-F4 问题仍未修复，建议在合入前重点关注 F2（topp_seed 设备拷贝移除的潜在运行时风险）。

PaddlePaddle-bot · 2026-05-26T15:36:51Z

+import triton  # noqa: E402
+import triton.language as tl  # noqa: E402
+
+_TRITON_TABLE_CACHE: dict[tuple[paddle.device], tuple[paddle.Tensor, paddle.Tensor]] = {}


🟡 建议 _TRITON_BUFFER_CACHE 为模块级全局字典，在长期运行的服务场景中会随不同 (device, dtype, vocab_size) 组合持续增长，无淘汰机制。

建议添加最大容量限制或使用 weakref 管理生命周期，避免显存/内存泄漏。

zeroRains and others added 2 commits May 25, 2026 12:54

[CP][Feature] support new sampler backend with triton (PaddlePaddle#7639

27b9c26

)

[Optimization] TopP=1.0 using _random_sample (PaddlePaddle#7892)

e52ec74

ckl117 had a problem deploying to Metax_ci May 25, 2026 05:07 — with GitHub Actions Failure