[Cherry-Pick][Optimization][Speculative Decoding]opt mtp logprob (#7883)#7884
Conversation
|
Thanks for your contribution! |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7884 +/- ##
==============================================
Coverage ? 72.46%
==============================================
Files ? 382
Lines ? 54470
Branches ? 8522
==============================================
Hits ? 39474
Misses ? 12228
Partials ? 2768
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览当前 Required CI 尚未完成：required 失败任务数 1，等待/运行中的 required 任务数 5。请先完成 Approval，并等待主测试及其余 required 任务结束后再判断是否可合入。
2 任务状态汇总日志列说明：失败任务直接使用日志链接；运行中任务使用 Job 链接。 2.1 Required任务 : 4/10 通过
2.2 可选任务 — 23/25 通过
3 失败详情（仅 required）Approval — 需要人工审批（置信度: 高）
4 本轮代码/日志上下文核验
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-26 16:00:56
📋 Review 摘要
PR 概述：通过位打包 meta[1] 将 message_flag（低8位）与 max_num_logprobs（高24位）合并传递，并移除 speculative decoding 下 max_logprobs 硬编码为 20 的上限，实现 MTP logprob top_logprobs:0 场景下约 10% 的性能提升。
变更范围：custom_ops/gpu_ops/speculate_decoding/（3个 C++ 文件）、fastdeploy/output/token_processor.py、fastdeploy/worker/gpu_model_runner.py、测试
影响面 Tag：[Speculative Decoding] [OP] [DataProcessor]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🟡 建议 | mtp_save_first_token_with_topk.cc:125 |
message_flag 打包前无高位截断保护 |
| ❓ 疑问 | speculate_get_output_with_topk.cc:81 |
actual_topk=0 边界场景是否有保护 |
| ❓ 疑问 | tests/output/test_process_batch_output.py:213 |
bit 宽注释不一致（C++ 写 24 bits，测试写 16 bits） |
📝 PR 规范检查
标题含两个官方 Tag（[Optimization] + [Speculative Decoding]），§D1 要求 Cherry-Pick 格式只包含一个官方 Tag。建议简化为一个更具体的 Tag。
标题建议（可直接复制）：
[Cherry-Pick][Speculative Decoding] opt mtp logprob (#7883)
总体评价
优化思路清晰，位打包设计合理，源张量 stride 由 SPEC_LOGPROB_K+1 改为 max_num_logprobs 与上游张量分配一致。actual_topk=0 边界和 message_flag 截断两处需要作者确认或修复后可合入。
| // Pack message_flag (low 8 bits) and max_num_logprobs (high 24 bits) into | ||
| // meta[1]. Receiver unpacks both to avoid reading unused topk slots. | ||
| int max_num_logprobs = logprob_token_ids.shape()[1]; | ||
| msg_sed.meta[1] = message_flag | (max_num_logprobs << 8); |
There was a problem hiding this comment.
🟡 建议 message_flag 打包前未做高位截断
message_flag | (max_num_logprobs << 8) 假设 message_flag < 256，但没有显式截断。当前 mtype 取值为 3/4，实际安全；但若未来扩展 flag 值 ≥ 256，高位会污染 max_num_logprobs，导致接收端解包错误。
建议加防御性截断：
msg_sed.meta[1] = (message_flag & 0xFF) | (max_num_logprobs << 8);speculate_save_output_with_topk.cc 同样位置同理。
| // Unpack message_flag (low 8 bits) and actual_topk (high 24 bits) from | ||
| // meta[1]. Keep packed value; Python unpacks message_flag and actual_topk. | ||
| output_tokens_data[1] = (int64_t)msg_rcv.meta[1]; | ||
| output_tokens_data[2] = (int64_t)msg_rcv.meta[2]; |
There was a problem hiding this comment.
❓ 疑问 actual_topk = 0 边界场景是否有保护？
当 top_logprobs=0 的请求进入 logprobs_reqs 时，max_logprobs = max([0]) = 0，从而 max_num_logprobs = 0，发送端内层循环不执行，sampled token 不写入消息结构体。
接收端 actual_topk = 0，copy 循环同样不执行，传到 Python 侧后：
tokens[:, :, :0] # shape=[batch, MAX_DRAFT_TOKENS, 0]
token_ids = [row[0] for row in tokens_lists[i][:accept_num[i]]] # IndexError!请确认：
top_logprobs=0的请求是否会进入logprobs_reqs（若不会则无问题）- 若会，需要在 C++ 侧保证
max_num_logprobs >= 1（至少写入 sampled token），或在 Python 侧对actual_topk == 0分支特殊处理。
| @@ -211,8 +211,9 @@ def test_speculative_decoding_use_logprobs(self): | |||
|
|
|||
| # stop_flag | |||
| processor.output_tokens[0, 0].set_tensor(paddle.to_tensor(2)) | |||
There was a problem hiding this comment.
❓ 疑问 bit 宽注释不一致
此处注释写 actual_topk (high 16 bits)，而 C++ 侧（mtp_save_first_token_with_topk.cc、speculate_save_output_with_topk.cc、speculate_get_output_with_topk.cc）的注释均写 high 24 bits。
实际实现是 >> 8 在 int32 上取高 24 位，建议统一注释为 high 24 bits，避免误导维护者对 max_num_logprobs 的范围误判。
6 participants
Motivation
MTP 投机解码 + logprob(top_logprobs:0) 性能提升约 10%。
max_logprobs硬编码为 20 的上限，改为按实际请求动态计算，top_logprobs:0下节省一次topk计算；message_flag（低8位）和max_num_logprobs（高24位）打包进消息头meta[1]，接收端只拷贝实际有效的 logprob 槽位，避免top_logprobs=0时写入和读取全量SPEC_LOGPROB_K+1个槽位的无效开销。Modifications
custom_ops/gpu_ops/speculate_decoding/speculate_save_output_with_topk.cc/draft_model/mtp_save_first_token_with_topk.cc：将message_flag（低8位）与max_num_logprobs（高16位）打包进meta[1]；内层循环上界由SPEC_LOGPROB_K+1改为max_num_logprobs，只写入实际所需列custom_ops/gpu_ops/speculate_decoding/speculate_get_output_with_topk.cc：接收端解包actual_topk = (meta[1] >> 8) & 0xFFFF，copy 循环上界改为actual_topkfastdeploy/worker/gpu_model_runner.py：投机解码下移除max_logprobs硬编码为 20 的上限，改为按实际请求动态计算fastdeploy/output/token_processor.py：解包mtype/actual_topk并切片tokens/scores[:, :, :actual_topk]；热路径中改用批量.tolist()减少 Python 侧逐元素开销tests/output/：更新测试以反映meta[1]打包格式Usage or Command
N/A
Accuracy Tests
tests/e2e/test_ernie_21b_mtp_multistep.py logprob结果与baseline存在差异，原因如下：
k 不同时，相等值之间的相对顺序不确定，本质上是 GPU 并行归约的非确定性。
当top3 和 top4 的value相等时，paddle.topk(logprobs, 3, axis=-1)[1][:3]与paddle.topk(logprobs, 20, axis=-1)[1][:3]的取值会有差异，但不影响结果的正确性，需更新baseline
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.