Skip to content

fix eb5 logprob #8071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bukejiyu merged 2 commits into from
Jun 24, 2026
Merged

fix eb5 logprob #8071

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
79600af
fix eb5 logprob
bukejiyu Jun 23, 2026
6ea2610
update ut
bukejiyu Jun 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion fastdeploy/worker/gpu_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3650,7 +3650,7 @@ def _get_prompt_logprobs_list(
continue
offset = self.share_inputs["cu_seqlens_q"][batch_id]
prompt_hidden_states = hidden_states[offset : offset + num_logits]
logits = self.model.compute_logits(prompt_hidden_states)
logits = self.model.compute_logits(prompt_hidden_states, self.forward_meta)

@PaddlePaddle-bot PaddlePaddle-bot Jun 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 这个修复只覆盖 GPU runner，但同名 prompt logprobs 路径仍在 xpu_model_runner.py:274、metax_model_runner.py:2818 调用 compute_logits(prompt_hidden_states)。

本文件属于通用 GPU runner，repo checklist A6 要求这类变更检查其他硬件 runner。XPU/Metax 也维护 prompt_logprobs_reqs 并在输出阶段调用 _get_prompt_logprobs_list，如果对应模型的 compute_logits 需要 forward_meta，这些路径会继续复现本 PR 修复的问题。

建议修复方式：同步把 XPU/Metax 的 prompt logprobs 分支改为传入 self.forward_meta，或在 PR 中明确这些硬件不支持该模型/能力并补充保护。

prompt_token_ids = request.prompt_token_ids[start_tok : start_tok + num_logits]
prompt_token_ids_tensor = paddle.to_tensor(prompt_token_ids, dtype="int64")
if logprobs_mode == "raw_logprobs":
Expand Down
4 changes: 3 additions & 1 deletion tests/worker/test_gpu_prompt_logprobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

import numpy as np
import paddle
from utils import MockForwardMeta

from fastdeploy.config import (
CacheConfig,
Expand Down Expand Up @@ -104,7 +105,7 @@ def __init__(self, vocab_size=128, hidden_size=128):
self.hidden_size = hidden_size
self.weight = paddle.rand([hidden_size, vocab_size], dtype="float32")

def compute_logits(self, x):
def compute_logits(self, x, forward_meta=None):

@PaddlePaddle-bot PaddlePaddle-bot Jun 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 这个 FakeModel 允许 forward_meta=None 且完全忽略该参数，所以下游即使回退成旧调用 compute_logits(prompt_hidden_states)，测试仍会通过。

这没有守住本 PR 的关键回归点：prompt logprobs 分支必须把 self.forward_meta 传给模型。

建议修复方式：让替身记录或断言收到的对象，例如 assert forward_meta is not None，或用 MockForwardMeta 中的 sentinel 影响 logits；参考值也用同一个 metadata 调用，这样漏传参数会直接失败。

return paddle.matmul(x.astype("float32"), self.weight)


Expand Down Expand Up @@ -175,6 +176,7 @@ def setup_model_runner(self):
model_runner.model = FakeModel(cfg.model_config.vocab_size, cfg.model_config.hidden_size)

model_runner.in_progress_prompt_logprobs = {}
model_runner.forward_meta = MockForwardMeta()

return model_runner

Expand Down
Loading