Skip to content

feat: Support "echo" for TensorRT-LLM models in OpenAI API frontend v1/completions - #8523

Merged
yinggeh merged 3 commits into
mainfrom
yinggeh/tri-302-add-echo-support-to-tensorrt-llm-and-python-backend
Nov 19, 2025
Merged

feat: Support "echo" for TensorRT-LLM models in OpenAI API frontend v1/completions#8523
yinggeh merged 3 commits into
mainfrom
yinggeh/tri-302-add-echo-support-to-tensorrt-llm-and-python-backend

Conversation

@yinggeh

@yinggeh yinggeh commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

What does the PR do?

Support "echo" for TensorRT-LLM models in OpenAI API frontend v1/completions
Note: Only limited TRT-LLM models support "echo" (inflight_batcher_llm, disaggregated_serving, llmapi)

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • feat

Related PRs:

Where should the reviewer start?

Test plan:

  • CI Pipeline ID:
    38760391

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #xxx

@yinggeh
yinggeh requested review from pskiran1 and whoisj November 19, 2025 03:49
@yinggeh yinggeh self-assigned this Nov 19, 2025
@yinggeh yinggeh changed the title feat: Support v1/completions "echo parameter for TensorRT-LLM models in OpenAI API frontend feat: Support "echo" for TensorRT-LLM models in OpenAI API frontend v1/completions Nov 19, 2025

@whoisj whoisj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yinggeh yinggeh changed the title feat: Support "echo" for TensorRT-LLM models in OpenAI API frontend v1/completions feat: Support "echo" for TensorRT-LLM models in OpenAI API frontend v1/completions Nov 19, 2025
@yinggeh
yinggeh merged commit 116c3b4 into main Nov 19, 2025
3 checks passed
@yinggeh
yinggeh deleted the yinggeh/tri-302-add-echo-support-to-tensorrt-llm-and-python-backend branch November 19, 2025 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Development

Successfully merging this pull request may close these issues.

2 participants