Skip to content

Add TRT-LLM support to serving-config and fix HTTP probe#976

Open
anandhu-eng wants to merge 3 commits into
devfrom
servingconfig-trt-llm
Open

Add TRT-LLM support to serving-config and fix HTTP probe#976
anandhu-eng wants to merge 3 commits into
devfrom
servingconfig-trt-llm

Conversation

@anandhu-eng
Copy link
Copy Markdown
Contributor

  • parse.py: fix duplicate _SGLANG_PATTERNS/_choose_patterns definitions; add _TRTLLM_PATTERNS matching real LLM Args dump format; update _choose_patterns and _detect_framework for TRT-LLM; add "trtllm" to argparse choices
  • run.sh: document valid --serving-framework values (auto|vllm|sglang|trtllm)
  • customize.py: replace broken /server_info TRT-LLM probe with /perf_metrics (unique to TRT-LLM); vLLM and SGLang detection order unchanged

🧾 PR Checklist

  • Target branch is dev

📌 Note: PRs must be raised against dev. Do not commit directly to main.

📁 File Hygiene & Output Handling

  • No unintended files (e.g., logs, cache, temp files, pycache, output folders) are committed

📝 Comments & Communication

  • Proper inline comments are added to explain important or non-obvious changes
  • PR title and description clearly state what the PR does and why
  • Related issues (if any) are properly referenced (Fixes #, Related to #, etc.)

🛡️ Safety & Security

  • No secrets or credentials are committed
  • Paths, shell commands, and environment handling are safe and portable

- parse.py: fix duplicate _SGLANG_PATTERNS/_choose_patterns definitions;
  add _TRTLLM_PATTERNS matching real LLM Args dump format; update
  _choose_patterns and _detect_framework for TRT-LLM; add "trtllm" to
  argparse choices
- run.sh: document valid --serving-framework values (auto|vllm|sglang|trtllm)
- customize.py: replace broken /server_info TRT-LLM probe with /perf_metrics
  (unique to TRT-LLM); vLLM and SGLang detection order unchanged

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@anandhu-eng anandhu-eng requested a review from a team as a code owner June 1, 2026 10:34
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2026

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants