docs: promote PyTorch/LLM API backend as the default quick start by faradawn · Pull Request #852 · triton-inference-server/tensorrtllm_backend

faradawn · 2026-04-08T23:39:34Z

Summary

Makes the PyTorch (LLM API) backend the primary getting started path — no TRT engine compilation required, just git clone → edit model.yaml → launch
Moves the legacy TensorRT engine quick start (trtllm-build, inflight_batcher_llm) to docs/README-engine-backend-archive.md and links to it from a short "Legacy" section
Adds an [!IMPORTANT] callout explaining the tensorrt_llm.bindings import error that occurs when launching from inside the cloned repo directory (verified fix: run from the parent of TensorRT-LLM/)

Motivation

The previous README led new users directly into the engine compilation workflow (convert checkpoint → trtllm-build → fill config templates), which requires significant setup time and is no longer the recommended path. The PyTorch/LLM API backend is simpler and sufficient for most use cases.

Test plan

Verified PyTorch quick start commands end-to-end on a 2x H100 NVL system using tritonserver:25.12-trtllm-python-py3 and TinyLlama/TinyLlama-1.1B-Chat-v1.0
Confirmed the bindings import error and fix (run from parent directory)
Legacy archive links resolve correctly

Replaces the broken Steps 2-4 in the PyTorch Backend section with commands that actually work. Key fixes: - Clone TRT-LLM source to /tmp to get the llmapi template (the git submodule in the repo is a placeholder and may be empty) - Run the server from /tmp to avoid Python resolving `tensorrt_llm` to the source tree instead of the installed package, which causes ModuleNotFoundError: No module named 'tensorrt_llm.bindings' - Add a NOTE callout explaining the import conflict pitfall - Add metrics curl example in Step 5 Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

…kend - Move PyTorch (LLM API) to the primary Getting Started section - Replace the 400-line engine backend quick start with a single link to docs/README-engine-backend-archive.md - Simplify PyTorch quick start to: git clone → edit model.yaml → launch - Add [!IMPORTANT] note about running from the clone parent directory (not inside TensorRT-LLM/) to avoid tensorrt_llm.bindings import error Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Saves the original engine-backend quick start (trtllm-build, inflight_batcher_llm) as docs/README-engine-backend-archive.md so existing users can still find it. The main README now links here from the Legacy section. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Remove the TRT-LLM version check from the quick start and switch the clone step to use the upstream repository without a pinned branch. Made-with: Cursor Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Replace the separate docs/README-engine-backend-archive.md with a "Legacy: TensorRT Engine Backend" section at the bottom of the root README. The archive duplicated most of the current README; the only unique content was the Quick Start (trtllm-build, inflight_batcher_llm model repo setup, config templating, and client examples), which is now preserved inline. TOC and Getting Started link updated to use the in-page anchor. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

faradawn · 2026-04-13T16:57:29Z

My understanding for what's no longer needed for PyTorch/LLM API backend

5-model repo structure (ensemble, preprocessing, postprocessing, tensorrt_llm, tensorrt_llm_bls) — replaced by single llmapi/ folder
fill_template.py config script — replaced by editing model.yaml
TRITON_MAX_BATCH_SIZE / INSTANCE_COUNT / MAX_QUEUE_DELAY_MS as explicit setup steps — these are now internal to the LLM API executor

What's might be relevant

                                               
  ┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────┐          
  │                   Topic                   │                             Engine backend                              │                            PyTorch/LLM API equivalent                            │              Currently documented?               │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤          
  │ Batch size > 1                            │ triton_max_batch_size in config.pbtxt, explicit inflight_fused_batching │ Continuous batching is automatic; just send concurrent requests                  │ No                                               │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤
  │ Scheduling                                │ batch_scheduler_policy in model config                                  │ Handled automatically by TRT-LLM executor                                        │ Not for LLM API                                  │          
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤          
  │ KV cache                                  │ Parameters in config.pbtxt                                              │ kv_cache_config.free_gpu_memory_fraction in model.yaml                           │ Briefly in llmapi.md multi-instance example      │          
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤          
  │ Early stopping / stop requests            │ Send stop request to Triton                                             │ LLM API has request cancellation, but not documented for Triton                  │ No                                               │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤          
  │ Quantization                              │ Engine build flags                                                      │ Specify a pre-quantized HF model (e.g., AWQ/GPTQ) in model.yaml, or quant_config │ No                                               │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤          
  │ LoRA                                      │ Covered in docs/lora.md                                                 │ LLM API has LoRA support but separate path                                       │ No                                               │
  ├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────┤          
  │ Decoding modes (top-k, beam search, etc.) │ config.pbtxt parameters                                                 │ Passed as sampling_param_* in request JSON                                       │ Partially (only max_tokens shown in quick start) │
  └───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────┘

Please me if I am wrong.

… start Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

whoisj

Just a simple, quick formatting review. I'll get to a deeper content review asap.

- Add missing `> ` to NOTE blockquote continuation line - Indent code blocks under bullet points so they render correctly - Add blank line before "For example one could run:" for readability - Left-align table columns and simplify descriptions for readability Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Move inline comments out of bash blocks as prose and split the cancel example into two separate code blocks for easier copy-paste. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

faradawn · 2026-04-15T23:27:42Z

Hi @whoisj , I've fixed

1. Start a long-running request... → moved out as prose before its own bash block
2. Cancel it → same treatment, and the single block is now split into two so each command can be independently copy-pasted

Let me know if there are other issues!

yinggeh · 2026-04-16T11:50:21Z

Correct me if I am wrong. The current TensorRT-LLM tests use inflight_batcher_llm model. If changing the recommended model to llmapi, there is no guarentee it works properly. cc @whoisj @pskiran1.
L0_backend_trtllm
L0_openai_trtllm

faradawn · 2026-04-16T22:37:47Z

I used Triton and LLMAPI and they worked on H100x2. Is there any command you'd like me to test?

Addresses reviewer concern that promoting llmapi as "Recommended" is premature without automated CI coverage. Changes: - Rename section from "— Recommended" to "— Simpler Setup" - Remove "Legacy" label from TensorRT Engine Backend sections - Add note that llmapi CI coverage is tracked in issue triton-inference-server#853 - Clarify engine backend note: it has full CI coverage Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

faradawn · 2026-04-16T23:28:12Z

Good catch @yinggeh — you are correct that automated CI for the llmapi path does not exist yet (neither in L0_backend_trtllm nor L0_openai_trtllm), so promoting it as "Recommended" was premature.

I've updated this PR to address that:

Renamed the section from "— Recommended" to "— Simpler Setup" to describe the practical benefit (no engine compilation) without implying it's fully validated at the same level as inflight_batcher_llm
Removed the "Legacy" label from the TensorRT Engine Backend sections — that path has full CI coverage and shouldn't be downplayed
Added a note in the llmapi section that CI coverage is being tracked separately (issue ci: add automated test coverage for llmapi (PyTorch backend) path #854)

Filed #854 to track adding an L0_llmapi_trtllm test. Let me know if this framing works or if you'd prefer a different approach.

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

- Add sub-section quick links under PyTorch Backend ToC entry - Fix top-level TensorRT Engine Backend ToC anchor to #tensorrt-engine-backend-1 - Fix inline redirect anchor to #tensorrt-engine-backend-1 - Remove CI tracking note from PyTorch Backend intro Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

yinggeh

LGTM

yinggeh · 2026-04-20T17:07:26Z

@whoisj Does @faradawn need to sign CLA? He's Nvidia internal.

whoisj · 2026-04-20T21:00:01Z

@whoisj Does @faradawn need to sign CLA? He's Nvidia internal.

Since he works for NVIDIA, he does NOT need to sign a CLA.

faradawn added 4 commits April 8, 2026 20:57

docs: simplify TRT-LLM clone instructions

e21c266

Remove the TRT-LLM version check from the quick start and switch the clone step to use the upstream repository without a pinned branch. Made-with: Cursor Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

faradawn force-pushed the docs/pytorch-quickstart branch from f2d89f6 to e21c266 Compare April 10, 2026 16:25

faradawn added 2 commits April 13, 2026 16:15

docs: add request cancellation example to PyTorch backend quick start

ab7cb8d

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

docs: add LLM API and SamplingParams reference links to PyTorch quick…

c70e045

… start Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

mudit-eng requested review from mudit-eng, whoisj and yinggeh April 13, 2026 20:39

docs: remove redundant SamplingParams link, one sentence per line

e3854bd

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

whoisj requested changes Apr 13, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread README.md Outdated

Comment thread README.md Outdated

Comment thread README.md

Comment thread README.md Outdated

whoisj requested changes Apr 15, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread README.md

docs: address PR review comments from whoisj

39dbc42

Move inline comments out of bash blocks as prose and split the cancel example into two separate code blocks for easier copy-paste. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

faradawn mentioned this pull request Apr 16, 2026

ci: add automated test coverage for llmapi (PyTorch backend) path #854

Open

docs: fix CI tracking issue number to triton-inference-server#854

ec3d7e0

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

yinggeh requested changes Apr 20, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread README.md

Comment thread README.md

yinggeh self-requested a review April 20, 2026 17:06

yinggeh approved these changes Apr 20, 2026

View reviewed changes

whoisj approved these changes Apr 20, 2026

View reviewed changes

whoisj merged commit 06a9c61 into triton-inference-server:main Apr 20, 2026
1 check passed

Uh oh!

Conversation

faradawn commented Apr 8, 2026

Summary

Motivation

Test plan

Uh oh!

faradawn commented Apr 13, 2026

Uh oh!

whoisj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

faradawn commented Apr 15, 2026

1. Start a long-running request... → moved out as prose before its own bash block

2. Cancel it → same treatment, and the single block is now split into two so each command can be independently copy-pasted

Uh oh!

yinggeh commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

faradawn commented Apr 16, 2026

Uh oh!

faradawn commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yinggeh left a comment

Choose a reason for hiding this comment

Uh oh!

yinggeh commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whoisj commented Apr 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yinggeh commented Apr 16, 2026 •

edited

Loading

faradawn commented Apr 16, 2026 •

edited

Loading

yinggeh commented Apr 20, 2026 •

edited

Loading