Skip to content

docs(trtllm): Update the trtllm installation instructions #11066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rmccorm4 merged 3 commits into from
Jul 2, 2026
Merged

docs(trtllm): Update the trtllm installation instructions #11066

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
244286d
Update the trtllm installation instructions
tanmayv25 Jun 29, 2026
c3410e5
docs: make TensorRT-LLM local-install note container-only
tanmayv25 Jun 30, 2026
3d922ef
Apply suggestion from @coderabbitai[bot]
rmccorm4 Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 5 additions & 15 deletions docs/getting-started/local-installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ versions and backend guides for run instructions: [SGLang](../backends/sglang/RE

### Option B: Install from PyPI

Supported for vLLM and SGLang only. Use Option A for TensorRT-LLM.

```bash
# Install uv (recommended Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
Expand All @@ -79,19 +81,6 @@ uv pip install --prerelease=allow "ai-dynamo[sglang]"
For CUDA 13 (B300/GB300), the container is recommended. See
[SGLang install docs](https://docs.sglang.io/get_started/install.html) for details.

**TensorRT-LLM**

```bash
sudo apt install python3-dev
pip install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu130
pip install --pre --extra-index-url https://pypi.nvidia.com "ai-dynamo[trtllm]"
```

TensorRT-LLM requires `pip` due to a transitive Git URL dependency that
`uv` doesn't resolve. We recommend using the TensorRT-LLM container for
broader compatibility. See the [TRT-LLM backend guide](../backends/trtllm/README.md)
for details.

**vLLM**

```bash
Expand Down Expand Up @@ -209,9 +198,10 @@ The default model `Qwen/Qwen3-0.6B` requires ~2GB of GPU memory. Larger models n

Start with a small model and scale up based on your hardware.

**Python 3.11 with TensorRT-LLM**
**TensorRT-LLM**

TensorRT-LLM does not support Python 3.11. If you see installation failures with TensorRT-LLM, check your Python version with `python3 --version`. Use Python 3.10 or 3.12 instead.
TensorRT-LLM is not supported via a local PyPI install. Use the
`tensorrtllm-runtime` container (Option A).

**Container runs but GPU not detected**

Expand Down
17 changes: 4 additions & 13 deletions docs/getting-started/local-installation.zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runt

### 选项 B：从 PyPI 安装

仅支持 vLLM 和 SGLang。TensorRT-LLM 请使用选项 A。

```bash
# Install uv (recommended Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
Expand All @@ -80,18 +82,6 @@ uv pip install --prerelease=allow "ai-dynamo[sglang]"
对于 CUDA 13（B300/GB300），推荐使用容器。详情请参阅
[SGLang 安装文档](https://docs.sglang.io/get_started/install.html)。

**TensorRT-LLM**

```bash
sudo apt install python3-dev
pip install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu130
pip install --pre --extra-index-url https://pypi.nvidia.com "ai-dynamo[trtllm]"
```

由于传递性 Git URL 依赖项 `uv` 无法解析，TensorRT-LLM 需要使用 `pip`。
为获得更广泛的兼容性，我们建议使用 TensorRT-LLM 容器。
详情请参阅 [TRT-LLM 后端指南](../backends/trtllm/README.md)。

**vLLM**

```bash
Expand Down Expand Up @@ -210,7 +200,8 @@ curl localhost:8000/v1/chat/completions \

**TensorRT-LLM 与 Python 3.11**

TensorRT-LLM 不支持 Python 3.11。如果你在安装 TensorRT-LLM 时看到失败，请使用 `python3 --version` 检查 Python 版本。请改用 Python 3.10 或 3.12。
TensorRT-LLM 仅支持容器路径。请使用 `tensorrtllm-runtime`
容器（选项 A）。

**容器运行但未检测到 GPU**

Expand Down
10 changes: 6 additions & 4 deletions docs/reference/release-artifacts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ Release history in this document begins at v0.6.0.

### Python Wheels

We recommend using the TensorRT-LLM NGC container instead of the `ai-dynamo[trtllm]` wheel. See the [NGC container collection](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo) for supported images.
For TensorRT-LLM, use the NGC container — not the `ai-dynamo[trtllm]` PyPI extra.
See the [NGC container collection](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo).

| Package | Description | Python | Platform | PyPI |
|---------|-------------|--------|----------|------|
Expand Down Expand Up @@ -106,11 +107,12 @@ docker pull nvcr.io/nvidia/ai-dynamo/snapshot-agent:1.2.1
> For detailed installation instructions, see the [Quickstart](https://docs.nvidia.com/dynamo/getting-started/quickstart) in the docs.

```bash
# Install Dynamo with a specific backend (Recommended)
# Install Dynamo with a specific backend (vLLM and SGLang)
uv pip install "ai-dynamo[vllm]==1.2.1"
uv pip install --prerelease=allow "ai-dynamo[sglang]==1.2.1"
# TensorRT-LLM requires the NVIDIA PyPI index and pip
pip install --pre --extra-index-url https://pypi.nvidia.com "ai-dynamo[trtllm]==1.2.1"

# TensorRT-LLM: use container (see above)
docker pull nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.2.1

# Install Dynamo core only
uv pip install ai-dynamo==1.2.1
Expand Down
Loading