docs: document FlashInfer sampler JIT workaround in install guides#10954
Merged
rmccorm4 merged 6 commits intoJul 3, 2026
Merged
Conversation
The base `uv pip install -e .` only installs the runtime and frontend; building-from-source step 7 now installs a backend extra ([vllm]/[sglang]/[trtllm]) so a worker can run. Add a Troubleshooting entry to both install guides (Building from Source and Local Installation, EN + zh-CN) for the FlashInfer sampler JIT failure on CUDA 13: the resolved wheels are version-skewed (torch 13.0 runtime headers vs vLLM tilelang 13.2 nvidia-cuda-nvcc), so the sampler kernel cannot JIT-compile and vLLM aborts at engine init. Workaround: VLLM_USE_FLASHINFER_SAMPLER=0. Upstream flashinfer#3493. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: carlory <baofa.fan@daocloud.io>
carlory
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
Contributor
github-actions
Bot
added
docs
external-contribution
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
WalkthroughThe source-build and local-installation guides now require backend extras for wheel installation and add a troubleshooting note for vLLM worker startup failures on CUDA 13 FlashInfer sampler JIT errors, including the ChangesSource install guidance
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
rmccorm4
reviewed
Jun 25, 2026
- Step 7: drop [trtllm] from the `uv pip install` alternative; TensorRT-LLM
cannot be installed with uv (it needs pip + the NVIDIA package index). Add a
combined note pointing to Local Installation / the TRT-LLM backend guide.
- Apply rmccorm4's note rewording ("will install the relevant framework
dependencies" instead of "is required").
- Mirrored in the zh-CN translation.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: carlory <baofa.fan@daocloud.io>
carlory
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
Contributor
|
/ok to test 6892cea |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
rmccorm4
reviewed
Jul 2, 2026
rmccorm4
reviewed
Jul 2, 2026
rmccorm4
reviewed
Jul 2, 2026
rmccorm4
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
Contributor
|
Pulled in latest changes to docs that remove uv pip install path documentation for TRTLLM: #11066 |
Co-authored-by: Ryan McCormick <mccormick.codes@gmail.com> Signed-off-by: 杨朱 · Kiki <baofa.fan@daocloud.io>
carlory
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
Co-authored-by: Ryan McCormick <mccormick.codes@gmail.com> Signed-off-by: 杨朱 · Kiki <baofa.fan@daocloud.io>
carlory
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
…ation Apply the same §7 install changes already made to the English page to the Simplified Chinese translation: drop `[trtllm]` from the `uv pip install` backend-extra list and point TensorRT-LLM users to the `tensorrtllm-runtime` container instead of the stale pip/NVIDIA-index path. Addresses rmccorm4's review thread asking for zh-CN parity with the English suggestions on PR ai-dynamo#10954. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: carlory <baofa.fan@daocloud.io>
carlory
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
rmccorm4
approved these changes
Jul 3, 2026
Contributor
|
/ok to test 0ba262c |
dagil-nvidia
added a commit
that referenced
this pull request
Jul 3, 2026
Regenerate the translation mirror from latest main so the #10954 zh edits land fully, and restore the quickstart Card locale hrefs. Signed-off-by: Dan Gil <dagil@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Troubleshooting entry for the FlashInfer sampler JIT failure that aborts a vLLM worker at engine init on CUDA 13 source/PyPI installs, plus a small correctness fix to the install procedure.
docs/getting-started/building-from-source.md(+ zh-CN): step 7 now installs a backend extra (.[vllm]/[sglang]/[trtllm]) — the baseuv pip install -e .only provides the runtime and frontend, so a worker cannot run without one. Adds the FlashInfer sampler Troubleshooting entry.docs/getting-started/local-installation.md(+ zh-CN): adds the same FlashInfer sampler Troubleshooting entry (itsai-dynamo[vllm]PyPI path hits the same failure on CUDA 13).Failure and root cause: the CUDA pip wheels resolved for a CUDA 13 install are version-skewed —
torchpins the runtime headers to 13.0 while vLLM'stilelangpullsnvidia-cuda-nvcc13.2. FlashInfer compiles its sampler kernel withnvccagainst those headers, and CCCL's compatibility check rejects the mismatch, so vLLM aborts during engine init. Workaround:VLLM_USE_FLASHINFER_SAMPLER=0. Tracked upstream at flashinfer#3493; full root-cause analysis in #10952.Closes #10952.
Validation
dynamo.frontend+dynamo.vllm(Qwen3-0.6B) run on CUDA 13 returns/v1/chat/completionswithVLLM_USE_FLASHINFER_SAMPLER=0set. Without the flag, the worker aborts at engine init withcuda/std/__cccl/cuda_toolkit.h: ... "CUDA compiler and CUDA toolkit headers are incompatible".nvidia-cuda-nvcc==13.2.78vsnvidia-cuda-runtime==13.0.96.fern check/fern docs broken-linksrun in CI (fern CLI unavailable locally).libcudart ... not foundafter testing: a basic aggregated vLLM run starts and serves correctly withoutLD_LIBRARY_PATH, so it isn't needed in these getting-started guides (it only affects disaggregated KV transfer).🤖 Generated with Claude Code
Summary by CodeRabbit
VLLM_USE_FLASHINFER_SAMPLER=0.