Skip to content

[fix][5875912] Fix autoquant-autodeploy example #878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Fridah-nv merged 2 commits into from
Feb 11, 2026
Merged

[fix][5875912] Fix autoquant-autodeploy example #878

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8e14fcf
fix autoquant-autodeploy example
Fridah-nv Feb 10, 2026
a0b47f0
minor
Fridah-nv Feb 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions examples/llm_autodeploy/api_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@

import uvicorn
from fastapi import FastAPI, HTTPException
from tensorrt_llm._torch.auto_deploy import LLM, AutoDeployConfig
from tensorrt_llm.builder import BuildConfig
from tensorrt_llm._torch.auto_deploy import LLM
from tensorrt_llm.llmapi.llm import RequestOutput
from tensorrt_llm.sampling_params import SamplingParams
from tensorrt_llm.serve.openai_protocol import (
Expand All @@ -45,11 +44,8 @@ def build_runner_from_config(args) -> LLM:
"""Builds a model runner from our config."""
mto.enable_huggingface_checkpointing()
model_kwargs = {"max_position_embeddings": args.max_seq_len, "use_cache": False}
build_config = BuildConfig(max_seq_len=args.max_seq_len, max_batch_size=args.max_batch_size)
build_config.plugin_config.tokens_per_block = args.max_seq_len

# setup AD config
ad_config = AutoDeployConfig(
llm = LLM(
model=args.ckpt_path,
compile_backend=args.compile_backend,
device=args.device,
Expand All @@ -58,9 +54,8 @@ def build_runner_from_config(args) -> LLM:
max_seq_len=args.max_seq_len,
max_num_tokens=args.max_num_tokens,
model_kwargs=model_kwargs,
attn_backend="triton",
attn_backend="flashinfer",
)
llm = LLM(**ad_config.to_llm_kwargs())

return llm

Expand Down