Skip to content

[Feature]: Expose SmoothQuant alpha for int8_sq quantization in quantize.pyΒ #16151

Description

@BillRenCN

πŸš€ The feature, motivation and pitch

ModelOpt's SmoothQuant calibration defaults to alpha=1.0 and TensorRT-LLM's
examples/quantization/quantize.py exposes no way to change it β€” the only override in the
tree is the hardcoded Gemma special case in quantize_by_modelopt.py
({"method": "smoothquant", "alpha": 0.5}). The SmoothQuant paper recommends 0.5–0.9
depending on the model.

In our sweeps on Qwen3-0.6B/1.7B (11 alphas Γ— 2 calib sets, real TRT engines, GSM8K-1319),
alpha=1.0 was never optimal and cost up to 20 GSM8K points vs the best alpha in the same
setting (worst case: Qwen3-0.6B, 15.39% at alpha=1.0 vs 35.41% at alpha=0.85, against a
36.32% bf16 baseline).

Proposal: add --smoothquant_alpha (float, default None = current behavior) to quantize.py
and thread it to quantize_and_export, where it sets the same algorithm dict the Gemma
special case already uses. Backward compatible; ModelOpt already pydantic-validates the
field (bounds [0, 1]). Happy to submit the PR (change is ready).

Alternatives

ModelOpt's hf_ptq.py has a --recipe mechanism, so an alternative is to contribute a recipe
YAML on the Model-Optimizer side instead. But TRT-LLM's quantize.py has no recipe mechanism,
so exposing the flag directly is the minimal, self-contained fix for this CLI. The two
approaches don't conflict.

Additional context

Full sweep (11 alphas Γ— 2 calib sets Γ— 2 Qwen3 sizes):

Model Calib alpha=1.0 (current) best alpha GSM8K cost of default
Qwen3-1.7B wikitext 39.80% 57.09% @ Ξ±=0.75 βˆ’17.3 pts
Qwen3-1.7B gsm8k-train 61.87% 63.76% @ Ξ±=0.85 βˆ’1.9 pts
Qwen3-0.6B wikitext 21.76% 31.54% @ Ξ±=0.75 βˆ’9.8 pts
Qwen3-0.6B gsm8k-train 15.39% 35.41% @ Ξ±=0.85 βˆ’20.0 pts

The PR is ready: +121/βˆ’1, includes a GPU-free unit test
(tests/unittest/trt/quantization/test_smoothquant_alpha.py). No behavior change when the
flag is unset.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Metadata

Metadata

Assignees

Labels

Model optimization<NV>Model-specific performance optimizations and tuningfeature requestNew feature or request. This includes new model, dtype, functionality support

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions