Skip to content

Commit bec2767

Browse files
committed
Update docstring in examples
1 parent 6d42d08 commit bec2767

4 files changed

Lines changed: 37 additions & 0 deletions

File tree

example/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,4 @@ tinyllama*
44
TinyLlama*/
55
quantized_model_saveload/
66
rotated_model_*
7+
results

example/vllm_inference/example_autobit_vllm_inference.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,18 @@
1010
Requirements:
1111
pip install vllm
1212
13+
Note:
14+
vLLM runs a DeepGEMM (FP8) kernel warmup at engine startup even for
15+
non-FP8 quantization such as mixed-precision GPTQ. If ``deep_gemm`` is not
16+
installed this fails with ``RuntimeError: DeepGEMM backend is not available
17+
or outdated``. OneComp-quantized models do not need DeepGEMM, so disable the
18+
FP8 path before running this script::
19+
20+
export VLLM_USE_DEEP_GEMM=0
21+
export VLLM_DEEP_GEMM_WARMUP=skip
22+
23+
See docs/user-guide/vllm-inference.md (Troubleshooting) for details.
24+
1325
Copyright 2025-2026 Fujitsu Ltd.
1426
1527
Author: Keiji Kimura

example/vllm_inference/example_gptq_vllm_inference.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,18 @@
1111
Requirements:
1212
pip install vllm
1313
14+
Note:
15+
vLLM runs a DeepGEMM (FP8) kernel warmup at engine startup even for
16+
non-FP8 quantization such as GPTQ. If ``deep_gemm`` is not installed this
17+
fails with ``RuntimeError: DeepGEMM backend is not available or outdated``.
18+
OneComp-quantized models do not need DeepGEMM, so disable the FP8 path
19+
before running this script::
20+
21+
export VLLM_USE_DEEP_GEMM=0
22+
export VLLM_DEEP_GEMM_WARMUP=skip
23+
24+
See docs/user-guide/vllm-inference.md (Troubleshooting) for details.
25+
1426
Copyright 2025-2026 Fujitsu Ltd.
1527
1628
Author: Keiji Kimura

example/vllm_inference/example_jointq_vllm_inference.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,18 @@
2222
Requirements:
2323
pip install vllm
2424
25+
Note:
26+
vLLM runs a DeepGEMM (FP8) kernel warmup at engine startup even for
27+
non-FP8 quantization. If ``deep_gemm`` is not installed this fails with
28+
``RuntimeError: DeepGEMM backend is not available or outdated``.
29+
OneComp-quantized models do not need DeepGEMM, so disable the FP8 path
30+
before running this script::
31+
32+
export VLLM_USE_DEEP_GEMM=0
33+
export VLLM_DEEP_GEMM_WARMUP=skip
34+
35+
See docs/user-guide/vllm-inference.md (Troubleshooting) for details.
36+
2537
Copyright 2025-2026 Fujitsu Ltd.
2638
2739
Author: Keiji Kimura

0 commit comments

Comments
 (0)