Skip to content

Commit cb049b2

Browse files
committed
docs(backends): note AMD-native quant lanes, GPTQ/AWQ/Marlin unsupported
design-rocm.md #15. Adds a concise ROCm/AMD subsection recording the ecosystem finding from design-rocm.md §A.2: GGUF (q4_0/q4_K/q8_0), FP8 (W8A8), and MXFP4 (via AMD Quark) are the AMD-native quant lanes, while GPTQ/AWQ/Marlin have no ROCm equivalent — Marlin, the fast kernel both lean on, is hand-written Nvidia tensor-core PTX. Cites design-rocm.md §A.2 and the vLLM ROCm quantization compatibility matrix, so future ROCm quant work doesn't get scoped as "port GPTQ to HIP" against the ecosystem's own grain. Co-Authored-By: Virgil <virgil@lethean.io>
1 parent 4969250 commit cb049b2

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

docs/backends.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,27 @@ Path `engine/hip`, package `hip`. The default `linux && amd64` build is native-f
4545

4646
Registers as `"rocm"`. GGUF loading works; safetensors model-pack loading is **not yet available** in the current quarantine landing (blocked on a missing upstream package — the load fails with an explicit message rather than guessing).
4747

48+
#### Quantisation lanes on AMD
49+
50+
Not every quantisation format is equally "native" on ROCm, and this shapes
51+
what a ROCm quant contribution should target. The AMD-native lanes are:
52+
53+
- **GGUF** (`q4_0`, `q4_K`, `q8_0`) — MLX-style group-affine quantisation;
54+
`engine/hip`'s existing `rocm_mlx_q4_projection` kernel family already
55+
serves this.
56+
- **FP8** (W8A8, via llm-compressor / AMD Quark) — 8-bit weight+activation.
57+
- **MXFP4** (via AMD Quark) — OCP microscaled 4-bit (`model/quant/mxfp4`),
58+
hardware-accelerated from CDNA4/MI350 onward.
59+
60+
**GPTQ, AWQ, and Marlin are unsupported on AMD** — this is not a gap to
61+
close by porting a kernel, it is the ecosystem's own shape: Marlin, the fast
62+
kernel both GPTQ and AWQ lean on, is hand-written Nvidia tensor-core PTX with
63+
no ROCm equivalent. Scoping ROCm quant work as "port GPTQ to HIP" swims
64+
against that grain; the native-feeling targets are GGUF, FP8, and MXFP4
65+
above. See `docs/design-rocm.md` §A.2 and the
66+
[vLLM ROCm quantization compatibility matrix](https://docs.vllm.ai/en/latest/features/quantization/)
67+
for the full picture.
68+
4869
### About `llama_cpp`
4970

5071
`llama_cpp` is still a slot in the preference order, but **no package in this repository registers it** as an `inference.Backend`. The serving layer provides `serving.HTTPBackend` (name `"http"`) and `serving.LlamaBackend` (name `"llama"`) that wrap an external llama.cpp HTTP server as a `TextModel` — but these are serving-level adapters, not registered inference backends.

0 commit comments

Comments
 (0)