Add Ascend NPU support (#2831)

* Add Ascend NPU support

* Document Ascend NPU CUDA API equivalents

* Validate torch NPU qlinear parity

* Add NPU-native GPTQ Hessian inverse

* Add torch QQQ backend for NPU

* Prepare README for 7.0.0 release

* Address NPU quality review comments

* Clean up 7.0 README release note

* Fix device normalization in kernel auto selection

Author: Qubitium

README.md

@@ -4,7 +4,7 @@
 </div>
 <h1 align="center">GPT-QModel</h1>
 </p>
-<p align="center">LLM model quantization (compression) toolkit with hw acceleration support for NVIDIA CUDA, AMD ROCm, Intel XPU, and Intel/AMD/Apple CPUs via HF, vLLM, and SGLang.</p>
+<p align="center">LLM model quantization (compression) toolkit with hw acceleration support for NVIDIA CUDA, AMD ROCm, Huawei Ascend NPU, Intel XPU, and Intel/AMD/Apple CPUs via HF, vLLM, and SGLang.</p>
 <p align="center">
     <a href="https://github.com/ModelCloud/GPTQModel/releases" style="text-decoration:none;"><img alt="GitHub release" src="https://img.shields.io/github/release/ModelCloud/GPTQModel.svg"></a>
     <a href="https://pypi.org/project/gptqmodel/" style="text-decoration:none;"><img alt="PyPI - Version" src="https://img.shields.io/pypi/v/gptqmodel"></a>
@@ -20,8 +20,7 @@
 </p>
 
 ## Latest News
-* 04/27/2026 6.1.0-dev `main`: ✨ Added `internvl_chat` model support.
-* 04/23/2026 6.1.0-dev `main`: ✨ Added `gemma3n`、`GLM-OCR`、`GLM-ASR` and `falcon_mamba` model support.
+* 04/28/2026 [7.0.0](https://github.com/ModelCloud/GPTQModel/releases/tag/v7.0.0): 🚀 Added Huawei Ascend NPU support through native torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3. Added `internvl_chat`, `gemma3n`, `GLM-OCR`, `GLM-ASR`, and `falcon_mamba` model support.
 * 04/16/2026 [6.1.0](https://github.com/ModelCloud/GPTQModel/releases/tag/v6.1.0): 🚀🔥⚡ CUDA kernels are now fully JIT-compiled, shrinking the wheel by about 300x and building only what you use; Marlin now supports NVIDIA `Turing+` GPUs, Machete kernel validation now covers supported GPUs, `GLM 5/5.1` joins the lineup, and LazyTurtle plus AWQ / multi-GPU MoE fixes make large-model quantization easier, lighter, and smoother.
 * 04/03/2026 [6.0.3](https://github.com/ModelCloud/GPTQModel/releases/tag/v6.0.3): 🎉 New quantization methods: `ParoQuant`, `GGUF`, `FP8`, `EXL3`, and `FOEM: First-Order Error Matters`. Added PrismML/Bonsai 1bit model quantization (inference only), faster ParoQuant/AWQ kernels, ParoQuant `optimization scope` control: `module` (Paro Lite) or `layer` (Paro reference), plus `Gemma4`, `MiniCPM-O`, `MiniCPM-V`, and `GLM4 MoE Lite` model support.
 * 03/19/2026 [5.8.0](https://github.com/ModelCloud/GPTQModel/releases/tag/v5.8.0): ✨HF Transformers 5.3.0 support with auto-defusing of `fused` models via pypi pkg: [Defuser](https://github.com/ModelCloud/Defuser). Qwen 3.5 family support added. New fast HF `cpu` kernels for GPTQ/AWQ added. Experimental INT8 `cpu` kernel added for GPTQ. 
@@ -191,14 +190,14 @@ GPT-QModel is a modular design supporting multiple quantization methods and feat
 
 ### Quant Method / Format / Backend Matrix
 
-Canonical backend names are shown below. Legacy aliases such as `BACKEND.TORCH`, `BACKEND.MARLIN`, `BACKEND.GEMM`, and `BACKEND.PARO` are still accepted and normalized to the matching canonical backend for the selected quant method.
+Canonical backend names are shown below. Method-specific aliases are only accepted where explicitly implemented by that quant method.
 
 | Quant Method | Formats | Backends / Kernels |
 | --- | --- | --- |
 | `METHOD.GPTQ` | `FORMAT.GPTQ`, `FORMAT.GPTQ_V2`, `FORMAT.MARLIN`, `FORMAT.BITBLAS` | `FORMAT.GPTQ`: `BACKEND.GPTQ_TORCH_ATEN`, `BACKEND.GPTQ_MACHETE`, `BACKEND.GPTQ_MARLIN`, `BACKEND.GPTQ_EXLLAMA_V2`, `BACKEND.GPTQ_TORCH_FUSED`, `BACKEND.GPTQ_TRITON`, `BACKEND.GPTQ_BITBLAS`, `BACKEND.GPTQ_TORCH`, `BACKEND.GPTQ_TORCH_INT8`<br>`FORMAT.GPTQ_V2`: `BACKEND.GPTQ_TORCH_ATEN`, `BACKEND.GPTQ_EXLLAMA_V2`, `BACKEND.GPTQ_TORCH_FUSED`, `BACKEND.GPTQ_TRITON`, `BACKEND.GPTQ_BITBLAS`, `BACKEND.GPTQ_TORCH`, `BACKEND.GPTQ_TORCH_INT8`<br>`FORMAT.MARLIN`: `BACKEND.GPTQ_MARLIN`<br>`FORMAT.BITBLAS`: `BACKEND.GPTQ_BITBLAS` |
 | `METHOD.AWQ` | `FORMAT.GEMM`, `FORMAT.GEMV`, `FORMAT.GEMV_FAST`, `FORMAT.LLM_AWQ`, `FORMAT.MARLIN`, `FORMAT.BITBLAS` | `FORMAT.GEMM`: `BACKEND.AWQ_TORCH_ATEN`, `BACKEND.AWQ_MACHETE`, `BACKEND.AWQ_MARLIN`, `BACKEND.AWQ_EXLLAMA_V2`, `BACKEND.AWQ_GEMM`, `BACKEND.AWQ_GEMM_TRITON`, `BACKEND.AWQ_TORCH_FUSED`, `BACKEND.AWQ_TORCH`, `BACKEND.AWQ_TORCH_INT8`, `BACKEND.AWQ_BITBLAS`<br>`FORMAT.GEMV`: `BACKEND.AWQ_GEMV`<br>`FORMAT.GEMV_FAST`: `BACKEND.AWQ_GEMV_FAST`<br>`FORMAT.LLM_AWQ`: `BACKEND.AWQ_GEMV_FAST`<br>`FORMAT.MARLIN`: `BACKEND.AWQ_MACHETE`, `BACKEND.AWQ_MARLIN`<br>`FORMAT.BITBLAS`: `BACKEND.AWQ_BITBLAS` |
 | `METHOD.PARO` | `FORMAT.PAROQUANT` | `BACKEND.PAROQUANT_CUDA`, `BACKEND.PAROQUANT_TRITON` |
-| `METHOD.QQQ` | `FORMAT.QQQ` | `BACKEND.QQQ` |
+| `METHOD.QQQ` | `FORMAT.QQQ` | `BACKEND.QQQ`, `BACKEND.QQQ_TORCH` |
 | `METHOD.GGUF` | `FORMAT.GGUF` | `BACKEND.GGUF_TRITON`, `BACKEND.GGUF_CPP_CUDA`, `BACKEND.GGUF_CPP_CPU`, `BACKEND.GGUF_TORCH` |
 | `METHOD.FP8` | `FORMAT.FP8` | `BACKEND.FP8_TORCH` |
 | `METHOD.BITSANDBYTES` | `FORMAT.BITSANDBYTES` | `BACKEND.BITSANDBYTES` |
@@ -216,7 +215,7 @@ Marlin uses `GPTQMODEL_MARLIN_USE_FP32` (default: enabled) to control fp32 accum
 * 🚀 Quantize MoE models with ease even with extreme routing activation bias via `Moe.Routing` and/or `FailSafe`.
 * 🚀 Data Parallelism for 80%+ quantization speed reduction with Multi-GPU.
 * 🚀 Optimized for Python >= 3.13t (free threading) with lock-free threading.
-* ✨ Linux, macOS, Windows platform support for CUDA (NVIDIA), XPU (Intel), ROCm (AMD), MPS (Apple Silicon), CPU (Intel/AMD/Apple Silicon).
+* ✨ Linux, macOS, Windows platform support for CUDA (NVIDIA), NPU (Huawei Ascend), XPU (Intel), ROCm (AMD), MPS (Apple Silicon), CPU (Intel/AMD/Apple Silicon).
 * ✨ `Dynamic` per-module mixed quantization control: each layer/module can have a unique quantization config or be excluded from quantization. 
 * 🚀 Intel Torch 2.8 fused kernel support for XPU [`Arc` + `Datacenter Max`] and CPU [`avx`, `amx`].
 * 🚀 Python 3.13.3t (free-threading, GIL disabled) support for multi-GPU accelerated quantization for MoE models and multi-core CPU boost for packing.

gptqmodel/__init__.py

@@ -173,6 +173,7 @@ def _build_device_thread_pool():
         workers={
             "cuda:per": 4,
             "xpu:per": 1,
+            "npu:per": 1,
             "mps": 8,
             "cpu": min(12, max(1, (os.cpu_count() or 1) + 1 // 2)),  # count + 1, fixed pool size > 1 check when count=3
             "model_loader:cpu": 2,

gptqmodel/looper/forward_executor.py

@@ -531,7 +531,7 @@ def _replica_progress(idx: int, total: int, device: torch.device, step: str) ->
                     replica = module_replicas[device]
                     submitter = (
                         device_thread_pool.submit_serial
-                        if device.type in ("cuda", "xpu", "mps")
+                        if device.type in ("cuda", "xpu", "npu", "mps")
                         else device_thread_pool.submit
                     )
 

gptqmodel/looper/gptq_processor.py

@@ -24,6 +24,7 @@
 from ..utils.device import get_device
 from ..utils.model import create_quant_module, find_modules, pack_module
 from ..utils.module_locks import parent_module_lock
+from ..utils.torch import HAS_NPU
 
 log = setup_logger()
 lock = threading.Lock()
@@ -276,6 +277,12 @@ def process(
                     f"CUDA thread context {current_cuda_device} does not match expected device {expected_device} "
                     f"while processing '{module.full_name}'."
                 )
+            if expected_device.type == "npu" and HAS_NPU:
+                current_npu_device = torch.device("npu", torch.npu.current_device())
+                assert current_npu_device == expected_device, (
+                    f"NPU thread context {current_npu_device} does not match expected device {expected_device} "
+                    f"while processing '{module.full_name}'."
+                )
 
         wq, q_scales, q_zeros, q_g_idx, duration, avg_loss, damp_percent, nsamples = g.quantize()
 

gptqmodel/looper/loop_processor.py

@@ -26,7 +26,7 @@
 from ..quantization.config import QuantizeConfig
 from ..utils.colors import ANSIColor, color_text
 from ..utils.logger import setup_logger
-from ..utils.torch import CPU, DEVICE_0, DEVICE_1
+from ..utils.torch import CPU, DEVICE_0, DEVICE_1, HAS_NPU
 
 log = setup_logger()
 
@@ -524,6 +524,13 @@ def _discover_accelerator_devices(self) -> List[str]:
             except Exception:  # pragma: no cover - defensive, XPU runtime differences
                 pass
 
+        if HAS_NPU:
+            try:
+                for idx in range(torch.npu.device_count()):
+                    devices.append(f"npu:{idx}")
+            except Exception:  # pragma: no cover - defensive, NPU runtime differences
+                pass
+
         return devices
 
     def _safe_query_metric(self, device_key: str, handle: Device):

gptqmodel/looper/qqq_processor.py

@@ -12,12 +12,14 @@
 from ..looper.loop_processor import DTYPE_SIZE_COLUMN, ExecutionConfig, MODULE_FEATURE_COLUMN, LoopProcessor
 from ..looper.named_module import NamedModule
 from ..models import BaseQModel
+from ..models._const import DEVICE
 from ..models.writer import (PROCESS_LOG_FWD_TIME, PROCESS_LOG_LAYER, PROCESS_LOG_MODULE, PROCESS_LOG_NAME,
                              PROCESS_LOG_TIME, QUANT_LOG_DAMP, QUANT_LOG_LOSS, QUANT_LOG_NSAMPLES)
-from ..nn_modules.qlinear.qqq import QQQLinear
+from ..nn_modules.qlinear.qqq import QQQLinear, QQQTorchLinear
 from ..quantization.config import METHOD, QuantizeConfig, resolve_quant_format
 from ..utils.fallback import normalize_fallback
 from ..quantization.qqq import QQQ
+from ..utils.backend import BACKEND
 from ..utils.logger import setup_logger, log_time_block
 from ..utils.model import create_quant_module, find_modules, move_to, pack_module
 from ..utils.torch import CPU
@@ -57,6 +59,16 @@ def __init__(
         self.calculate_w_wq_diff = calculate_w_wq_diff
         self.avg_losses = []
 
+    def _quant_linear_kernel(self):
+        device = self.qcfg.device
+        if isinstance(device, DEVICE):
+            return (QQQTorchLinear, BACKEND.QQQ_TORCH) if device == DEVICE.NPU else (QQQLinear, BACKEND.QQQ)
+        if isinstance(device, torch.device):
+            return (QQQTorchLinear, BACKEND.QQQ_TORCH) if device.type == "npu" else (QQQLinear, BACKEND.QQQ)
+        if isinstance(device, str) and device.split(":")[0].lower() == "npu":
+            return QQQTorchLinear, BACKEND.QQQ_TORCH
+        return QQQLinear, BACKEND.QQQ
+
     def set_calibration_dataset(self, calibration_dataset):
         """Rejects dataset replacement because QQQ capture is fixed at construction."""
 
@@ -252,6 +264,7 @@ def submodule_finalize(self, module: NamedModule, model: BaseQModel, **kwargs):
 
         layers = find_modules(model.model)
         module_label = getattr(module, "full_name", getattr(module, "name", ""))
+        quant_linear_cls, backend = self._quant_linear_kernel()
 
         # replace module with quantized module
         with log_time_block(
@@ -261,7 +274,7 @@ def submodule_finalize(self, module: NamedModule, model: BaseQModel, **kwargs):
         ):
             create_quant_module(
                 name=module.full_name,
-                linear_cls=QQQLinear,
+                linear_cls=quant_linear_cls,
                 bits=self.qcfg.runtime_bits,
                 desc_act=self.qcfg.desc_act,
                 dynamic=self.qcfg.dynamic,
@@ -273,13 +286,14 @@ def submodule_finalize(self, module: NamedModule, model: BaseQModel, **kwargs):
                 lm_head_name=model.lm_head,
                 pack_dtype=self.qcfg.pack_dtype,
                 format=resolve_quant_format(self.qcfg.format, self.qcfg.method),
+                backend=backend,
                 register_buffers=False,
             )
 
         # pack module
         qModules = {
             name: submodule
-            for name, submodule in find_modules(model.model, [QQQLinear]).items()
+            for name, submodule in find_modules(model.model, [quant_linear_cls]).items()
             if name == module.full_name
         }
         with log_time_block(
@@ -294,7 +308,7 @@ def submodule_finalize(self, module: NamedModule, model: BaseQModel, **kwargs):
                 q_zeros=q_zeros,
                 q_g_idx=q_g_idx,
                 layers=layers,
-                quant_linear_cls=QQQLinear,
+                quant_linear_cls=quant_linear_cls,
                 lock=self.lock,
                 q_scales_extra=q_scales_extra,
                 quantize_config=self.qcfg,

gptqmodel/models/_const.py

@@ -13,7 +13,14 @@
 
 from ..utils import BACKEND
 from ..utils.rocm import IS_ROCM
-from ..utils.torch import HAS_CUDA, HAS_MPS, HAS_XPU
+from ..utils.torch import HAS_CUDA, HAS_MPS, HAS_NPU, HAS_XPU
+
+
+def _optional_torch_device(spec: str, fallback: torch.device) -> torch.device:
+    try:
+        return device(spec)
+    except (RuntimeError, ValueError):
+        return fallback
 
 
 CPU = device("cpu")
@@ -22,6 +29,8 @@
 CUDA_0 = device("cuda:0")
 XPU = device("xpu")
 XPU_0 = device("xpu:0")
+NPU = _optional_torch_device("npu", CPU)
+NPU_0 = _optional_torch_device("npu:0", CPU)
 MPS = device("mps")
 ROCM = device("cuda:0")  # rocm maps to fake cuda
 
@@ -34,6 +43,7 @@ class DEVICE(str, Enum):
     CPU = "cpu"  # All CPU: Optimized for IPEX is CPU has AVX, AVX512, AMX, or XMX instructions
     CUDA = "cuda"  # Nvidia GPU: Optimized for Ampere+
     XPU = "xpu"  # Intel GPU: Datacenter Max + Arc
+    NPU = "npu"  # Ascend NPU: portable Torch kernels
     MPS = "mps"  # MacOS GPU: Apple Silicon/Metal)
     ROCM = "rocm"  # AMD GPU: ROCm maps to fake cuda
 
@@ -54,7 +64,7 @@ def type(self) -> str:
     @property
     def index(self) -> int | None:
         """Default index used when materialising a torch.device from this enum."""
-        if self in (DEVICE.CUDA, DEVICE.ROCM, DEVICE.XPU):
+        if self in (DEVICE.CUDA, DEVICE.ROCM, DEVICE.XPU, DEVICE.NPU):
             return 0
         return None
 
@@ -96,6 +106,8 @@ def normalize_device(type_value: str | DEVICE | int | torch.device) -> DEVICE:
             return DEVICE.CUDA
         elif HAS_XPU:
             return DEVICE.XPU
+        elif HAS_NPU:
+            return DEVICE.NPU
         elif HAS_MPS:
             return DEVICE.MPS
         else:
@@ -123,6 +135,8 @@ def get_best_device(backend: BACKEND = BACKEND.AUTO) -> torch.device:
         return CUDA_0
     elif HAS_XPU:
         return XPU_0
+    elif HAS_NPU:
+        return NPU_0
     elif HAS_MPS:
         return MPS
     else:

gptqmodel/models/base.py

@@ -772,12 +772,20 @@ def quantize(
 
         preferred_backend = requested_backend
         if preferred_backend in (None, BACKEND.AUTO):
+            quant_device = self.quantize_config.device
+            if isinstance(quant_device, DEVICE):
+                quant_device_type = quant_device.type
+            elif isinstance(quant_device, torch.device):
+                quant_device_type = quant_device.type
+            else:
+                quant_device_type = str(quant_device).split(":")[0].lower()
+
             if export_quant_method == METHOD.AWQ:
                 if format_code == FORMAT.GEMM:
                     # Weight-only RTN->AWQ export should stay on the portable torch kernel.
                     preferred_backend = (
                         BACKEND.AWQ_TORCH
-                        if self.quantize_config.uses_weight_only_lifecycle()
+                        if self.quantize_config.uses_weight_only_lifecycle() or quant_device_type == "npu"
                         else BACKEND.AWQ_GEMM
                     )
                 elif format_code == FORMAT.BITBLAS:
@@ -789,7 +797,7 @@ def quantize(
                 else:
                     raise ValueError(f"Unsupported FORMAT: `{self.quantize_config.format}` with `METHOD.AWQ`")
             elif self.quantize_config.method == METHOD.QQQ:
-                preferred_backend = BACKEND.QQQ
+                preferred_backend = BACKEND.QQQ_TORCH if quant_device_type == "npu" else BACKEND.QQQ
             elif self.quantize_config.method == METHOD.PARO:
                 preferred_backend = BACKEND.PAROQUANT_CUDA
             elif self.quantize_config.method == METHOD.EXL3:

gptqmodel/models/loader.py

@@ -75,7 +75,7 @@
     make_quant,
     simple_dispatch_model,
 )
-from ._const import DEVICE, normalize_device
+from ._const import DEVICE, HAS_NPU, normalize_device
 
 
 log = setup_logger()
@@ -1161,6 +1161,11 @@ def build_layerwise_device_map(
                     device_strs = [f"xpu:{i}" for i in range(num_gpus)]
                 else:
                     raise RuntimeError("XPU is not available")
+            elif device == DEVICE.NPU:
+                if HAS_NPU:
+                    device_strs = [f"npu:{i}" for i in range(num_gpus)]
+                else:
+                    raise RuntimeError("NPU is not available")
             else:
                 device_strs = ["cpu"] * num_gpus
 
@@ -1311,6 +1316,8 @@ def assign(mod, device_id):
                 num_gpus = torch.cuda.device_count()
             elif device is DEVICE.XPU:
                 num_gpus = torch.xpu.device_count()
+            elif device is DEVICE.NPU:
+                num_gpus = torch.npu.device_count()
             device_map = build_layerwise_device_map(model, device, layers, ignore_modules, num_gpus)
         else:
             device_map = dict(explicit_device_map)

gptqmodel/nn_modules/exllamav3_torch.py

@@ -26,6 +26,33 @@
 _EXL3_LOP3_BIAS = 0x3B603B60
 
 
+def _tensor_core_perm_values() -> list[int]:
+    perm = [0] * 256
+    for t in range(32):
+        r0 = (t % 4) * 2
+        r1 = r0 + 1
+        r2 = r0 + 8
+        r3 = r0 + 9
+        c0 = t // 4
+        c1 = c0 + 8
+        perm[t * 8 + 0] = r0 * 16 + c0
+        perm[t * 8 + 1] = r1 * 16 + c0
+        perm[t * 8 + 2] = r2 * 16 + c0
+        perm[t * 8 + 3] = r3 * 16 + c0
+        perm[t * 8 + 4] = r0 * 16 + c1
+        perm[t * 8 + 5] = r1 * 16 + c1
+        perm[t * 8 + 6] = r2 * 16 + c1
+        perm[t * 8 + 7] = r3 * 16 + c1
+    return perm
+
+
+def _inverse_perm_values(perm: list[int]) -> list[int]:
+    inv = [0] * len(perm)
+    for index, value in enumerate(perm):
+        inv[value] = index
+    return inv
+
+
 def _half_scalar_from_bits(bits: int) -> float:
     # Convert a uint16 bit pattern to its IEEE-754 binary16 (float16) value
     # without allocating a torch tensor. The previous implementation used
@@ -50,29 +77,13 @@ def _half_scalar_from_bits(bits: int) -> float:
 @lru_cache(maxsize=None)
 def _tensor_core_perm(device_type: str, device_index: int | None) -> torch.Tensor:
     device = torch.device(device_type, device_index)
-    perm = [0] * 256
-    for t in range(32):
-        r0 = (t % 4) * 2
-        r1 = r0 + 1
-        r2 = r0 + 8
-        r3 = r0 + 9
-        c0 = t // 4
-        c1 = c0 + 8
-        perm[t * 8 + 0] = r0 * 16 + c0
-        perm[t * 8 + 1] = r1 * 16 + c0
-        perm[t * 8 + 2] = r2 * 16 + c0
-        perm[t * 8 + 3] = r3 * 16 + c0
-        perm[t * 8 + 4] = r0 * 16 + c1
-        perm[t * 8 + 5] = r1 * 16 + c1
-        perm[t * 8 + 6] = r2 * 16 + c1
-        perm[t * 8 + 7] = r3 * 16 + c1
-    return torch.tensor(perm, dtype=torch.long, device=device)
+    return torch.tensor(_tensor_core_perm_values(), dtype=torch.long, device=device)
 
 
 @lru_cache(maxsize=None)
 def _tensor_core_perm_i(device_type: str, device_index: int | None) -> torch.Tensor:
-    perm = _tensor_core_perm(device_type, device_index)
-    return torch.argsort(perm)
+    device = torch.device(device_type, device_index)
+    return torch.tensor(_inverse_perm_values(_tensor_core_perm_values()), dtype=torch.long, device=device)
 
 
 @lru_cache(maxsize=None)