fix(qlinear): size pack_original qweight/qzeros by bits/32 for 3-bit GPTQ - #2984
Conversation
…GPTQ The previous pack_factor shortcut (32 // bits) gives 10 for 3-bit, but the 3-bit custom packing loop consumes 32 values per 3 rows. Using math.ceil(dim / pack_factor) over-allocated the output buffer and made the loop overrun. Size packed tensors by math.ceil(dim * bits / 32) instead, which is exact for 2/4/8/3-bit layouts.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔍 3-bit while loop relies on in_features being a multiple of 32
The 3-bit branch (gptqmodel/nn_modules/qlinear/__init__.py:1404-1425) advances row by 3 per iteration and consumes 32 input rows per iteration. This only terminates cleanly when qweight.shape[0] is a multiple of 3, i.e. when int_weight.shape[0] is a multiple of 32. With the new ceil(dim*3/32) sizing, a non-multiple-of-32 dim would produce a shape[0] that is not a multiple of 3, and the while loop could read past int_weight or write past qweight. This precondition (32-alignment of in_features) is enforced elsewhere for the block/gpu packers (:991), and 3-bit only supports int32, so this is likely safe in practice, but worth confirming that callers of pack_original always pad in_features to a multiple of 32 for 3-bit.
(Refers to lines 1404-1425)
Was this helpful? React with 👍 or 👎 to provide feedback.
| int_weight = int_weight.numpy().astype(self.pack_np_math_dtype) | ||
|
|
||
| qweight = np.zeros((math.ceil(int_weight.shape[0] / self.pack_factor), int_weight.shape[1]), | ||
| qweight = np.zeros((math.ceil(int_weight.shape[0] * self.bits / 32), int_weight.shape[1]), |
There was a problem hiding this comment.
🔴 Packed weight buffers become half-sized and drop data for 16-bit and 8-bit storage formats
The packed weight and zero-point buffers are sized using a hardcoded 32 (math.ceil(int_weight.shape[0] * self.bits / 32) at gptqmodel/nn_modules/qlinear/__init__.py:1398 and the matching line at :1431) instead of the actual storage-word width, so when weights are stored in a 16-bit or 8-bit container the buffer is allocated too small and only part of the weights get written.
Impact: Models packed with a non-32-bit storage format silently lose weight data, producing corrupted/incorrect outputs.
Why hardcoding 32 diverges from pack_dtype_bits for the 2/4/8-bit path
The old code sized these buffers as math.ceil(dim / self.pack_factor) where self.pack_factor = self.pack_dtype_bits // self.bits (gptqmodel/nn_modules/qlinear/__init__.py:659). That equals ceil(dim * bits / pack_dtype_bits). Replacing pack_dtype_bits with the literal 32 is only equivalent when pack_dtype_bits == 32.
TorchQuantLinear (gptqmodel/nn_modules/qlinear/torch.py:129) and TritonV2QuantLinear declare SUPPORTS_PACK_DTYPES = [torch.int8, torch.int16, torch.int32], and these classes use this base pack_original. For e.g. pack_dtype=int16, bits=4: old size = dim/4 rows, new size = dim*4/32 = dim/8 rows — half. The 2/4/8 loop then only iterates qweight.shape[0] rows, packing just the first half of int_weight and dropping the rest.
This also mismatches the pre-registered buffer shape (_register_gptq_buffers at :753 uses ceil(in_features / self.pack_factor)) and the dequantize path (dequantize_weight at :866-879), both of which key off pack_factor/pack_dtype_bits.
Using self.pack_dtype_bits instead of 32 fixes all cases: 2/4/8-bit at any supported width, and 3-bit (which only supports int32, so pack_dtype_bits == 32).
| qweight = np.zeros((math.ceil(int_weight.shape[0] * self.bits / 32), int_weight.shape[1]), | |
| qweight = np.zeros((math.ceil(int_weight.shape[0] * self.bits / self.pack_dtype_bits), int_weight.shape[1]), |
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
| zeros = zeros.numpy().astype(self.pack_np_math_dtype) | ||
| qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] / self.pack_factor)), dtype=self.pack_np_math_dtype) | ||
| qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] * self.bits / 32)), dtype=self.pack_np_math_dtype) |
There was a problem hiding this comment.
🔴 Packed zero-point buffer becomes half-sized and drops data for 16-bit and 8-bit storage formats
The packed zero-point buffer is sized using a hardcoded 32 (math.ceil(zeros.shape[1] * self.bits / 32) at gptqmodel/nn_modules/qlinear/__init__.py:1431) instead of the actual storage-word width, so when zero points are stored in a 16-bit or 8-bit container the buffer is allocated too small and only part of the zero points get written.
Impact: Models packed with a non-32-bit storage format silently lose zero-point data, producing corrupted/incorrect outputs.
Same hardcoded-32 divergence on the qzeros path
See BUG-0001: self.pack_factor = self.pack_dtype_bits // self.bits (:659), so ceil(dim / pack_factor) equals ceil(dim * bits / pack_dtype_bits), not ceil(dim * bits / 32) when pack_dtype_bits != 32. For int16/int8 pack dtypes supported by TorchQuantLinear/TritonV2QuantLinear, this under-allocates qzeros and the 2/4/8 loop packs only part of the columns. The fix is to use self.pack_dtype_bits instead of 32.
| qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] * self.bits / 32)), dtype=self.pack_np_math_dtype) | |
| qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] * self.bits / self.pack_dtype_bits)), dtype=self.pack_np_math_dtype) |
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
PackableQuantLinear.pack_originalsizes its packedqweightandqzerosbuffers withmath.ceil(dim / self.pack_factor). For 3-bit weights,pack_factor = 32 // 3 = 10, but the 3-bit custom packing loop consumes 32 integer values to produce 3 packed rows (32 × 3 bits = 96 bits). Theceil(dim / 10)allocation therefore over-allocates and the loop overruns the input array.Changes
pack_originalqweightandqzeroswithmath.ceil(dim * self.bits / 32)instead ofmath.ceil(dim / self.pack_factor).32 / bitsis an integer) and exact for the 3-bit custom layout.Verification
ruff check gptqmodel/nn_modules/qlinear/__init__.py: passedgit diff --check: passedpytest tests/test_pack.py -q: 18 passed (covers 3-bitpack_original)