Skip to content

fix(qlinear): size pack_original qweight/qzeros by bits/32 for 3-bit GPTQ #2984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Qubitium merged 1 commit into from
Jul 26, 2026
+2 −2
Merged

fix(qlinear): size pack_original qweight/qzeros by bits/32 for 3-bit GPTQ #2984

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions gptqmodel/nn_modules/qlinear/__init__.py
View file
Open in desktop

@devin-ai-integration devin-ai-integration Bot Jul 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 3-bit while loop relies on in_features being a multiple of 32

The 3-bit branch (gptqmodel/nn_modules/qlinear/__init__.py:1404-1425) advances row by 3 per iteration and consumes 32 input rows per iteration. This only terminates cleanly when qweight.shape[0] is a multiple of 3, i.e. when int_weight.shape[0] is a multiple of 32. With the new ceil(dim*3/32) sizing, a non-multiple-of-32 dim would produce a shape[0] that is not a multiple of 3, and the while loop could read past int_weight or write past qweight. This precondition (32-alignment of in_features) is enforced elsewhere for the block/gpu packers (:991), and 3-bit only supports int32, so this is likely safe in practice, but worth confirming that callers of pack_original always pad in_features to a multiple of 32 for 3-bit.

(Refers to lines 1404-1425)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -1395,7 +1395,7 @@ def pack_original(self, linear: nn.Module, scales: t.Tensor, zeros: t.Tensor, g_
int_weight = int_weight.to(t.int32).T.contiguous()
int_weight = int_weight.numpy().astype(self.pack_np_math_dtype)

qweight = np.zeros((math.ceil(int_weight.shape[0] / self.pack_factor), int_weight.shape[1]),
qweight = np.zeros((math.ceil(int_weight.shape[0] * self.bits / 32), int_weight.shape[1]),

@devin-ai-integration devin-ai-integration Bot Jul 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Packed weight buffers become half-sized and drop data for 16-bit and 8-bit storage formats

The packed weight and zero-point buffers are sized using a hardcoded 32 (math.ceil(int_weight.shape[0] * self.bits / 32) at gptqmodel/nn_modules/qlinear/__init__.py:1398 and the matching line at :1431) instead of the actual storage-word width, so when weights are stored in a 16-bit or 8-bit container the buffer is allocated too small and only part of the weights get written.
Impact: Models packed with a non-32-bit storage format silently lose weight data, producing corrupted/incorrect outputs.

Why hardcoding 32 diverges from pack_dtype_bits for the 2/4/8-bit path

The old code sized these buffers as math.ceil(dim / self.pack_factor) where self.pack_factor = self.pack_dtype_bits // self.bits (gptqmodel/nn_modules/qlinear/__init__.py:659). That equals ceil(dim * bits / pack_dtype_bits). Replacing pack_dtype_bits with the literal 32 is only equivalent when pack_dtype_bits == 32.

TorchQuantLinear (gptqmodel/nn_modules/qlinear/torch.py:129) and TritonV2QuantLinear declare SUPPORTS_PACK_DTYPES = [torch.int8, torch.int16, torch.int32], and these classes use this base pack_original. For e.g. pack_dtype=int16, bits=4: old size = dim/4 rows, new size = dim*4/32 = dim/8 rows — half. The 2/4/8 loop then only iterates qweight.shape[0] rows, packing just the first half of int_weight and dropping the rest.

This also mismatches the pre-registered buffer shape (_register_gptq_buffers at :753 uses ceil(in_features / self.pack_factor)) and the dequantize path (dequantize_weight at :866-879), both of which key off pack_factor/pack_dtype_bits.

Using self.pack_dtype_bits instead of 32 fixes all cases: 2/4/8-bit at any supported width, and 3-bit (which only supports int32, so pack_dtype_bits == 32).

Suggested change
qweight = np.zeros((math.ceil(int_weight.shape[0] * self.bits / 32), int_weight.shape[1]),
qweight = np.zeros((math.ceil(int_weight.shape[0] * self.bits / self.pack_dtype_bits), int_weight.shape[1]),
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

dtype=self.pack_np_math_dtype)
if self.bits in [2, 4, 8]:
for row in range(qweight.shape[0]):
Expand Down Expand Up @@ -1428,7 +1428,7 @@ def pack_original(self, linear: nn.Module, scales: t.Tensor, zeros: t.Tensor, g_
self.register_buffer("qweight", t.from_numpy(qweight.astype(self.pack_np_dtype)))

zeros = zeros.numpy().astype(self.pack_np_math_dtype)
qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] / self.pack_factor)), dtype=self.pack_np_math_dtype)
qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] * self.bits / 32)), dtype=self.pack_np_math_dtype)

@devin-ai-integration devin-ai-integration Bot Jul 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Packed zero-point buffer becomes half-sized and drops data for 16-bit and 8-bit storage formats

The packed zero-point buffer is sized using a hardcoded 32 (math.ceil(zeros.shape[1] * self.bits / 32) at gptqmodel/nn_modules/qlinear/__init__.py:1431) instead of the actual storage-word width, so when zero points are stored in a 16-bit or 8-bit container the buffer is allocated too small and only part of the zero points get written.
Impact: Models packed with a non-32-bit storage format silently lose zero-point data, producing corrupted/incorrect outputs.

Same hardcoded-32 divergence on the qzeros path

See BUG-0001: self.pack_factor = self.pack_dtype_bits // self.bits (:659), so ceil(dim / pack_factor) equals ceil(dim * bits / pack_dtype_bits), not ceil(dim * bits / 32) when pack_dtype_bits != 32. For int16/int8 pack dtypes supported by TorchQuantLinear/TritonV2QuantLinear, this under-allocates qzeros and the 2/4/8 loop packs only part of the columns. The fix is to use self.pack_dtype_bits instead of 32.

Suggested change
qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] * self.bits / 32)), dtype=self.pack_np_math_dtype)
qzeros = np.zeros((zeros.shape[0], math.ceil(zeros.shape[1] * self.bits / self.pack_dtype_bits)), dtype=self.pack_np_math_dtype)
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

if self.bits in [2, 4, 8]:
for col in range(qzeros.shape[1]):
for j in range(self.pack_factor):
Expand Down