[GPU] Support non-transposed INT4 compressed weights in FullyConnected for shared weights scenario#36845
Open
kwieloch-intel wants to merge 4 commits into
Open
Conversation
kwieloch-intel
force-pushed
the
non_transposed_weights
branch
from
July 14, 2026 17:09
07f3a86 to
f35396b
Compare
kwieloch-intel
marked this pull request as ready for review
July 15, 2026 12:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes compilation failure when shared weights feature (
ep.share_ep_contexts) is enabled for models with non-transposed INT4 compressed weights.Ticket: CVS-186154
Problem Description
When the shared weights feature is enabled, model weights are represented as
Parameternodes instead ofConstantnodes. TheConvertMatMulToFullyConnectedtransformation inserts a runtimeTransposenode to convert weights from[K, N]to[N, K]layout (required by the FC primitive's default transposed-weights assumption).For constant weights, this transpose is folded at compile time. For parameter weights, it must execute at runtime, but the GPU permute kernel does not support INT4 data types, causing a compilation failure:
Implemented Solution
Instead of transposing INT4 parameter weights at runtime, pass them directly in non-transposed layout (
[K, N]) and let oneDNN handle this format natively viaformat_tag::ab. The infrastructure forweights_transposed=falsealready existed in the GPU plugin but was never triggered for compressed weights.Main Changes
1.
convert_matmul_to_fc.cpp: Skip transpose for parameter INT4 weightsWhen compressed weights originate from a
Parameternode and the originalMatMulhastranspose_b=false, use the non-transposed FC path (transpose_b=false) instead of inserting a Transpose node.2.
fully_connected_onednn.cpp: Fix decompression scale/zero-point dimension indexThe decompression scale group size calculation assumed weights are always in transposed layout
[N, K], usingget_dim(weight_rank - 1)to find the IFM (K) dimension. For non-transposed weights[K, N], K is atget_dim(weight_rank - 2).3.
remote_tensor.cpp: Enable sub-byte tensor copy for shared weightsThe remote tensor copy path rejected sub-byte types (i4/u4) with an assertion. When shared weights are copied to GPU memory, this blocked INT4 parameter weights. Added flat contiguous copy path for sub-byte types.
4.
convert_matmul_to_fc_test.cpp: Add minimal unit testMinimal unit test to verify that Parameter INT4 compressed weights skip the Transpose and produce FC with
transpose_b=false.Reproduction Steps
A detailed description is available in the description section of the JIRA ticket: CVS-186154.
AI assistance:
Note
AI assistance used to implement fix and unit test. The generated code were verified for correctness, manually built, executed, and fixed as needed.