Skip to content

[GPU] Support non-transposed INT4 compressed weights in FullyConnected for shared weights scenario#36845

Open
kwieloch-intel wants to merge 4 commits into
openvinotoolkit:masterfrom
kwieloch-intel:non_transposed_weights
Open

[GPU] Support non-transposed INT4 compressed weights in FullyConnected for shared weights scenario#36845
kwieloch-intel wants to merge 4 commits into
openvinotoolkit:masterfrom
kwieloch-intel:non_transposed_weights

Conversation

@kwieloch-intel

@kwieloch-intel kwieloch-intel commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

This PR fixes compilation failure when shared weights feature (ep.share_ep_contexts) is enabled for models with non-transposed INT4 compressed weights.

Ticket: CVS-186154


Problem Description

When the shared weights feature is enabled, model weights are represented as Parameter nodes instead of Constant nodes. The ConvertMatMulToFullyConnected transformation inserts a runtime Transpose node to convert weights from [K, N] to [N, K] layout (required by the FC primitive's default transposed-weights assumption).

For constant weights, this transpose is folded at compile time. For parameter weights, it must execute at runtime, but the GPU permute kernel does not support INT4 data types, causing a compilation failure:

[GPU] Could not find a suitable kernel for transpose:Transpose_130977
params raw string: INT4_BFYX_v1_p0_0_v1_p0_0_v8192_p0_0_v2048_p0_0;INT4_BFYX_v1_p0_0_v1_p0_0_v2048_p0_0_v8192_p0_0

Implemented Solution

Instead of transposing INT4 parameter weights at runtime, pass them directly in non-transposed layout ([K, N]) and let oneDNN handle this format natively via format_tag::ab. The infrastructure for weights_transposed=false already existed in the GPU plugin but was never triggered for compressed weights.

Main Changes

1. convert_matmul_to_fc.cpp: Skip transpose for parameter INT4 weights

When compressed weights originate from a Parameter node and the original MatMul has transpose_b=false, use the non-transposed FC path (transpose_b=false) instead of inserting a Transpose node.

2. fully_connected_onednn.cpp: Fix decompression scale/zero-point dimension index

The decompression scale group size calculation assumed weights are always in transposed layout [N, K], using get_dim(weight_rank - 1) to find the IFM (K) dimension. For non-transposed weights [K, N], K is at get_dim(weight_rank - 2).

3. remote_tensor.cpp: Enable sub-byte tensor copy for shared weights

The remote tensor copy path rejected sub-byte types (i4/u4) with an assertion. When shared weights are copied to GPU memory, this blocked INT4 parameter weights. Added flat contiguous copy path for sub-byte types.

4. convert_matmul_to_fc_test.cpp: Add minimal unit test

Minimal unit test to verify that Parameter INT4 compressed weights skip the Transpose and produce FC with transpose_b=false.

Reproduction Steps

A detailed description is available in the description section of the JIRA ticket: CVS-186154.

AI assistance:

Note

AI assistance used to implement fix and unit test. The generated code were verified for correctness, manually built, executed, and fixed as needed.

@github-actions github-actions Bot added the category: GPU OpenVINO GPU plugin label Jul 13, 2026
@kwieloch-intel
kwieloch-intel force-pushed the non_transposed_weights branch from 07f3a86 to f35396b Compare July 14, 2026 17:09
@kwieloch-intel
kwieloch-intel marked this pull request as ready for review July 15, 2026 12:31
@kwieloch-intel
kwieloch-intel requested review from a team as code owners July 15, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: GPU OpenVINO GPU plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant