Skip to content

[GPU] Fix per_layer_inputs shape in SDPAToPagedAttention for Gemma-4-e2b#36908

Open
hyunback wants to merge 2 commits into
openvinotoolkit:masterfrom
hyunback:fix_gemma4_e2b_crash
Open

[GPU] Fix per_layer_inputs shape in SDPAToPagedAttention for Gemma-4-e2b#36908
hyunback wants to merge 2 commits into
openvinotoolkit:masterfrom
hyunback:fix_gemma4_e2b_crash

Conversation

@hyunback

@hyunback hyunback commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Problem

Gemma-4-e2b-it crashes during GPU compile_model with PagedAttention enabled.

The SDPAToPagedAttention pass merges batch and seq dims of inputs_embeds into 2D. It inserts an Unsqueeze to produce [total_tokens, 1, hidden] for downstream consumers. However, the per_layer_inputs parameter (PLE) retains its original shape [?, ?, ?, 256].

When added to the PLE projection output [?, 1, 35, 256], broadcasting on dim1 (1 vs ?) makes all downstream shapes dynamic. This propagates through every layer's residual Add. All PA query shapes become [?, ?] instead of [?, 2048].

On GPU, this causes a crash — heads_num must be computed at compile time from a static query dimension. On CPU, it silently degrades into unnecessary dynamic shape inference across 34 layers.

Solution

Set per_layer_inputs dim1=1 in SDPAToPagedAttention. This matches the PA mode convention where batch×seq are merged and Unsqueeze inserts seq=1. It is consistent with what the GenAI CB runtime already provides: [total_num_tokens, 1, num_hidden_layers, hidden_size].

All 35 PA query shapes become static. GPU/CPU compile succeeds. The broadcast-induced dynamic propagation is eliminated.

Tickets:

AI Assistance:

  • AI assistance used: yes
  • If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

Fix dynamic shape broadcast in PLE (Per Layer Embedding) path after PA transformation.
When inputs_embeds is merged to 2D in PA mode, per_layer_inputs dim1 must be set to 1 to match the Unsqueeze-inserted seq dimension,
preventing dynamic propagation that causes GPU compile crash and unnecessary dynamic shape inference on CPU.
Validated on gemma-4-e2b-it: all 35 PA query shapes become static, GPU/CPU compile succeeds.

Signed-off-by: hyunback <hyunback.kim@intel.com>
@hyunback hyunback requested review from a team as code owners July 16, 2026 00:32
@github-actions github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin labels Jul 16, 2026
@hyunback hyunback added WIP work in progress and removed category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin WIP work in progress labels Jul 16, 2026
Replaced with PLE fix validation (static query compiles, dynamic query throws).

Signed-off-by: hyunback <hyunback.kim@intel.com>
@github-actions github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin labels Jul 16, 2026
@hyunback hyunback added WIP work in progress do not merge labels Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin do not merge WIP work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant