[GPU] Fix per_layer_inputs shape in SDPAToPagedAttention for Gemma-4-e2b#36908
Open
hyunback wants to merge 2 commits into
Open
[GPU] Fix per_layer_inputs shape in SDPAToPagedAttention for Gemma-4-e2b#36908hyunback wants to merge 2 commits into
hyunback wants to merge 2 commits into
Conversation
Fix dynamic shape broadcast in PLE (Per Layer Embedding) path after PA transformation. When inputs_embeds is merged to 2D in PA mode, per_layer_inputs dim1 must be set to 1 to match the Unsqueeze-inserted seq dimension, preventing dynamic propagation that causes GPU compile crash and unnecessary dynamic shape inference on CPU. Validated on gemma-4-e2b-it: all 35 PA query shapes become static, GPU/CPU compile succeeds. Signed-off-by: hyunback <hyunback.kim@intel.com>
Replaced with PLE fix validation (static query compiles, dynamic query throws). Signed-off-by: hyunback <hyunback.kim@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Gemma-4-e2b-it crashes during GPU compile_model with PagedAttention enabled.
The SDPAToPagedAttention pass merges batch and seq dims of
inputs_embedsinto 2D. It inserts an Unsqueeze to produce[total_tokens, 1, hidden]for downstream consumers. However, theper_layer_inputsparameter (PLE) retains its original shape[?, ?, ?, 256].When added to the PLE projection output
[?, 1, 35, 256], broadcasting on dim1 (1 vs ?) makes all downstream shapes dynamic. This propagates through every layer's residual Add. All PA query shapes become[?, ?]instead of[?, 2048].On GPU, this causes a crash —
heads_nummust be computed at compile time from a static query dimension. On CPU, it silently degrades into unnecessary dynamic shape inference across 34 layers.Solution
Set
per_layer_inputsdim1=1 in SDPAToPagedAttention. This matches the PA mode convention where batch×seq are merged and Unsqueeze inserts seq=1. It is consistent with what the GenAI CB runtime already provides:[total_num_tokens, 1, num_hidden_layers, hidden_size].All 35 PA query shapes become static. GPU/CPU compile succeeds. The broadcast-induced dynamic propagation is eliminated.
Tickets:
AI Assistance: