Skip to content

[GPU] Enable DynamicQuantize sharing to reduce redundant quantization#36866

Open
hyunback wants to merge 1 commit into
openvinotoolkit:masterfrom
hyunback:gemma4_dq_sharing
Open

[GPU] Enable DynamicQuantize sharing to reduce redundant quantization#36866
hyunback wants to merge 1 commit into
openvinotoolkit:masterfrom
hyunback:gemma4_dq_sharing

Conversation

@hyunback

@hyunback hyunback commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

When DynamicQuantize (DQ) nodes are inserted before FullyConnected operations during GPU plugin graph optimization, identical DQ nodes with the same input are created independently for each FC consumer. This results in redundant quantization computation and increased graph complexity.

Solution

Register SharedOpOptimization after DynamicQuantize insertion to deduplicate identical DQ nodes sharing the same input. Relax single-user assertion in dynamic_quantize graph node to allow shared consumers.

image

Gemma-4-26B-A4B-it

  • 156 DQ nodes -> 121 DQ nodes (35 eliminated, ~22% reduction)
  • gate+up FC sharing: 30 pairs across all 30 layers (gate_proj and up_proj share the same FFN input after RMS normalization)
  • Q+K FC sharing: 5 pairs in sliding window layers (L5, L11, L17, L23, L29 have separate Q/K FCs sharing the same attention input)

SD3.5 large

Model DQ Sharing OFF DQ Sharing ON Reduced
0 (text encoder) 98 98 0
1 (text encoder) 258 258 0
2 240 192 48
3 (transformer) 768 616 152
4 (decoder) 8 4 4
Total 1372 1168 204

Tickets:

AI Assistance:

  • *AI assistance used: yes
  • If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

@hyunback
hyunback requested review from a team as code owners July 14, 2026 07:14
@github-actions github-actions Bot added the category: GPU OpenVINO GPU plugin label Jul 14, 2026
…ation

Register SharedOpOptimization after DynamicQuantize insertion to deduplicate
identical DQ nodes sharing the same input. Relax single-user assertion in
dynamic_quantize graph node to allow shared consumers.
Gemma4: 156 -> 121 DQ nodes (35 saved: 30 gate/up + 5 Q/K on sliding layers).

Signed-off-by: hyunback <hyunback.kim@intel.com>
@hyunback
hyunback force-pushed the gemma4_dq_sharing branch from 4cc31fe to eff2da2 Compare July 16, 2026 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant