[GPU][CM] optimize CM VLSDPA and CM PA performance. by ceciliapeng2011 · Pull Request #36748 · openvinotoolkit/openvino

ceciliapeng2011 · 2026-07-07T05:24:44Z

Details

[CMPA/VLSDPA] Reduce softmax critical-path latency - update tree
- Introduce online_softmax_update_tree to shorten reduction dependency chains through tree-based reduction.
- Introduce the CMFLA_USE_TREE_SOFTMAX JIT option and enable it conditionally for KV-compressed mode in CM PA and document the FP16 vs. INT8 trade-offs.
[CMPA/VLSDPA] Reduce softmax critical-path latency - fold log2(e)
- Introduce CMFLA_Q_SCALED_BY_LOG2 JIT option and enable it by default.
- Fold log2(e) into the Q scaling factor (q_scale_factor), enabling a more efficient exp2-based computation path with fewer element-wise operations.
[CMPA/VLSDPA] Improve transpose and data-movement efficiency
- Add transpose_St_to_P_half, a dedicated score-to-probability transpose path that reduces data movement and shuffle overhead before the P × V stage.
[CMPA] Enhance K/V cache utilization through prefetch scheduling
- Rework K/V prefetching, including warm-up prefetches, to better overlap K-phase and V-phase memory accesses and reduce cache miss penalties.
[VLSDPA] Remove VL SDPA host synchronization for cu_seqlens
- Replace explicit host copy and synchronization logic with mem_lock-based access during dispatch preparation.
- Remove legacy runtime-parameter plumbing and synchronization helpers.
- Mark cu_seq_lens as a shape-inference dependency, ensuring dispatch logic can safely consume the data through memory dependencies.
[VLSDPA] Optimize CM VLSDPA kernel with KV blocking and block-level softmax amortization
- Introduce configurable CMFLA_KV_BLK to process multiple KV sub-tiles within a single online softmax update.
- Reorganize the main compute loop to:
  - Compute batched K × Qᵀ score tiles.
  - Apply masking per KV sub-tile.
  - Perform a single softmax update for the entire block.
  - Accumulate P × V results.
- Add conditional rescaling to avoid unnecessary first-iteration rescale operations.
- Select the KV blocking factor heuristically based on head size and padding characteristics to balance softmax amortization and register pressure.

Tickets:

AI Assistance:

AI assistance used: yes
AI was used for part of optimization opportunity probe and code generation; and human code review and manual checks were performed.

…lf-width transpose — 9.25 ms → 7.654 ms (−17%) Apply optimizations from aboutSHW commit 0621405 (River Li): - Item 13: fold log2e into Q pre-scale (qscale = scale_factor * log2e); removes 16 mul/tile from softmax critical path, St lands in log2 domain - Item 14: KV blocking (KV_BLK=2, default); rO rescale amortized over 2 tiles — valid amortization unlike Q-row doubling because exp count is unchanged - Item 15: online_softmax_update_tree; balanced binary tree max/sum reduction, depth log2(BLK_ROWS)=5 vs linear chain depth 31 for BLK_ROWS=32 - Item 16: transpose_St_to_P_half; cast float->half before GRF shuffle so the 4-pass select network runs at half data-path width Also includes prerequisite items already in aboutSHW but absent here: - V prefetch moved to K phase (item 2) — gives V full K-DPAS lead time - Pre-loop warm-up prefetch (item 4) — cold-start for kv_pos=0 - Unified PV path (item 5) — eliminates kv_pos==0 branch via max_comp=0 trick Combined result on 2-seq x 3432 (PTL 4xe, 16h, d=64): 9.25 ms -> 7.654 ms (-17%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit 334e3d9)

(cherry picked from commit a8a10c4)

cm_attention_common.hpp: - Add constexpr log2e and q_scale_factor (fold log2e into Q pre-scale, removing per-element mul from softmax critical path) - Add online_softmax_update_tree (tree-reduction variant, depth log2 vs linear) - Add transpose_St_to_P_half (float->half before GRF shuffle, halves mov width) cm_pa_xe2.hpp: - Add CMPA_USE_TREE_SOFTMAX macro + pa_softmax_update dispatch macro - Switch Q scale from scale_factor to q_scale_factor (both U8 and FP16 paths) - Replace online_softmax_update + Transpose2DMatrix call sites with pa_softmax_update + transpose_St_to_P_half at all 4 locations cm_sdpa_common.hpp: - Remove now-duplicate online_softmax_update_tree and transpose_St_to_P_half (inherited from cm_attention_common.hpp via existing include) paged_attention_gen.cpp: - Emit CMPA_USE_TREE_SOFTMAX=1 for INT8, =0 for FP16: tree softmax wins 1-5% on INT8 (dequant amortises scratch), regresses ~9% on FP16 (register pressure from scratch matrix) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit 92dc157)

(cherry picked from commit cbe4e60)

…branch) (cherry picked from commit f11f6c4)

Copilot

Pull request overview

This PR updates the Intel GPU plugin’s CM attention kernels (VL SDPA and paged attention) to improve throughput by restructuring SDPA’s KV processing (blocking + amortized softmax), introducing/using a tree-reduction softmax path, improving transpose/data-movement, and removing an explicit host-sync path for cu_seqlens by switching dispatch-time reads to mem_lock with proper dependency wiring.

Changes:

Remove VL SDPA runtime host copy/synchronization for cu_seqlens; wire it as a shape-infer dependency and read it via mem_lock in dispatch.
Add KV-blocking and tree softmax usage in VL SDPA CM code paths; add a float→half optimized transpose helper for score→P tiles.
Add JIT gating for tree softmax in paged attention and update PA kernels to use the new softmax/transpose helpers.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/plugins/intel_gpu/src/graph/vl_sdpa.cpp	Removes the old host-sync `cu_seqlens` copy helper.
src/plugins/intel_gpu/src/graph/include/vl_sdpa_inst.h	Adds `cu_seqlens` as a shape-infer dependency; removes old runtime-param plumbing.
src/plugins/intel_gpu/src/graph/impls/cm/vl_sdpa_opt.cpp	Reads `cu_seqlens` via `mem_lock` in dispatch; adds default KV blocking heuristic; removes runtime params path.
src/plugins/intel_gpu/src/graph/impls/cm/paged_attention_gen.cpp	Adds JIT constant to enable tree softmax only for KV-compressed mode.
src/plugins/intel_gpu/src/graph/impls/cm/include/cm_sdpa_common.hpp	Introduces KV blocking + tree softmax usage and accumulator initialization; uses optimized transpose helper.
src/plugins/intel_gpu/src/graph/impls/cm/include/cm_pa_xe2.hpp	Adds tree-softmax gating macro and switches PA kernels to new softmax/transpose helpers and q pre-scale factor.
src/plugins/intel_gpu/src/graph/impls/cm/include/cm_attention_common.hpp	Adds tree-reduction softmax helper and score-tile float→half transpose helper; introduces `q_scale_factor`.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Bug --- Running on GPU hangs and dies with: clWaitForEvents, error code: -14 CL_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST clFinish, error code: -5 CL_OUT_OF_RESOURCES All 9 test cases with are affected; passes. here means the CM kernel performs out-of-bounds SVM addressing. Root cause ---------- The kernel argument dump for the first failing case (num_head=1, head_size=16, cu_seqlens={0,16}, Q/K/V buffers = 512 B each) shows: scalar 9 (q_token_pitch) = 256 <-- expected 16 scalar 10 (k_token_pitch) = 256 <-- expected 16 scalar 11 (v_token_pitch) = 256 <-- expected 16 computes in half-elements, so with (instead of 16) it walks 16x past the end of the 256-element Q buffer as soon as . The wrong value comes from , which unconditionally reads the per-token stride from (the pitch of layout dim 0, ). The kernel contract expects the memory to be with the token axis on dim 0, which is only true when the surrounding ops are fused into (giving non-empty ). In this pipeline the fusion did not run: 1. (core, early) creates with **empty** orders and leaves the four surrounding ops in place. 2. -> rewrites those Transposes to because the permuted dim has size 1. 3. (plugin, late) requires the exact pattern , no longer matches, and does not fuse -> stays empty. VLSDPA's input layout is then the post-Reshape view = . The token axis is , so the correct per-token stride is , but the impl reads . Silent, layout-dependent OOB -> GPU hang. Fix --- 1) Fuse the surrounding at VLSDPA-creation time in , so the fusion cannot be undone by any later pass (in particular ). When all four adjacent transposes are single-consumer , peel them and build VLSDPA(q, k, v, cu_seqlens, {1,0,2}, {1,0,2}, {1,0,2}, {1,0,2}) then . When any of the four is missing or has a different order, fall back to the previous behavior (empty orders, no fusion). 2) Defense in depth: add in the CM VLSDPA impl, invoked from (main thread, before async kernel compilation), , and the dispatch data-func lambda. Any future path that reintroduces an unfused now fails fast at with a message naming the token-stride contract, instead of a silent GPU hang at inference time. Testing ------- - : 18/18 pass (previously 9x cases aborted with ). - / updated to assert the fused reference model (VLSDPA carries orders; no surrounding transposes). - Verified the assert path by temporarily disabling (1): the failing case surfaces as an at model-compile time with the intended message, no GPU hang. Files ----- - src/core/src/pass/sdpa_to_vlsdpa.cpp - src/common/transformations/tests/common_optimizations/ sdpa_to_vlsdpa_test.cpp - src/plugins/intel_gpu/src/graph/impls/cm/vl_sdpa_opt.cpp

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

e-ddykim · 2026-07-15T06:06:16Z

+// online_softmax_update_tree and transpose_St_to_P_half are defined in cm_attention_common.hpp

+#ifdef CM_HAS_LSC_UNTYPED_2D
 template<bool use_causal_mask, int num_heads, int num_kv_heads, int head_size, int is_qkv_fused = 0>


Please check if is_qkv_fused is still necessary.

e-ddykim · 2026-07-15T06:41:27Z

    jit.make("CMPA_WG_SEQ_LEN", get_wg_seq_len(params));
+    // Tree softmax: wins on INT8 (dequant amortises scratch, shorter dep-chain helps),
+    // regresses ~9% on FP16 (extra scratch matrix increases register pressure).
+    jit.make("CMFLA_USE_TREE_SOFTMAX", get_kv_compressed(params) ? 1 : 0);


How about adding a test that compares the outputs of the new tree softmax and the current linear softmax? Actually, we already have existing tests, but for INT8 xAttention the tolerance is allowed up to 4%, which could potentially hide small differences caused by changes in the reduction order.

openvino/src/plugins/intel_gpu/tests/unit/test_cases/paged_attention_gpu_test.h

Line 2278 in d8047fb

EXPECT_LE(mismatch_count, int(data_output_mem->count() * 0.04));

e-ddykim · 2026-07-15T08:06:42Z

no perf. impact from llm daily tests

ceciliapeng2011 and others added 5 commits July 7, 2026 13:19

some fix. opt3

831cc6c

(cherry picked from commit a8a10c4)

gpu(vl_sdpa): remove host sync and read cu_seqlens via mem_lock

aefc561

(cherry picked from commit cbe4e60)

gate rescale rO (skip first iter only when KV blocking amortizes the …

6f2b4f2

…branch) (cherry picked from commit f11f6c4)

ceciliapeng2011 requested review from a team as code owners July 7, 2026 05:24

github-actions Bot added category: GPU OpenVINO GPU plugin category: CPU OpenVINO CPU plugin category: build OpenVINO cmake script / infra and removed category: CPU OpenVINO CPU plugin category: build OpenVINO cmake script / infra labels Jul 7, 2026

ceciliapeng2011 added 3 commits July 7, 2026 13:32

roll back unexpected files.

4b7d163

Merge branch 'master' into cecilia/opt/cm_pa_vlsdpa

32c40bc

clean up

21ab814

ceciliapeng2011 requested review from Copilot and riverlijunjie July 13, 2026 06:17

Copilot started reviewing on behalf of ceciliapeng2011 July 13, 2026 06:18 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/include/cm_pa_xe2.hpp

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/include/cm_pa_xe2.hpp

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/include/cm_sdpa_common.hpp

PA: CMFLA_USE_TREE_SOFTMAX and CMFLA_Q_SCALED_BY_LOG2 decouple

ae0ec39

ceciliapeng2011 force-pushed the cecilia/opt/cm_pa_vlsdpa branch from 04fee76 to aecbb8f Compare July 14, 2026 05:36

Change vlsdpa kernel signature

aecbb8f

ceciliapeng2011 requested a review from Copilot July 14, 2026 05:37

Copilot started reviewing on behalf of ceciliapeng2011 July 14, 2026 05:38 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/vl_sdpa_opt.cpp

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/vl_sdpa_opt.cpp Outdated

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/paged_attention_gen.cpp Outdated

ceciliapeng2011 mentioned this pull request Jul 14, 2026

Feature branch for QWen3-VL performance assessment. #36863

Draft

4 tasks

Potential fix for pull request finding

71db11d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ceciliapeng2011 assigned riverlijunjie Jul 14, 2026

ceciliapeng2011 requested a review from Copilot July 14, 2026 06:45

Copilot started reviewing on behalf of ceciliapeng2011 July 14, 2026 06:45 View session

safe guard to cu_seqlens and q/k/v offsets/pitches.

941752f

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/vl_sdpa_opt.cpp Outdated

fix

6cf27e9

ceciliapeng2011 requested a review from a team as a code owner July 14, 2026 08:33

ceciliapeng2011 requested a review from Copilot July 14, 2026 08:41

clang format

48df96b

github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: transformations OpenVINO Runtime library - Transformations labels Jul 14, 2026

Copilot started reviewing on behalf of ceciliapeng2011 July 14, 2026 08:54 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/graph/impls/cm/vl_sdpa_opt.cpp Outdated

Potential fix for pull request finding

6605746

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ceciliapeng2011 requested a review from e-ddykim July 15, 2026 03:29

ceciliapeng2011 assigned e-ddykim Jul 15, 2026

license header

66149a6

ceciliapeng2011 unassigned riverlijunjie Jul 15, 2026

e-ddykim reviewed Jul 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GPU][CM] optimize CM VLSDPA and CM PA performance.#36748

[GPU][CM] optimize CM VLSDPA and CM PA performance.#36748
ceciliapeng2011 wants to merge 17 commits into
openvinotoolkit:masterfrom
ceciliapeng2011:cecilia/opt/cm_pa_vlsdpa

ceciliapeng2011 commented Jul 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

e-ddykim Jul 15, 2026

Uh oh!

e-ddykim Jul 15, 2026

Uh oh!

e-ddykim commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ceciliapeng2011 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

Tickets:

AI Assistance:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

e-ddykim Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

e-ddykim Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

e-ddykim commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ceciliapeng2011 commented Jul 7, 2026 •

edited

Loading