Skip to content

[ET-VK][sdpa] Vendor-adaptive head_dim output-tiling (TILE_N4=2) in GQA AV coop-GEMV#21118

Merged
SS-JIA merged 1 commit into
gh/SS-JIA/576/origfrom
gh/SS-JIA/577/orig
Jul 22, 2026
Merged

[ET-VK][sdpa] Vendor-adaptive head_dim output-tiling (TILE_N4=2) in GQA AV coop-GEMV#21118
SS-JIA merged 1 commit into
gh/SS-JIA/576/origfrom
gh/SS-JIA/577/orig

Conversation

@pytorchbot

Copy link
Copy Markdown
Collaborator

This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #21064 by @SS-JIA
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/SS-JIA/577/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/SS-JIA/577/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/SS-JIA/576/orig
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/SS-JIA/577/orig
Differential Revision: D112906313
@diff-train-skip-merge

…QA AV coop-GEMV

Pull Request resolved: #21064

Builds on the GQA-reuse AV coop-GEMV (parent commit). The base
`sdpa_compute_out_gqa_coop` shader assigns one head_dim texel (TILE_N4=1) per
workgroup x-slot. This adds a head_dim output-tiled variant,
`sdpa_compute_out_gqa_coop_tile2` (TILE_N4=2): each workgroup owns 2 head_dim
texels and emits G x TILE_N4 outputs, so the per-context-texel attn-weight loads
and the shared-memory tree reduction are amortized over twice as many outputs.
The AV coop GEMV GLSL is already a TILE_N4-parameterized template (the
`partial_n_tile` uniform branch handles the D4 % TILE_N4 != 0 tail), so this
change is purely a new codegen variant plus dispatch wiring — no shader-body
change.

Selection is vendor-adaptive: `pick_sdpa_av_shader` appends the `_tile2` suffix
only on Adreno (`graph->device_is_adreno()`). Tiling is a consistent win on
Adreno (AV ~1.14-1.63x over the base GQA variant) but a regression on Mali at
common decode contexts (~0.67-0.86x, interleaved median-of-N), so Mali and other
vendors keep the base `sdpa_compute_out_gqa_coop`.
`pick_sdpa_av_global_wg_size` keys off the same `_tile2` suffix: the tiled
variant's x-dim collapses from D4 to div_up(D4, 2) since each workgroup now
covers 2 head_dim texels.

Because the variant is Adreno-only in production, the test-only `gqa_override`
knob (see SDPA.h) is extended so tests can pin the variant on any device:
`kGqaOverrideForceTile2` / `kGqaOverrideForceBase` force the tiled / base
variant regardless of vendor (via `resolve_use_tile2`), giving the tiled shader
deterministic coverage on Mali / SwiftShader. `resolve_use_gqa` is unchanged
(any non-`ForceNonGqa` value still forces the GQA family, VK_CHECK'd for
eligibility).
ghstack-source-id: 405400514
@exported-using-ghexport

Differential Revision: [D112906313](https://our.internmc.facebook.com/intern/diff/D112906313/)
@pytorchbot
pytorchbot requested a review from SS-JIA as a code owner July 22, 2026 02:35
@pytorch-bot

pytorch-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21118

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit b2804f7 with merge base 37400d9 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 22, 2026
@SS-JIA
SS-JIA merged commit 26c762b into gh/SS-JIA/576/orig Jul 22, 2026
192 of 198 checks passed
@SS-JIA
SS-JIA deleted the gh/SS-JIA/577/orig branch July 22, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants