[GPU] Enable f8e4m3fn quantized-KV GroupQueryAttention (dynamic path) by ankitm3k · Pull Request #36806 · openvinotoolkit/openvino

ankitm3k · 2026-07-09T20:35:58Z

Details:

Enables the com.microsoft GroupQueryAttention op with an f8e4m3fn quantized KV cache on the
Intel GPU plugin, for the dynamic (Slice+Concat) cache-assembly path. This extends the i8/i4 GPU
support (#36751, #36798) to fp8. f8e4m3fn KV is part of the com.microsoft GQA spec
(T_CACHE = {float, float16, bfloat16, uint8, int8, float8e4m3fn}, scales T_KV_SCALE = tensor(float),
symmetric, ±448 clamp) and already worked on CPU/TEMPLATE via the device-agnostic
GroupQueryAttentionDecomposition; only GPU primitive gaps remained.

Spec reference: https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftgroupqueryattention

fp8e4m3_t is a 1-byte struct with no arithmetic/convert overloads, so beyond widening the
supported-type lists the reference kernels needed fp8-aware handling (all GPU-plugin-local; core op
and decomposition untouched):

Slice / Concat / ScatterUpdate ref kernels: include f8_utils.cl for the fp8e4m3_t typedef,
and on the identity (no-activation) store copy the byte directly when the tensor is fp8 —
ACTIVATION/TO_OUTPUT_TYPE don't compile on the struct. Gated by an INPUT*_IS_F8E4M3 jit flag
(the f8_utils.cl include is guarded on it, so it is not pulled into — and does not bloat —
non-fp8 kernels); non-fp8 paths are textually unchanged.
reorder_data.cl: (a) an f8→f8 layout reorder stores the already-encoded value directly;
(b) a non-float input reordered to fp8 encodes from float to resolve the ambiguous fp8-encoder
overload. Both gated on fp8 flags.
Add f8e4m3 to the Slice/Concat/ScatterUpdate impl and kernel supported-type lists.

Tests: un-skip the 2 IE_GPU.onnx_model_gqa_f8e4m3fnkv_{per_channel,per_tensor} conformance tests
(present KV checked bit-exact, float output by ULP tolerance). GQA quantized-KV suite is 36/36
across IE_GPU / IE_CPU / INTERPRETER (i8/i4/f8); a dedicated scatter_update_gpu_f8e4m3 unit test
covers the fp8 scatter byte-copy path, and 356 GPU reorder unit tests remain green (the
shared-kernel change is a regression-safe, fp8-gated addition).

Out of scope / follow-up: the static (ScatterUpdate) f8 path still fails implementation selection
in add_required_reorders for the fp8 dtype (i8/i4 static pass through the same path unchanged); it
is tracked as a separate follow-up and no static f8 GQA test is added here.

Builds on #36798 (int8/int4 GPU GQA), now merged to master; this PR is rebased on top.

Tickets:

CVS-190527

AI Assistance:

AI assistance used: yes
Aided GPU debug, unit testing & understand OCL pass.

ankitm3k · 2026-07-09T20:50:39Z

build_jenkins

p-durandin · 2026-07-10T14:09:55Z

@ankitm3k please fix code style

ankitm3k · 2026-07-10T15:58:46Z

@ankitm3k please fix code style

fixed

ankitm3k · 2026-07-14T13:57:42Z

build_jenkins

Extends the GPU quantized-KV GroupQueryAttention support (i8/i4) to the f8e4m3fn KV cache on the dynamic (Slice+Concat) cache-assembly path. f8e4m3fn KV is part of the com.microsoft GQA spec (T_CACHE) and already worked on CPU/TEMPLATE via the device-agnostic decomposition; only GPU primitive gaps remained. fp8e4m3_t is a 1-byte struct with no arithmetic/convert overloads, so the reference kernels needed f8-aware handling on top of the type-list widenings: - Slice / Concat / ScatterUpdate ref kernels: include f8_utils.cl for the fp8e4m3_t typedef, and on the identity (no-activation) store copy the byte directly when the tensor is fp8 - ACTIVATION/TO_OUTPUT_TYPE do not compile on the struct. Gated by an INPUT*_IS_FP8 jit flag; non-fp8 paths are unchanged. - reorder_data.cl: (a) f8->f8 layout reorder stores the already-encoded value directly; (b) for a non-float input reordered to fp8, encode from float to resolve the ambiguous fp8 encoder overload. Both gated on fp8 flags. - Add f8e4m3 to the Slice/Concat/ScatterUpdate impl and kernel supported-type lists. Tests: un-skip the 2 IE_GPU gqa_f8e4m3fnkv tests (dynamic). GQA quant suite is 36/36 across GPU/CPU/INTERPRETER (i8/i4/f8); 356 GPU reorder unit tests green (no regression from the shared-kernel change). Out of scope: the static (ScatterUpdate) f8 path still fails impl selection in add_required_reorders for the fp8 dtype (i8/i4 static pass); tracked as a follow-up. No static f8 GQA test is added here.

Mirror of the i8/i4 review feedback for the f8e4m3 dynamic path: - slice_gpu_f8e4m3.bfyx - f8 Slice output (dynamic Slice+Concat path). - concat_gpu.dynamic_f8e4m3 - f8 Concat in/out on a dynamic shape (as GQA uses it). Both cover the f8 primitive paths enabled for a quantized KV cache. ScatterUpdate f8 is not tested here as the static (ScatterUpdate) f8 path is out of scope / a documented follow-up.

- Use a named tolerance_bits constant instead of a bare magic number in run(). - Annotate the f8e4m3fn test tensors with their shapes.

…o, add f8 scatter test - Guard f8_utils.cl includes behind the f8e4m3 jit flag in the concatenation/ scatter_update/slice ref kernels so it is not pulled in (and does not bloat the kernel binary) when the input is not f8e4m3. - Rename INPUT0_IS_FP8/INPUT2_IS_FP8 jit macros to *_IS_F8E4M3 since only f8e4m3 is handled. - Add an f8e4m3 scatter_update unit test.

github-actions · 2026-07-16T11:19:14Z

CI Doctor — Merge Queue failure on this PR

Pipeline: Windows (VS 2022, Python 3.11, Release)
Failure: JS API E2E Electron cleanup fails on Windows - Permission denied removing locked electron.exe/dll
Automatic restart: ✅ Re-run of failed jobs requested (reason: Transient Windows file-locking flake in JS API E2E cleanup - all actual tests passed, only the 'after all' hook failed trying to rm Electron DLL files)

Possible remedy

This failure is unrelated to your GPU fp8 changes — the JS API E2E test cleanup is a known Windows flake (6th occurrence).
The automatic re-run has been triggered; it is likely to succeed.
If the re-run fails again, the permanent fix is to replace rm -rf demo-electron-app-project in src/bindings/js/node/tests/e2e/electron-app.test.js afterAll hook with fs.rmSync(path, { recursive: true, force: true }) or the rimraf package.
Alternatively, add taskkill /F /IM electron.exe before the directory removal to release locked handles.

What happened

The JS API E2E test job (JS API (26)) on Windows failed in its "after all" cleanup hook: rm -rf demo-electron-app-project was denied permission to remove electron.exe, d3dcompiler_47.dll, libEGL.dll, and libGLESv2.dll because Windows kept file handles open after the Electron process completed. Both actual E2E tests (should install dependencies, should run electron package and verify output) passed successfully.

This is a recurring known pattern — same failure seen 5 previous times on PRs #36780, #36763, #36796, #36887, #36888.

Generated by CI Failure Doctor — Merge Queue · 133.3 AIC · ⌖ 7.74 AIC · ⊞ 18.6K · ◷

Follow-up to the openvinotoolkit#36806 review: adds a uint8 Slice case (values above the int8 range) next to the existing slice_gpu_i8 test, covering the u8 output path used by the quantized-KV GroupQueryAttention Slice+Concat assembly.

ankitm3k requested a review from Kotomi-Du July 9, 2026 20:35

ankitm3k requested review from a team as code owners July 9, 2026 20:35

github-actions Bot added category: GPU OpenVINO GPU plugin category: ONNX FE OpenVINO ONNX FrontEnd labels Jul 9, 2026

ankitm3k force-pushed the gqa_gpu_f8_support branch from b6784b8 to f877b1f Compare July 9, 2026 20:50

ankitm3k force-pushed the gqa_gpu_f8_support branch from f877b1f to a086ade Compare July 10, 2026 11:16

ankitm3k force-pushed the gqa_gpu_f8_support branch from a086ade to 9ab99c3 Compare July 10, 2026 15:58

ankitm3k force-pushed the gqa_gpu_f8_support branch 2 times, most recently from d042bfd to 251d038 Compare July 14, 2026 13:55

ankitm3k force-pushed the gqa_gpu_f8_support branch from 251d038 to 7015b83 Compare July 15, 2026 08:53

tkrupa-intel requested changes Jul 15, 2026

View reviewed changes

ankitm3k added 3 commits July 15, 2026 21:12

[GPU] GQA f8: address review - named tolerance, test shape info

7e2d888

- Use a named tolerance_bits constant instead of a bare magic number in run(). - Annotate the f8e4m3fn test tensors with their shapes.

ankitm3k force-pushed the gqa_gpu_f8_support branch from 7015b83 to 7e2d888 Compare July 15, 2026 15:46

ankitm3k requested review from e-ddykim and tkrupa-intel July 15, 2026 17:42

tkrupa-intel approved these changes Jul 16, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/tests/unit/test_cases/slice_gpu_test.cpp

ankitm3k added this pull request to the merge queue Jul 16, 2026

e-ddykim removed this pull request from the merge queue due to a manual request Jul 16, 2026

p-durandin added the do not merge label Jul 16, 2026

p-durandin assigned e-ddykim Jul 16, 2026

p-durandin added this to the 2026.4 milestone Jul 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GPU] Enable f8e4m3fn quantized-KV GroupQueryAttention (dynamic path)#36806

[GPU] Enable f8e4m3fn quantized-KV GroupQueryAttention (dynamic path)#36806
ankitm3k wants to merge 4 commits into
openvinotoolkit:masterfrom
ankitm3k:gqa_gpu_f8_support

ankitm3k commented Jul 9, 2026 •

edited

Loading

Uh oh!

ankitm3k commented Jul 9, 2026

Uh oh!

p-durandin commented Jul 10, 2026

Uh oh!

ankitm3k commented Jul 10, 2026 •

edited

Loading

Uh oh!

ankitm3k commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ankitm3k commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

ankitm3k commented Jul 9, 2026

Uh oh!

p-durandin commented Jul 10, 2026

Uh oh!

ankitm3k commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ankitm3k commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

CI Doctor — Merge Queue failure on this PR

Possible remedy

What happened

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ankitm3k commented Jul 9, 2026 •

edited

Loading

ankitm3k commented Jul 10, 2026 •

edited

Loading