[GPU] Enable dynamic quantization for MXFP8 & regular FP8 dtypes by tkrupa-intel · Pull Request #35283 · openvinotoolkit/openvino

tkrupa-intel · 2026-04-13T11:36:53Z

Details:

This PR enables f8e4m3 and f8e5m2 as well as their microscaling (mx) variations.
They are only enabled for dynamic quantization. Static quantization is out of scope of this effort.
Dynamic quantize op itself is enabled as an OpenCL kernel within OpenVINO (expanded on existing kernel which was already there for int8). There also exists oneDNN implementation: Extend set_scales with quantization formula uxlfoundation/oneDNN#3978. At the moment it's not used as adopting it proved challenging inside OpenVINO (see CVS-176111). However, it may be incorporated later if the need of performance improvement arises.
Fully connected op is enabled to accept fp8 & mxfp8 inputs (oneDNN only).
Example of config file:

 root@docker:~/frameworks.ai.openvino.gpu.embargo_support# cat config2.json
{
    "GPU": { "DYNAMIC_QUANTIZATION_GROUP_SIZE": "32" }
}

Some parts of the code have mentions of fp4 and mxfp4. This PR does NOT enable these dtypes, however their enablement is being developed as a follow up to this.
The PR introduces a various util functions used for handling FP8 types in the OpenCL code.
The output data types of dynamic quantize op is inferred from the data types of the prequantized weights. Should a user-defined property prove to be necessary (e.g. for some mixed precision workloads), it'll be added in a separate PR.

Tickets:

CVS-173894

Copilot

Pull request overview

Enables GPU-side dynamic quantization for FP8 (f8e4m3/f8e5m2) and MXFP8 (FP8 weights + f8e8m0 microscaling) flows, extending kernel selector / OpenCL kernels and updating transformation pipelines and tests to recognize and exercise these dtypes.

Changes:

Extend GPU dynamic quantize + reorder infrastructure to support FP8/MXFP8 datatypes (kernel selector enums/keys, JIT type helpers, OpenCL kernels, runtime type mappings).
Update transformation pipelines (GPU plugin + core preprocess/MOC) to mark dequantization for FP8/MXFP patterns, and reuse identical DynamicQuantize nodes when already present.
Add/extend unit + functional test coverage for FP8/MXFP8 FC compression conversion and dynamic quantization scenarios, with platform-based skipping where required.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/plugins/intel_gpu/tests/unit/transformations/convert_fc_to_compressed_test.cpp	Adds FC-to-compressed conversion tests for FP8 weights + MX scale patterns.
src/plugins/intel_gpu/tests/unit/test_cases/dynamic_quantize_gpu_test.cpp	Extends dynamic quantize unit tests to vary scale dtype and cover FP8/MXFP8.
src/plugins/intel_gpu/tests/functional/subgraph_tests/dynamic/matmul_weights_decompression.cpp	Adds MXFP toggle and FP8/MXFP instantiations for matmul decompression functional tests.
src/plugins/intel_gpu/tests/functional/shared_tests_instances/skip_tests_config.cpp	Disables FP8 dyn-quant matmul decompression tests on non-systolic platforms.
src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp	Adds FP8/MXFP types to quant/dequant marking and related pipeline logic.
src/plugins/intel_gpu/src/plugin/transformations/dynamic_quantize_fully_connected.hpp	Adds include needed for updated dynamic quantize FC transformation.
src/plugins/intel_gpu/src/plugin/transformations/dynamic_quantize_fully_connected.cpp	Sets dyn-quant output dtype based on weight dtype (incl. FP8/MXFP) and reuses identical DynamicQuantize nodes.
src/plugins/intel_gpu/src/plugin/transformations/compressed_weights_pattern.hpp	Extends compressed-weights pattern to match FP8 and optional scale convert.
src/plugins/intel_gpu/src/plugin/common_utils.cpp	Marks FP8 types as supported element types in GPU plugin utilities.
src/plugins/intel_gpu/src/kernel_selector/kernels/reorder/reorder_kernel_base.cpp	Adjusts reorder calc type selection for FP8/MXFP.
src/plugins/intel_gpu/src/kernel_selector/kernels/reorder/reorder_kernel.cpp	Enables reorder kernel support + JIT flags for FP8/MXFP IO.
src/plugins/intel_gpu/src/kernel_selector/kernels/dynamic_quantize/dynamic_quantize_kernel_ref.cpp	Enables FP8/MXFP outputs and adds validation for unsupported precomputed-reduction combos.
src/plugins/intel_gpu/src/kernel_selector/kernels/dynamic_quantize/dynamic_quantize_kernel_opt.cpp	Enables FP8/MXFP outputs and adds validation for unsupported precomputed-reduction combos.
src/plugins/intel_gpu/src/kernel_selector/kernel_selector_params.h	Extends kernel selector datatype key bitset with FP8/MXFP.
src/plugins/intel_gpu/src/kernel_selector/kernel_selector_params.cpp	Wires ParamsKey enablement for FP8/MXFP input/output types.
src/plugins/intel_gpu/src/kernel_selector/kernel_selector_common.cpp	Adds stringification for FP8/MXFP datatypes.
src/plugins/intel_gpu/src/kernel_selector/jitter.cpp	Adds FP8/MXFP type jit constants and switches float/half conversions to `_convert_*` helpers.
src/plugins/intel_gpu/src/kernel_selector/common_types.h	Adds FP8/MXFP to Datatype and WeightsType enums.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/reorder_data.cl	Conditionally includes FP8 helpers for reorder kernel.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/include/batch_headers/sub_group_block_write.cl	Extends block write helpers to preserve source type info for FP8-friendly casting.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/include/batch_headers/f8_utils.cl	Introduces FP8/MXFP conversion utilities for OpenCL kernels.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/include/batch_headers/common.cl	Adds extended `as`/convert helpers and vector conversion overloads used by FP8 codepaths.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/dynamic_quantize_gpu_ref.cl	Extends reference dynamic-quantize OpenCL kernel for FP8/MXFP output.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/dynamic_quantize_gpu_opt.cl	Extends optimized dynamic-quantize OpenCL kernel for FP8/MXFP output.
src/plugins/intel_gpu/src/graph/program.cpp	Improves error message when inserting intermediate nodes with existing deps.
src/plugins/intel_gpu/src/graph/primitive_inst.cpp	Allows dynamic_quantize output dtype assertions to include FP8.
src/plugins/intel_gpu/src/graph/impls/onednn/utils.cpp	Maps GPU FP8/MXFP types to oneDNN types and handles offsets.
src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp	Enables FP8 compressed FC support and validates scale dtype constraints.
src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp	Treats FP8 inputs as dyn-quant sources for oneDNN FC arg wiring.
src/plugins/intel_gpu/src/graph/impls/ocl_v2/utils/jitter.cpp	Adds FP8/MXFP type JIT constants and `_convert_*` usage in ocl_v2 path.
src/plugins/intel_gpu/src/graph/impls/ocl/kernel_selector_helper.cpp	Adds conversions between clDNN and kernel_selector FP8/MXFP types.
src/plugins/intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp	Registers OCL dynamic_quantize impl for FP8/MXFP output types.
src/plugins/intel_gpu/src/graph/graph_optimizer/add_required_reorders.cpp	Fixes a comment typo (“arbitrary” → “arbitrarily”).
src/plugins/intel_gpu/src/graph/dynamic_quantize.cpp	Makes dynamic_quantize output dtype follow attrs and disables runtime skipping for floating-point quantization dtypes.
src/plugins/intel_gpu/src/graph/debug_helper.cpp	Adds memory dumping support for FP8/MXFP buffers.
src/core/src/preprocess/pre_post_process.cpp	Updates pre/post-processing pipeline to detect MXFP patterns and adjust dequant marking.
src/common/transformations/src/transformations/common_optimizations/moc_transformations.cpp	Updates MOC low-precision pipeline to detect MXFP patterns and adjust dequant marking.
src/common/transformations/include/ov_ops/dynamic_quantize.hpp	Adds attribute equality helpers to compare/reuse DynamicQuantize configs.
src/common/low_precision_transformations/src/low_precision.cpp	Adds MXFP pattern detection helper used by pipelines.
src/common/low_precision_transformations/include/low_precision/low_precision.hpp	Declares MXFP pattern detection helper.

t-jankowski

ok for core

This reverts commit 2d59ee8.

This reverts commit 3befb0e.

This reverts commit c953fce.

An approval was submitted by mistake

v-Golubev

Please see my comments above

maxnick

LGTM

praasz · 2026-06-24T15:50:09Z


    std::shared_ptr<Node> clone_with_new_inputs(const ov::OutputVector& new_args) const override;

+    bool visit_attributes(AttributeVisitor& visitor) override;


Why is required?

Please see #35283 (comment)

In addition, it is needed for correct model caching work: visit_attributes is used during ov::Model serialization/deserialization

But serialization cannot just be added if attribute is not officially supported
If it was not serialized before it was not official Added in IR and must be reverted

If this change means the attribute is adde in IR it must be reverted.
Or it is used for internal serialization only?

v-Golubev

LGTM

CuriousPanCake

LGTM from the transformations side

…toolkit#36641) Removes duplicated code and reuses ov::test::utils::initMatMulDecompressionSubgraph(). Requested in review of openvinotoolkit#35283. Note on why smoke_MatMulCompressedWeights_dyn_quan_scalar_wzp's flag is changed from true to false: the purpose of this test is checking if wzp with empty shape doesn't cause an error throw in DynamicQuantizeFullyConnected (see openvinotoolkit#32691). After moving to ov::test::utils::initMatMulDecompressionSubgraph() setting this flag to false results in shape being empty (with true it's {1, 1}). After the changes the same combination of params with flag set to true is run in smoke_MatMulCompressedWeights_dyn_quan so I don't add it to smoke_MatMulCompressedWeights_dyn_quan_scalar_wzp.

tkrupa-intel added 2 commits April 13, 2026 11:12

[GPU] Enable dynamic quantization for MXFP8 & regular FP8 dtypes

395ff23

Code formatting

fb6ff10

tkrupa-intel requested a review from p-durandin April 13, 2026 11:36

tkrupa-intel requested review from a team as code owners April 13, 2026 11:36

github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations category: LP transformations OpenVINO Low Precision transformations labels Apr 13, 2026

Remove unnecessary matcher macro

2556a2b

mlukasze requested a review from Copilot April 14, 2026 04:56

Copilot started reviewing on behalf of mlukasze April 14, 2026 04:57 View session

Copilot AI reviewed Apr 14, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/kernel_selector/cl_kernels/dynamic_quantize_gpu_ref.cl Outdated

Comment thread src/common/low_precision_transformations/src/low_precision.cpp Outdated

Comment thread src/plugins/intel_gpu/src/kernel_selector/cl_kernels/dynamic_quantize_gpu_opt.cl

t-jankowski approved these changes Apr 14, 2026

View reviewed changes

Comment thread src/core/src/preprocess/pre_post_process.cpp Outdated

tkrupa-intel added 3 commits April 14, 2026 08:55

Apply review suggestions

15807e8

Merge branch 'master' into tkrupa/mxfp8

a926796

Fix potential memory issue

6e5833b

tkrupa-intel changed the title ~~[GPU] Enable dynamic quantization for MXFP8 & regular FP8 dtypes~~ [GPU][DO NOT MERGE] Enable dynamic quantization for MXFP8 & regular FP8 dtypes Apr 21, 2026

tkrupa-intel added 4 commits April 21, 2026 08:22

CI out of resources issue test

2d59ee8

Revert "CI out of resources issue test"

60db5d3

This reverts commit 2d59ee8.

CI out of resources issue test - nested node search

3befb0e

Revert "CI out of resources issue test - nested node search"

355b0fa

This reverts commit 3befb0e.

tkrupa-intel force-pushed the tkrupa/mxfp8 branch from 15fe77d to c953fce Compare April 24, 2026 12:42

tkrupa-intel added 4 commits April 24, 2026 12:47

Bring back block write macro

c953fce

Revert "Bring back block write macro"

e22b817

This reverts commit c953fce.

Inline expensive wrapper functions

a4a70e2

Get rid of expensive convert wrappers for non-fp dtypes

71777fd

maxnick requested changes Jun 22, 2026

View reviewed changes

v-Golubev reviewed Jun 22, 2026

View reviewed changes

Implement review suggestions

712e6e0

praasz reviewed Jun 24, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/kernel_selector/primitive_db_gen.py Outdated

Move detection of guard patterns to a common function

6af3fa7

v-Golubev previously approved these changes Jun 24, 2026

View reviewed changes

v-Golubev requested changes Jun 24, 2026

View reviewed changes

maxnick approved these changes Jun 24, 2026

View reviewed changes

github-actions Bot added the category: CPP API OpenVINO CPP API bindings label Jun 24, 2026

tkrupa-intel added 2 commits June 24, 2026 15:15

Apply review suggestions

22758b1

Code style

abf2d23

praasz reviewed Jun 24, 2026

View reviewed changes

CuriousPanCake reviewed Jun 24, 2026

View reviewed changes

Comment thread src/common/transformations/src/transformations/common_optimizations/moc_transformations.cpp Outdated

tkrupa-intel added 2 commits June 25, 2026 06:09

Apply review suggestions

8b7ac91

Remove unnecessary check

c475d81

v-Golubev approved these changes Jun 25, 2026

View reviewed changes

CuriousPanCake approved these changes Jun 25, 2026

View reviewed changes

Fix zp handling

850ea41

p-durandin approved these changes Jun 25, 2026

View reviewed changes

p-durandin enabled auto-merge June 25, 2026 12:13

p-durandin added this pull request to the merge queue Jun 25, 2026

Merged via the queue into openvinotoolkit:master with commit c508c75 Jun 25, 2026
211 checks passed

tkrupa-intel mentioned this pull request Jun 30, 2026

[GPU][Tests] Refactor matmul_weights_decompression.cpp test #36641

Merged

tkrupa-intel deleted the tkrupa/mxfp8 branch July 2, 2026 06:43

tkrupa-intel mentioned this pull request Jul 14, 2026

Don't serialize DQ attrs #36875

Closed

DingZhangIntel mentioned this pull request Jul 15, 2026

[TRANSFORMATIONS] Serialize DynamicQuantize attributes via visit_attributes #36903

Closed

merezman mentioned this pull request Jul 16, 2026

[GPU] Enable dynamic quantization for MXFP4 & regular FP4 dtypes merezman/openvino#1

Draft


		std::shared_ptr<Node> clone_with_new_inputs(const ov::OutputVector& new_args) const override;

		bool visit_attributes(AttributeVisitor& visitor) override;

Uh oh!

Conversation

tkrupa-intel commented Apr 13, 2026

Details:

Tickets:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

t-jankowski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v-Golubev left a comment

Choose a reason for hiding this comment

Uh oh!

maxnick left a comment

Choose a reason for hiding this comment

Uh oh!

praasz Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

tkrupa-intel Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

v-Golubev Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

praasz Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

v-Golubev left a comment

Choose a reason for hiding this comment

Uh oh!

CuriousPanCake left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

praasz Jun 26, 2026 •

edited

Loading