[GPU] Enable dynamic quantization for MXFP4 & regular FP4 dtypes by merezman · Pull Request #36944 · openvinotoolkit/openvino

merezman · 2026-07-17T11:08:20Z

Details:

This is a continuation of the fp8 PR #35283 that introduced f8e4m3 and f8e5m2 support.
This PR extends the fp8 implementation by adding support for f4e2m1 and its microscaling (mx) variation.
The PR builds upon the existing FP8 utility functions and extends them to handle FP4 types in OpenCL code.
Similar to the fp8 implementation, f4e2m1 is only enabled for dynamic quantization
Dynamic quantize op is extended with OpenCL kernel support for fp4 data types (building upon the existing kernel infrastructure for fp8).
Fully connected op is enabled to accept fp4 & mxfp4 inputs (oneDNN only).
All transformations and optimizations introduced in the mxfp8 PR are compatible with the new fp4 data types
Testing approach follows the same methodology as the mxfp8 implementation

Tickets:

CVS-172558

Bump product version 2025.4.0

…4_rebase Resolved conflicts and adapted FP4 implementation to FP8 updates: GPU Kernels: - Updated reorder_data.cl with FP4 packing/unpacking logic (2 elements per byte) - Added FP4 INPUT unpacking using byte-level operations - Added FP4 OUTPUT packing with atomic operations - Merged IS_LOW_BIT_FP to include all FP4 and FP8 variants (F8E8M0 added) - Updated f8_utils.cl include path from batch_headers to root Dynamic Quantize: - Updated dynamic_quantize_gpu_opt.cl and dynamic_quantize_gpu_ref.cl - Merged FP8 updates while preserving FP4 support - Kept DynamicQuantize reuse optimization in dynamic_quantize_fully_connected.cpp - Used float instead of half for scale calculations (more precise) Code Organization: - Removed MOE-related files (deleted in target branch) - Accepted target branch updates for paged_attention implementations - Updated submodules (onednn, onednn_gpu) - Merged workflow and build configuration updates

…RG_HINT_MAX_GROUP_SIZE)

- Restore f4e2m1 data type in oneDNN FullyConnected: * Add f4e2m1 to is_dyn_quan_input checks (3 places) * Add f4e2m1 to fp_compressed_case (input/weights/output check) * Remove bf16 from compressed_case output types (match original) - Restore FP4 unit tests in dynamic_quantize_gpu_test.cpp: * dynamic_quantization_mxf4e2m1 test (MXFP4 with f8e8m0 scale) * dynamic_quantization_f4e2m1 test (standard FP4) - Restore FP4 functional tests in matmul_weights_decompression.cpp: * Add f4e2m1 to smoke_MatMulCompressedWeights_dyn_quan_mxfp8 test * Add f4e2m1 to smoke_MatMulCompressedWeights_dyn_quan_fp8 test These FP4 changes were lost during merge when target branch introduced new FP8 support that replaced/restructured some of the original FP4 tests.

Copilot

Pull request overview

Enables FP4 (f4e2m1) and MXFP4 (f4e2m1 + f8e8m0 scales) dynamic-quantization support in the Intel GPU plugin, extending the existing FP8 dynamic-quantization path across kernel selector plumbing, OpenCL kernels, oneDNN FC acceptance, and test coverage.

Changes:

Add FP4/MXFP4 data type support to dynamic_quantize and reorder kernels (kernel selector + OpenCL implementations).
Extend plugin graph/runtime and oneDNN FC paths to recognize and handle FP4 compressed/dynamic-quantized inputs.
Add/extend unit + functional tests for FP4 and MXFP4 scenarios.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/plugins/intel_gpu/tests/unit/transformations/convert_fc_to_compressed_test.cpp	Adds FC→FCCompressed transformation tests for FP4 and MXFP4 weights/scales.
src/plugins/intel_gpu/tests/unit/test_cases/dynamic_quantize_gpu_test.cpp	Adds unit tests for dynamic quantization outputting FP4 and MXFP4.
src/plugins/intel_gpu/tests/functional/subgraph_tests/dynamic/matmul_weights_decompression.cpp	Extends matmul decompression functional coverage to FP4/MXFP4 and adjusts thresholds.
src/plugins/intel_gpu/src/plugin/transformations/compressed_weights_pattern.hpp	Updates compressed-weights pattern to include FP4 and f8e8m0 types.
src/plugins/intel_gpu/src/plugin/common_utils.cpp	Marks f4e2m1 as supported element type in plugin utility checks.
src/plugins/intel_gpu/src/kernel_selector/kernels/reorder/reorder_kernel.cpp	Enables reorder kernel key support and JIT flags for FP4 input/output.
src/plugins/intel_gpu/src/kernel_selector/kernels/reorder/reorder_kernel_base.cpp	Adjusts reorder calc type selection to handle FP4.
src/plugins/intel_gpu/src/kernel_selector/kernels/dynamic_quantize/dynamic_quantize_kernel_ref.cpp	Enables FP4 output type and updates validation/JIT flags.
src/plugins/intel_gpu/src/kernel_selector/kernels/dynamic_quantize/dynamic_quantize_kernel_opt.cpp	Enables FP4 output type and updates JIT flags for optimized kernel.
src/plugins/intel_gpu/src/kernel_selector/kernel_selector_params.h	Adds FP4 datatype bit to ParamsKey.
src/plugins/intel_gpu/src/kernel_selector/kernel_selector_params.cpp	Implements enabling FP4 in input/output type keys.
src/plugins/intel_gpu/src/kernel_selector/kernel_selector_common.cpp	Adds FP4 to datatype string conversion.
src/plugins/intel_gpu/src/kernel_selector/jitter.cpp	Adds FP4 type JIT constants and weights-type mapping.
src/plugins/intel_gpu/src/kernel_selector/common_types.h	Adds FP4 to Datatype and WeightsType enums.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/reorder_data.cl	Implements FP4 unpack/pack logic in reorder kernel.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/include/f4_utils.cl	New FP4 conversion/packing utilities for OpenCL kernels.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/include/batch_headers/common.cl	Adds FP4 `_as_*` prefix support for common JIT macros.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/dynamic_quantize_gpu_ref.cl	Extends reference dynamic quantize kernel to FP4 output packing.
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/dynamic_quantize_gpu_opt.cl	Extends optimized dynamic quantize kernel to FP4 output packing.
src/plugins/intel_gpu/src/graph/primitive_inst.cpp	Allows FP4 as a supported dynamic_quantize output type during shape updates.
src/plugins/intel_gpu/src/graph/impls/onednn/utils.cpp	Maps FP4 to oneDNN datatype and adjusts offset handling for 4-bit types.
src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp	Extends FC oneDNN impl eligibility to FP4 compressed cases.
src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp	Treats FP4 inputs as dynamic-quantized inputs for FC handling.
src/plugins/intel_gpu/src/graph/impls/ocl/kernel_selector_helper.cpp	Adds FP4 mappings between clDNN and kernel_selector types.
src/plugins/intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp	Adds FP4 to supported output types list for OCL dynamic_quantize impl.
src/plugins/intel_gpu/src/graph/impls/ocl_v2/utils/jitter.cpp	Adds FP4 type JIT constants for OCL v2 jitter path.
src/plugins/intel_gpu/src/graph/debug_helper.cpp	Adds FP4 dumping support in debug helper.
src/common/transformations/src/transformations/fp16_compression/mark_decompression_convert_constant_folding.cpp	Extends compressed-float constant marking to FP4.
src/common/low_precision_transformations/src/layer_transformation.cpp	Renames FP8 support accessor and adds FP4 to the supported low-bit float set.
src/common/low_precision_transformations/include/low_precision/layer_transformation.hpp	Updates low-precision API declarations for low-bit float support (FP8+FP4).

 namespace low_precision {
 namespace precision_set {
    LP_TRANSFORMATIONS_API const std::vector<element::Type>& get_int8_support();
-    LP_TRANSFORMATIONS_API const std::vector<element::Type>& get_fp8_support();
+    LP_TRANSFORMATIONS_API const std::vector<element::Type>& get_low_bit_float_support();
    LP_TRANSFORMATIONS_API const std::vector<element::Type>& get_int8_int16_int32_support();
 } // namespace precision_set


+        auto scale_convert = std::make_shared<ov::op::v0::Convert>(scale_const, ov::element::f16);
+        auto scale = std::make_shared<ov::op::v1::Multiply>(weights_convert, scale_convert);
+           auto no_bias = std::make_shared<ov::intel_gpu::op::Placeholder>();
+        auto fc = std::make_shared<ov::intel_gpu::op::FullyConnected>(input1, scale, no_bias);


+        auto weights_const = ov::op::v0::Constant::create(ov::element::f4e2m1, ov::Shape{ 32, 16 }, { 1 });
+        auto scale_const = ov::op::v0::Constant::create(ov::element::f8e8m0, ov::Shape{ 32, 1 }, { 1 });
+           auto no_bias = std::make_shared<ov::intel_gpu::op::Placeholder>();
+        auto fc_compressed = std::make_shared<ov::intel_gpu::op::FullyConnectedCompressed>(input1, weights_const, no_bias, scale_const);


        dump<ov::float8_e5m2>(actual_mem, stream, file_stream, dump_raw);
    else if (mem_dt == cldnn::data_types::f8e4m3)
        dump<ov::float8_e4m3>(actual_mem, stream, file_stream, dump_raw);
+         else if (mem_dt == cldnn::data_types::f4e2m1)
+        dump<ov::float4_e2m1>(actual_mem, stream, file_stream, dump_raw);
    else if (mem_dt == cldnn::data_types::f8e8m0)
        dump<ov::float8_e8m0>(actual_mem, stream, file_stream, dump_raw);


+        if((activations_precision == ov::element::f4e2m1 || weights_precision == ov::element::f4e2m1) && scale_precision_to_use== ov::element::f8e8m0) {
+            rel_threshold = threshold_f16;
+        } else if (activations_precision == ov::element::f16) {
+            abs_threshold = threshold_f16;
        } else {
            abs_threshold = 1e-4f;
        }


akladiev and others added 30 commits September 23, 2025 17:00

Bump product version 2025.4.0

8ad01ac

Merge pull request openvinotoolkit#38 from akladiev/bump_product_version

98a833e

Bump product version 2025.4.0

wip

77688c0

wip

5db1ef5

wip

189f8e2

Apply doesFunctionContainF8DynQuanPatterns() in place of a WA

c6406a6

Add tests

ac4f8f4

Remove fusion code and enable running the standalone ocl kernel

6d89957

Add xetla's f8 utils

480b3ea

Add missing file

d10a3e1

Add f8e8m0 utils

8498ebf

Minor fix

d9666c1

Fix ocl code

e286397

Enable fp8 for reorder

5b1cbc2

Fix cl code

a298ae2

Fix typo

b39da31

Change names

b26fa58

Fix _intel_convert_e8m0_to_f32()

42f1335

Modify dyn quan kernels

73d5248

Fix typo

243cf09

Add necessary convert func call

0240ea0

Fix type

46384c5

Fix typo

f30bdc7

Fix typo

73ed5fe

wip

0523616

Disable GHA workflows

82f549d

Disable GHA workflows

133e609

wip

a8ce6ee

Fix f16 to f8e5m2 conversion functions

4c21268

Change unit tests' diff threshold for fp dtypes

d4b43b6

merezman and others added 16 commits May 26, 2026 09:29

Move val

be792c2

Merge open source 6b94f3d

0571bcf

Update ref kernel

095fe85

Fix typo

87ba77b

Update onednn_gpu submodule URL to public repo

07834fa

Restore MOE to match origin/merge_open_source_6b94f3de7e (keep DNNL_A…

83e1b26

…RG_HINT_MAX_GROUP_SIZE)

Fix opt kernel

98d0e80

Fix ref kernel

ad4e851

Move fp4_utils

e150d81

Merge master

1bda28e

Revert .github changes

d4c1c61

Clean changes

ae096cc

Use atomic write in ref kernel

2577ad6

Clean

f6b8211

github-actions Bot added category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations category: LP transformations OpenVINO Low Precision transformations labels Jul 17, 2026

sys-openvino-ci added the ExternalIntelPR External contributor from Intel label Jul 17, 2026

merezman marked this pull request as ready for review July 17, 2026 12:28

merezman requested review from a team as code owners July 17, 2026 12:28

tkrupa-intel self-requested a review July 17, 2026 12:33

maxnick self-assigned this Jul 17, 2026

maxnick requested a review from Copilot July 17, 2026 14:28

Copilot started reviewing on behalf of maxnick July 17, 2026 14:28 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GPU] Enable dynamic quantization for MXFP4 & regular FP4 dtypes#36944

[GPU] Enable dynamic quantization for MXFP4 & regular FP4 dtypes#36944
merezman wants to merge 172 commits into
openvinotoolkit:masterfrom
merezman:merezman/dyn_quant_f4_rebase

merezman commented Jul 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

merezman commented Jul 17, 2026

Details:

Tickets:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants