[GPU] Keep RMS normalization nodes in FP32 to prevent NaN from FP16 overflow by zhanmyz · Pull Request #36504 · openvinotoolkit/openvino

zhanmyz · 2026-06-22T10:30:57Z

Details:

Keep RMS normalization nodes in FP32 to prevent NaN caused by FP16 overflow in models with large intermediate activations (ParlerTTS, flan-T5).

Description of the issue (symptom, root-cause, how it was resolved)

Symptom:
- ParlerTTS Mini text_encoder (T5-based, google/flan-t5-base) produced all-NaN outputs on GPU FP16 (encoder_last_hidden_state, logits).
- GPU FP32 and CPU modes passed.
Root Cause:
- The first INF appears at:
  - __module.encoder.block.1.layer.1.DenseReluDense.wo/aten::linear/MatMul
  - prefix dump: INF_COUNT=5, FINITE_MAX=60928
- Those INF values propagate through residual add and trigger NaN at downstream RMS (INF -> rsqrt(INF)=0 -> INF*0=NaN).
- The failure is caused by FP16 precision-path policy on this chain, not by an isolated RMS kernel implementation bug.
Resolution:
- Added MarkRMS MatcherPass inside MarkSugraphsToKeepInMixedPrecision (mark_subgraphs_to_keep_in_mixed_precision.cpp).
- The pass matches all ov::op::internal::RMS nodes via wrap_type<RMS>() and marks them with disable_conversion(node, f16).
- Intentionally do not force all MatMul ops to FP32: that would over-broaden precision changes and risk unnecessary performance impact, while the required fix is to enforce the downstream RMS FP32 contract and let mixed-precision alignment choose equivalent valid forms.
- Nodes already marked by other passes (e.g. DisableFP16CompForGemma3RMSPattern) are skipped.
- This triggers automatically when keep_precision_sensitive_in_fp32=true is set in ConvertPrecision, which is the standard GPU plugin configuration.
- RMS is element-wise and memory-bound — keeping it in FP32 has negligible performance impact across all models.

Reproduction step and snapshot (if applicable. Do not attach for customer model)

python -m pytest test_ovc_mo.py
--tb=native
--env_conf=.automation/env_config.yml
--test_conf=.automation/test_configs/desktop_test_config.yml
-m "not launch_only_if_manually_specified"
--pregen_irs=irs_mapping.csv
--tf_models_version=1.15.2
--modules pipelines/production/pytorch/heavy/parler_tts.py
-k "Pytorch_ParlerTTSMini_batch_1_device_GPU_precision_FP16"
--dynamism_type=None
--log-cli-level INFO

Problematic graph

Original IR

- Current IR

Checklist

Is it a proper fix? (not a workaround)
Did you include test case for this fix, if necessary?
Did you review existing test that can be extended to cover this scenario? Which test did you review?

Tickets:

CVS-187992

…verflow ### Details: - Added DisableFP16CompForRMSNormBlock ModelPass that unconditionally marks all RMS nodes with disable_conversion(f16), preventing NaN caused by FP16 overflow in models like ParlerTTS (T5-based). - The pass is a catch-all complement to DisableFP16CompForGemma3RMSPattern: it handles RMS nodes not covered by specific pattern matchers. - Only RMS nodes are marked (not MatMul consumers), so performance impact is negligible (RMS is element-wise, memory-bound). - Added unit test verifying RMS stays FP32 after ConvertPrecision. ### Tickets: - *CVS-187992* Signed-off-by: zhanmyz <yazhan.ma@intel.com>

Copilot

Pull request overview

This PR addresses GPU FP16 numerical instability in RMS normalization by preventing FP16 precision conversion for RMS nodes, avoiding INF→NaN propagation in models with large intermediate activations (e.g., ParlerTTS / flan-T5) when INFERENCE_PRECISION_HINT=f16 is used.

Changes:

Adds a new GPU ModelPass (DisableFP16CompForRMSNormBlock) that marks ov::op::internal::RMS with ov::disable_conversion(..., f16) (skipping nodes already marked).
Integrates the new pass into the Intel GPU transformations pipeline after RMS fusion and existing pattern-based RMS protections.
Adds a unit test intended to validate that RMS stays protected while MatMul consumers are not marked.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
`src/plugins/intel_gpu/tests/unit/transformations/disable_fp16_compression_rms_test.cpp`	Adds a unit test for the new pass behavior around RMS vs MatMul marking.
`src/plugins/intel_gpu/src/plugin/transformations/disable_fp16_comp_for_rms_norm_block.hpp`	Declares the new ModelPass to keep RMS nodes from being FP16-converted.
`src/plugins/intel_gpu/src/plugin/transformations/disable_fp16_comp_for_rms_norm_block.cpp`	Implements marking of all `ov::op::internal::RMS` nodes to disable FP16 conversion.
`src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp`	Registers the new pass in the GPU transformation pipeline prior to `ConvertPrecision`.

…t test ### Details: - Add found_rms/found_matmul flags to ensure test fails if target nodes are missing after transformation passes, instead of silently passing. - Addresses code review feedback on PR. ### Tickets: - *CVS-187992* Signed-off-by: zhanmyz <yazhan.ma@intel.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

… config ### Details: - Added transformation_callback(rms) guard in DisableFP16CompForRMSNormBlock to honor PassConfig callbacks, consistent with other GPU plugin passes. - Updated unit test to use ConvertPrecision with the same parameters as the real GPU pipeline (keep_precision_sensitive_in_fp32=true, convert_input_output_precision=false, empty type_to_fuse_map). - Added output element type assertions: RMS stays f32, MatMul becomes f16. ### Tickets: - *CVS-187992* Signed-off-by: zhanmyz <yazhan.ma@intel.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Signed-off-by: zhanmyz <yazhan.ma@intel.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

…r (marks all RMS nodes). - Update pass RTTI and registration in GPU transformations pipeline. - Harden unit test by removing brittle MatMul output type assertion. Signed-off-by: zhanmyz <yazhan.ma@intel.com>

### Details: - Renamed disable_fp16_comp_for_rms_norm_block.* to disable_fp16_comp_for_all_rms.* to match pass scope (all RMS nodes). - Updated includes in transformations_pipeline.cpp and disable_fp16_compression_rms_test.cpp. - No behavior change in pass logic; this is naming and discoverability cleanup requested by code review. ### Tickets: - *CVS-187992*

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

CuriousPanCake · 2026-07-03T08:17:43Z

Shouldn't this be a part of the ConvertPrecision transformation where we mark sensitive nodes to keep them in high precision?

mryzhov

We have a special transformations for such cases, please extend them instead of introducing the new one: src/common/transformations/src/transformations/fp16_compression/mark_subgraphs_to_keep_in_mixed_precision.cpp

…epInMixedPrecision ### Details: - Add MarkRMS pass inside MarkSugraphsToKeepInMixedPrecision to mark all ov::op::internal::RMS nodes with disable_conversion(f16). This keeps RMS in FP32 during ConvertPrecision, preventing NaN caused by FP16 overflow in models with large intermediate activations (ParlerTTS/T5). - The fix extends the existing precision-sensitivity infrastructure rather than introducing a separate plugin-specific pass. RMS is element-wise and memory-bound, so keeping it in FP32 has negligible performance impact. ### Tickets: - Tickets: CVS-187992 Signed-off-by: zhanmyz <yazhan.ma@intel.com>

zhanmyz · 2026-07-07T06:55:26Z

Shouldn't this be a part of the ConvertPrecision transformation where we mark sensitive nodes to keep them in high precision?

You're right. I've moved the fix into MarkSugraphsToKeepInMixedPrecision (inside mark_subgraphs_to_keep_in_mixed_precision.cpp) as a MarkRMS pass, alongside MarkDivWithEps, MarkExp, and MarkRandomUniform. The separate GPU-only pass has been removed.

zhanmyz · 2026-07-07T06:56:13Z

We have a special transformations for such cases, please extend them instead of introducing the new one: src/common/transformations/src/transformations/fp16_compression/mark_subgraphs_to_keep_in_mixed_precision.cpp

Done. Added MarkRMS class inside mark_subgraphs_to_keep_in_mixed_precision.cpp and registered it in MarkSugraphsToKeepInMixedPrecision::run_on_model. The standalone GPU pass (disable_fp16_comp_for_all_rms.cpp/.hpp) has been deleted. Unit test moved to mark_subgraph_to_keep_in_mixed_precision_test.cpp as well.

Rewrite MarkRMS_keeps_rms_in_fp32 test to use model_ref + FunctionsComparator bidirectional comparison, consistent with all other tests in mark_subgraph_to_keep_in_mixed_precision_test.cpp. The reference model now also marks gamma constant (expected behavior: PropagateUpMarkToKeepInMixedPrecision propagates to Constants feeding into marked nodes, same as eps_const in MarkDivWithEps tests). Tickets: CVS-187992 Signed-off-by: zhanmyz <yazhan.ma@intel.com>

…ck check ### Details: - Move the is_conversion_disabled check from inside the callback into the wrap_type predicate. This avoids entering the callback for RMS nodes that are already marked, as suggested in code review. ### Tickets: - CVS-187992 Signed-off-by: zhanmyz <yazhan.ma@intel.com>

CuriousPanCake

LGTM

github-actions · 2026-07-09T07:49:24Z

CI Doctor — Merge Queue failure on this PR

Pipeline: Linux (Ubuntu 24.04, Python 3.12)
Failure: Smart CI fails: fastcore 2.0.1 breaks ghapi 1.0.4 - L.starmap AttributeError
Automatic restart: ❌ Not triggered — deterministic Dependencies failure; a restart would fail identically until the dependency is fixed.

Possible remedy

In .github/actions/smart-ci/requirements.txt, pin fastcore to fastcore>=1.5.4,<2.0.0 (add the <2.0.0 upper bound). This is a one-line fix and will unblock the entire merge queue.
Alternatively, upgrade ghapi to a version that supports fastcore 2.x (check the ghapi release notes for compatibility).
After merging the fix, re-queue any blocked PRs.

What happened

The Smart_CI job crashed before evaluating any PR changes because fastcore 2.0.1 (a major version bump released today) removed the L.starmap() method that ghapi 1.0.4 depends on in GhApi.__init__. This failure is unrelated to the GPU RMS changes in this PR — it is a CI infrastructure dependency issue affecting all merge-queue runs today.

This is the 3rd occurrence of this failure today (07:11Z, 07:26Z, 07:40Z). Previous affected runs: 29000379233 (PR #36780), 29000703414 (PR #36773).

Generated by CI Failure Doctor — Merge Queue · 123.7 AIC · ⌖ 16.5 AIC · ⊞ 17.6K · ◷

e-ddykim

no perf. impact from llm daily tests

…m FP16 overflow (openvinotoolkit#36504)" (openvinotoolkit#36892) ### Details: - Revert PR openvinotoolkit#36504 which forces all RMS normalization nodes to FP32 - This change causes significant global LLM performance regression. ### Root Cause PR openvinotoolkit#36504 was introduced to fix NaN in ParlerTTS Mini (CVS-187992) by keeping RMS nodes in FP32. However, it unconditionally marks ALL RMS nodes across all models, adding FP16↔FP32 conversion overhead on every transformer layer. ### Tickets: - *190812*

zhanmyz requested review from a team as code owners June 22, 2026 10:30

zhanmyz added the category: GPU OpenVINO GPU plugin label Jun 22, 2026

zhanmyz requested review from Copilot and wilson-seok June 23, 2026 01:48

Copilot started reviewing on behalf of zhanmyz June 23, 2026 01:51 View session