Skip to content

[CPU] Fuse affine-free RMSNorm to keep variance math in f32 under bf16 enforcement#36718

Open
goyaladitya05 wants to merge 1 commit into
openvinotoolkit:masterfrom
goyaladitya05:fix/cpu-rms-fusion-no-gamma
Open

[CPU] Fuse affine-free RMSNorm to keep variance math in f32 under bf16 enforcement#36718
goyaladitya05 wants to merge 1 commit into
openvinotoolkit:masterfrom
goyaladitya05:fix/cpu-rms-fusion-no-gamma

Conversation

@goyaladitya05

@goyaladitya05 goyaladitya05 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Details:

Cause: RMSNorm without gamma (norm_elementwise_affine=False, used by DiT-family diffusion transformers such as LTX-Video) never fused into the internal RMS op, because CPU registered RMSFusion with enable_without_gamma=false. The norm stayed as a decomposed Power -> ReduceMean -> Add -> Sqrt -> Divide -> Multiply chain, and bf16 enforcement lowered its intermediates (x², rsqrt) to bf16. One forward pass is barely affected, but over a 30-step diffusion sampling loop the rounding compounds into a latent variance collapse (~20% std loss vs f32), grey, bad-quality video.
PyTorch on the same hardware is unaffected: _fused_rms_norm computes in f32 internally and rounds once at the output.

This PR:

  • Registered RMSFusion with enable_without_gamma=true (gpu already does; this is why the GPU f16 path was not affected).
  • Supports single-input internal::RMS in the RMSNorm node (kernel's scalar-scale path with an exact 1.0f) and in the DecomposeRMSNorm fallback.
  • The existing JIT RMS kernel already matches PyTorch numerics: bf16 I/O, f32 accumulation. MatMuls stay in bf16.

Partial fix for: #36715

AI Assistance:

  • AI assistance used: no / yes yes
  • If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).
    used at several points. Verified by running inference using LTX-Video using original reproduction script (output attached).

Copilot AI review requested due to automatic review settings July 4, 2026 15:11
@goyaladitya05 goyaladitya05 requested review from a team as code owners July 4, 2026 15:11
@github-actions github-actions Bot added the category: CPU OpenVINO CPU plugin label Jul 4, 2026
@sys-openvino-ci sys-openvino-ci added the ExternalPR External contributor label Jul 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a CPU bf16 accuracy regression for affine-free RMSNorm (no gamma) commonly used in DiT-family diffusion transformers (e.g., LTX-Video). It ensures the RMS variance computation stays in f32 by enabling fusion into the internal RMS op even when gamma is absent, matching the intended mixed-precision behavior (bf16 I/O with f32 internal math) and mitigating variance collapse over iterative sampling loops.

Changes:

  • Enable CPU ov::pass::RMSFusion for RMSNorm patterns without gamma (enable_without_gamma=true) in the post-LPT pipeline.
  • Extend CPU RMSNorm execution and the DecomposeRMSNorm fallback to support single-input internal ov::op::internal::RMS (identity scale path).
  • Add/extend CPU functional tests to cover fusion and single-input RMS behavior (including a subgraph reproducing an AdaLN-style “(1 + scale)” modulation).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/x64/rms_norm_no_gamma.cpp New functional subgraph test that builds an affine-free RMSNorm pattern and asserts it fuses to a single internal RMS node under CPU execution.
src/plugins/intel_cpu/tests/functional/custom/single_layer_tests/instances/x64/rms_norm.cpp Adds single-input (“no gamma”) shape sets and instantiates the RMS test suite for them.
src/plugins/intel_cpu/tests/functional/custom/single_layer_tests/classes/rms_norm.cpp Updates the RMS single-layer test model/input generation to handle optional scale input (single-input internal RMS).
src/plugins/intel_cpu/src/transformations/transformation_pipeline.cpp Registers ov::pass::RMSFusion with enable_without_gamma=true for CPU x64 post-LPT transformations.
src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/decompose_rms_norm.cpp Updates decomposition of internal RMS to support the no-gamma (single-input) form.
src/plugins/intel_cpu/src/nodes/rms_norm.cpp Updates the CPU RMSNorm node to accept an optional scale input (identity scale when absent) and adjusts primitive descriptors/shape checks accordingly.

@goyaladitya05

Copy link
Copy Markdown
Contributor Author

#36718 and #36719 combined are able to fix the issue. I validated by building OpenVINO from these pull requests and running the reproduction script, obtained output is attached. Quality improved a lot, inference time was increased by about ~1 sec, but the speedup against fp32 is still almost 3.8x.

ltx_t2v_output.5.mp4

@maxnick

maxnick commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

@nshchego , could you please review?

@likholat

Copy link
Copy Markdown
Contributor

@nshchego, could you please take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: CPU OpenVINO CPU plugin ExternalPR External contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants