[CPU] Fuse affine-free RMSNorm to keep variance math in f32 under bf16 enforcement#36718
[CPU] Fuse affine-free RMSNorm to keep variance math in f32 under bf16 enforcement#36718goyaladitya05 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses a CPU bf16 accuracy regression for affine-free RMSNorm (no gamma) commonly used in DiT-family diffusion transformers (e.g., LTX-Video). It ensures the RMS variance computation stays in f32 by enabling fusion into the internal RMS op even when gamma is absent, matching the intended mixed-precision behavior (bf16 I/O with f32 internal math) and mitigating variance collapse over iterative sampling loops.
Changes:
- Enable CPU
ov::pass::RMSFusionfor RMSNorm patterns without gamma (enable_without_gamma=true) in the post-LPT pipeline. - Extend CPU RMSNorm execution and the DecomposeRMSNorm fallback to support single-input internal
ov::op::internal::RMS(identity scale path). - Add/extend CPU functional tests to cover fusion and single-input RMS behavior (including a subgraph reproducing an AdaLN-style “(1 + scale)” modulation).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/x64/rms_norm_no_gamma.cpp | New functional subgraph test that builds an affine-free RMSNorm pattern and asserts it fuses to a single internal RMS node under CPU execution. |
| src/plugins/intel_cpu/tests/functional/custom/single_layer_tests/instances/x64/rms_norm.cpp | Adds single-input (“no gamma”) shape sets and instantiates the RMS test suite for them. |
| src/plugins/intel_cpu/tests/functional/custom/single_layer_tests/classes/rms_norm.cpp | Updates the RMS single-layer test model/input generation to handle optional scale input (single-input internal RMS). |
| src/plugins/intel_cpu/src/transformations/transformation_pipeline.cpp | Registers ov::pass::RMSFusion with enable_without_gamma=true for CPU x64 post-LPT transformations. |
| src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/decompose_rms_norm.cpp | Updates decomposition of internal RMS to support the no-gamma (single-input) form. |
| src/plugins/intel_cpu/src/nodes/rms_norm.cpp | Updates the CPU RMSNorm node to accept an optional scale input (identity scale when absent) and adjusts primitive descriptors/shape checks accordingly. |
|
#36718 and #36719 combined are able to fix the issue. I validated by building OpenVINO from these pull requests and running the reproduction script, obtained output is attached. Quality improved a lot, inference time was increased by about ~1 sec, but the speedup against fp32 is still almost 3.8x. ltx_t2v_output.5.mp4 |
|
@nshchego , could you please review? |
|
@nshchego, could you please take a look? |
Details:
Cause: RMSNorm without gamma (norm_elementwise_affine=False, used by DiT-family diffusion transformers such as LTX-Video) never fused into the internal RMS op, because CPU registered RMSFusion with enable_without_gamma=false. The norm stayed as a decomposed Power -> ReduceMean -> Add -> Sqrt -> Divide -> Multiply chain, and bf16 enforcement lowered its intermediates (x², rsqrt) to bf16. One forward pass is barely affected, but over a 30-step diffusion sampling loop the rounding compounds into a latent variance collapse (~20% std loss vs f32), grey, bad-quality video.
PyTorch on the same hardware is unaffected: _fused_rms_norm computes in f32 internally and rounds once at the output.
This PR:
Partial fix for: #36715
AI Assistance:
used at several points. Verified by running inference using LTX-Video using original reproduction script (output attached).