Skip to content

Guard against zero std_cfg in rescale_noise_cfg (#13425)#13704

Open
jbbqqf wants to merge 1 commit intohuggingface:mainfrom
jbbqqf:fix/13425-rescale-noise-cfg-zero-std
Open

Guard against zero std_cfg in rescale_noise_cfg (#13425)#13704
jbbqqf wants to merge 1 commit intohuggingface:mainfrom
jbbqqf:fix/13425-rescale-noise-cfg-zero-std

Conversation

@jbbqqf
Copy link
Copy Markdown

@jbbqqf jbbqqf commented May 10, 2026

Fixes #13425.

Summary

rescale_noise_cfg divides std_text by std_cfg without any numerical guard:

std_cfg = noise_cfg.std(dim=list(range(1, noise_cfg.ndim)), keepdim=True)
noise_pred_rescaled = noise_cfg * (std_text / std_cfg)   # ← nan / inf when std_cfg == 0

If noise_cfg has zero variance (e.g. a constant or zero-valued guided prediction at the start of a schedule, or numerical edge cases), std_cfg == 0, the division produces nan/inf, and the silent corruption propagates through the rest of the diffusion process — no exception is raised.

Fix

Clamp std_cfg to torch.finfo(noise_cfg.dtype).eps before the divide. In the degenerate case the rescaling becomes a no-op (0 * <finite> == 0) instead of nan. In the non-degenerate case the clamp is below numerical noise, so the rescaling result is unchanged in practice.

std_cfg = std_cfg.clamp(min=torch.finfo(noise_cfg.dtype).eps)
noise_pred_rescaled = noise_cfg * (std_text / std_cfg)

The fix was applied to the canonical copy in pipelines/stable_diffusion/pipeline_stable_diffusion.py and propagated to all 31 "# Copied from" duplicates via python utils/check_copies.py --fix_and_overwrite. python utils/check_copies.py (no flags) is green.

Reproduce BEFORE/AFTER yourself (copy-paste)

# --- BEFORE: origin/main, expect NaN ---
git fetch origin main && git checkout origin/main
python - <<'PY'
import torch
from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion import rescale_noise_cfg
noise_cfg = torch.zeros(1, 4, 8, 8)            # std == 0
noise_pred_text = torch.randn_like(noise_cfg)
result = rescale_noise_cfg(noise_cfg, noise_pred_text, guidance_rescale=1.0)
print('any nan:', torch.isnan(result).any().item())  # Expected: True
PY

# --- AFTER: this branch ---
git fetch origin pull/<PR>/head:fix && git checkout fix
python - <<'PY'
import torch
from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion import rescale_noise_cfg
noise_cfg = torch.zeros(1, 4, 8, 8)
noise_pred_text = torch.randn_like(noise_cfg)
result = rescale_noise_cfg(noise_cfg, noise_pred_text, guidance_rescale=1.0)
print('any nan:', torch.isnan(result).any().item())  # Expected: False
print('any inf:', torch.isinf(result).any().item())  # Expected: False
print('max abs:', result.abs().max().item())          # Expected: 0.0
PY

I confirmed both runs locally:

  • origin/mainany nan: True
  • this branch → any nan: False, any inf: False, max abs: 0.0

A second sanity check with a non-degenerate noise_cfg = torch.randn(...) produces an unchanged result on this branch (the clamp is below numerical noise).

Notes

  • 32 files touched, all the change is the same 4-line guard before the divide.
  • No public API change; pure numerical hardening.

🤖 Disclosure: Authored with assistance from Claude (Anthropic). Reproducer was run against both origin/main and this branch end-to-end.

`rescale_noise_cfg` divides `std_text` by `std_cfg` without any numerical
guard. When `noise_cfg` has zero variance (e.g. a constant or zero-valued
guided prediction at the start of a schedule), `std_cfg` is `0`, the division
produces `nan`/`inf`, and `noise_cfg * (std_text / 0)` silently corrupts the
diffusion output (huggingface#13425).

Clamp `std_cfg` to `torch.finfo(noise_cfg.dtype).eps` before dividing so the
rescaling becomes a no-op in the degenerate case (`noise_cfg == 0` produces
`0 * <finite> == 0`) instead of `nan`.

Propagated to all 32 in-tree copies via `python utils/check_copies.py
--fix_and_overwrite`.
@github-actions github-actions Bot added fixes-issue size/M PR with diff < 200 LOC pipelines and removed size/M PR with diff < 200 LOC labels May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Division by zero in rescale_noise_cfg can produce NaNs during inference

1 participant