Float32 and float16 get different results

[jyangliu]

I just use controlnet to test float32 and float16 precision. And I get different results.Here is code:

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch
from diffusers.utils import load_image
import os

dtype = torch.float32
controlnet_model_path = "lllyasviel/control_v11f1p_sd15_depth"
controlnet = ControlNetModel.from_pretrained(controlnet_model_path, torch_dtype=dtype)

pipeline = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=dtype)
pipeline.to("cuda")

depth_map = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/depth.png")
generator = torch.manual_seed(33)
image = pipeline(
    prompt="best quality, high quality",
    image=depth_map,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
    num_inference_steps=30,
    generator=generator,
).images[0]
save_type = 'fp16' if dtype==torch.float16 else 'fp32'
os.makedirs(f"./workspace/official_controlnet", exist_ok=True)
# image.save(f"./workspace/official_controlnet/{save_type}_wo.png")
image.save(f"./workspace/official_controlnet/{save_type}.png")

float32 result is:

float16 result is:

I found a solution for this question: not use generator parameter in pipeline. Modified code is:

image = pipeline(
    prompt="best quality, high quality",
    image=depth_map,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
    num_inference_steps=30,
    # generator=generator,
).images[0]

With that setting, I got the same result like this:

Btw, my system is: windows 10 + torch2.0.0 + cuda116

[tolgacangoz]

I think it would not be appropriate to expect reproducible results for different floating point types. Eventually, they end up with different values during calculations, thus different pixels. For example, if we do just 6 multiplications, we start to see different results on the integer parts:

>>> import torch as th

>>> a = th.tensor(-2.3, dtype=th.float16)
>>> a ** 7
tensor(-341.2500, dtype=torch.float16)
>>> b = th.tensor(-2.3, dtype=th.float32)
>>> b ** 7
tensor(-340.4825)

Are you sure that you get the same picture when not passing the generator? Could you share a Colab notebook?

[jyangliu]

Ok, you can find my Jupyter notebook in here test_official_controlnet

[jyangliu]

And I don't expect float32 and float16 have the same values, but they should't be so different visually.

[tolgacangoz]

Thank you for sharing, it seems that you are right!
When I define the generator with CUDA and give this generator to the pipeline, the images seem almost identical, and the pixel values generated deviate only a few.

generator = torch.Generator('cuda').manual_seed(33)

Also, defining generator = torch.manual_seed(33) and not passing it to pipeline behaves like it; but if we pass generator = torch.Generator('cpu').manual_seed(33) to pipeline, the results are completely different between float32 and float16. I think this is not an expected behavior for a CPU-seeded generation 🤔.

[bghira]

try it without controlnet - just do the most basic example you can.