StableDiffusionXLControlNetInpaintPipeline.from_single_file and Custom SDXL Turbo model

[Yakonrus]

Hi

https://civitai.com/models/129666?modelVersionId=294995
Can anyone tell me how I can use this model in the Inpainting pipeline?

controlnet_xl = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
)
controlnet_xl.to("cuda:0")

controlnet_xl_base_vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix",
    torch_dtype=torch.float16,
)
controlnet_xl_base_vae.to("cuda:0")

controlnet_xl_base_pipe = StableDiffusionXLControlNetInpaintPipeline.from_single_file(
    "./weights/RealitiesEdgeXLSDXL_TURBOXLV2.safetensors",
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
    add_watermarker=False,
    controlnet=controlnet_xl,
    vae=controlnet_xl_base_vae,
)
controlnet_xl_base_pipe.to("cuda:0")

ValueError: Trying to set a tensor of shape torch.Size([320, 4, 3, 3]) in "weight" (which has shape torch.Size([320, 9, 3, 3])), this look incorrect.

[fabiorigano]

you should maybe first convert the model to diffusers format in this way:

python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path ".\RealitiesEdgeXLSDXL_TURBOXLV2.safetensors" --from_safetensors --pipeline_class_name StableDiffusionXLInpaintPipeline --dump_path "<path/to/converted/model>"

then you can load it:

controlnet_xl_base_pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained("<path/to/converted/model>",torch_dtype=torch.float16,controlnet=controlnet_xl,vae=controlnet_xl_base_vae)```

[sayakpaul]

#6986 will hopefully make this much more streamlined.