SDXL_base-1.0 VS SDXL_base-1.0-inpainting 0.1 for inpainting

[john09282922]

Describe the bug

Dear all,

I have quick question on Unet config on SDXL-base-1.0 and SDXL-base-1.0-inpainting-0.1.
In the each Unet config, each in_channel parameter has different value like 4 and 9.
Why they have different setting?
and which model is better for inpainting? my experiment results is that utilizing SDXL-base-1.0 is better than inpainting-0.1 and possible to utilize other recent architecture.
But not sure, is it okay to use base-1.0 for inpainting? even if not adding masking work during training.

@yiyixuxu @hlky @asomoza @vladmandic

Reproduction

.

Logs

System Info

.

Who can help?

No response

[asomoza]

Why they have different setting?

Hi, they have different channels because the model accepts an image and a mask as input channels.

which model is better for inpainting?

It depends, you can still use the base SDXL for inpainting as you tried, but it's not trained for that task so if you get a decent or not image depends on luck, the mask and the type of image you're using. On the other side, the inpainting model for SDXL has a problem where it desaturates the image so the inpainting is not 100% accurate with the original image in respect to colors and saturation.

possible to utilize other recent architecture.

yes, you can try differential diffusion and controlnets.

I have answered this question multiple times with more detail, you can search for them if you want more details.

[john09282922]

Why they have different setting?

Hi, they have different channels because the model accepts an image and a mask as input channels.

which model is better for inpainting?

It depends, you can still use the base SDXL for inpainting as you tried, but it's not trained for that task so if you get a decent or not image depends on luck, the mask and the type of image you're using. On the other side, the inpainting model for SDXL has a problem where it desaturates the image so the inpainting is not 100% accurate with the original image in respect to colors and saturation.

possible to utilize other recent architecture.

yes, you can try differential diffusion and controlnets.

I have answered this question multiple times with more detail, you can search for them if you want more details.

Thanks for the answer, I want to hear about more detail information on question.1
In the inpainting code,

if num_channel_unet=9, initially make lantent_model_input and make noise_pred.
elseif num_channel_unet=4, make latents and then applying mask part like (1-mask)*latent + mask * latent.

Not sure, which method is better or correct?

@asomoza, @vladmandic @hlky

[asomoza]

This is not more an issue and more of a discussion so converting, and please stop tagging multiple people on each update, they already know when you update the main issue/discussion and also if I answered they probably won't answer it anymore, and there's one that is not part of the diffusers team, so if you want an answer from them you should open an issue in the correct repository.