SDXL Fooocus Inpaint

[WaterKnight1998]

Is your feature request related to a problem? Please describe.
I have seen that diffusers StableDiffusionXLInpaintPipeline generates worse results than SD 1.5 pipeline.

Describe the solution you'd like.
Include Fooocus inpaint patch, you could specify with a new loader.
Weights are available right now in hub.
https://huggingface.co/lllyasviel/fooocus_inpaint

[Laidawang]

they also seem to use fooocus_inpaint_head.pth I'm not quite sure what this will do, I read the code and maybe an additional patch for unet?

The inpaint_v26.fooocus.patch is more similar to a lora, and then the first 50% executes base_model + lora, and the last 50% executes base_model.
There is no doubt that fooocus has the best inpainting effect and diffusers has the fastest speed, it would be perfect if they could be combined.

[asomoza]

Actually it seems more like a controlnet, something more like this one: https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl.

They also use a custom sampler for the inpainting, but I agree, it would be nice to be able to use those in diffusers.

You can read about it here: lllyasviel/Fooocus#414

[WaterKnight1998]

The inpaint_v26.fooocus.patch is more similar to a lora, and then the first 50% executes base_model + lora, and the last 50% executes base_model. There is no doubt that fooocus has the best inpainting effect and diffusers has the fastest speed, it would be perfect if they could be combined.

I was reading the code and they download the model here: https://github.com/lllyasviel/Fooocus/blob/dc5b5238c83c63b4d7814ba210da074ddc341213/modules/config.py#L398-L399

This function is called here: https://github.com/lllyasviel/Fooocus/blob/dc5b5238c83c63b4d7814ba210da074ddc341213/modules/async_worker.py#L301 You can see inpaint_patch_model_path is passed to base_model_additional_loras. They have an strange coded for applying the lora.

After model is loaded you can see in following tabs that they apply the head in top of the result of applying the lora

[Laidawang]

Actually it seems more like a controlnet, something more like this one: https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl.实际上，它看起来更像是一个控制网，更像是这个：https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl。

They also use a custom sampler for the inpainting, but I agree, it would be nice to be able to use those in diffusers.他们还使用自定义采样器进行修复，但我同意，如果能够在扩散器中使用它们那就太好了。

You can read about it here: lllyasviel/Fooocus#414您可以在这里阅读：lllyasviel/Fooocus#414

I have read the comparison between Fooocus and comfyui of loading lora. I think they are basically the same.
COMFY: https://github.com/comfyanonymous/ComfyUI/blob/53c8a99e6c00b5e20425100f6680cd9ea2652218/comfy/lora.py#L13
FOOOCUS:
https://github.com/lllyasviel/Fooocus/blob/dc5b5238c83c63b4d7814ba210da074ddc341213/ldm_patched/modules/lora.py#L13

This can also be confirmed from the code provided by @WaterKnight1998. it just defined different names to ensure that only fooocus can load it correctly.

[WaterKnight1998]

Yup, that's the problem I saw. I had a difficult time trying to load in diffusers I didn't managed to map keys of layers into diffusers expected format :(

[Laidawang]

https://github.com/lllyasviel/Fooocus/blob/main/modules/inpaint_worker.py#L187 Another thing worth considering is how to implement this patch for inpaint head model.

[WaterKnight1998]

Actually it seems more like a controlnet, something more like this one: https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl.实际上，它看起来更像是一个控制网，更像是这个：https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl。
They also use a custom sampler for the inpainting, but I agree, it would be nice to be able to use those in diffusers.他们还使用自定义采样器进行修复，但我同意，如果能够在扩散器中使用它们那就太好了。
You can read about it here: lllyasviel/Fooocus#414您可以在这里阅读：lllyasviel/Fooocus#414

I have read the comparison between Fooocus and comfyui of loading lora. I think they are basically the same. COMFY: https://github.com/comfyanonymous/ComfyUI/blob/53c8a99e6c00b5e20425100f6680cd9ea2652218/comfy/lora.py#L13 FOOOCUS: https://github.com/lllyasviel/Fooocus/blob/dc5b5238c83c63b4d7814ba210da074ddc341213/ldm_patched/modules/lora.py#L13

Ok, both codes are the same. Is it possible to load ComfyUI weights in diffusers?

[WaterKnight1998]

https://github.com/lllyasviel/Fooocus/blob/main/modules/inpaint_worker.py#L187 Another thing worth considering is how to implement this patch for inpaint head model.

But the code is just updating the first conv, no?

[Laidawang]

https://github.com/lllyasviel/Fooocus/blob/main/modules/inpaint_worker.py#L187 Another thing worth considering is how to implement this patch for inpaint head model.

But the code is just updating the first conv, no?

You are right, but we also need to use it in diffusers as input to start with

[Laidawang]

Maybe consider loading it in comfy and saving it as overall weights and then using it in diffusers?

[Laidawang]

But as I saw in fooocus, the base model will still be used in the second stage, so the most elegant way is to load and unload it freely.

[WaterKnight1998]

But as I saw in fooocus, the base model will still be used in the second stage, so the most elegant way is to load and unload it freely.

What do you mean with this?

[Laidawang]

For example, in fooocus inpainting, assuming that 30 steps of sampling are performed, xl_base_model + inpainting_model will be used in the first 15 steps, and xl_base_model will be switched to separate inference in the last 15 steps.
https://github.com/lllyasviel/Fooocus/blob/main/modules/async_worker.py#L307 see here.

[asomoza]

yeah I saw it afterwards, they switched to a custom model for inpainting, how good is the inpainting? can any of you post an example? if its really good maybe I can try or even better, someone from the diffusers team, but they'll probably need solid proof to work on it.