Skip to content

Commit 92b6b43

Browse files
committed
add some visuals
1 parent 49ea4d1 commit 92b6b43

1 file changed

Lines changed: 37 additions & 2 deletions

File tree

docs/source/en/modular_diffusers/quicktour.md

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -984,13 +984,14 @@ image = pipeline(
984984
image.save("modular_ipa_out.png")
985985
```
986986

987-
988-
## A more practical example
987+
## Building Advanced Workflows: The Modular Way
989988

990989
We've learned the basic components of the Modular Diffusers System. Now let's tie everything together with more practical example that demonstrates the true power of Modular Diffusers: working between with multiple pipelines that can share components.
991990

992991
In this example, we'll generate latents from a text-to-image pipeline, then refine them with an image-to-image pipeline. We will use IP-adapter, LoRA, and ControlNet.
993992

993+
### Base Text-to-Image
994+
994995
Let's setup the text-to-image workflow. Instead of putting all blocks into one complete pipeline, we'll create separate `text_blocks` for encoding prompts, `t2i_blocks` for generating latents, and `decoder_blocks` for creating final images.
995996

996997

@@ -1179,6 +1180,8 @@ image.save("modular_part2_t2i.png")
11791180

11801181
```
11811182

1183+
### Lora
1184+
11821185
Now let's add a LoRA to our pipeline. With the modular approach we will be able to reuse intermediate outputs from blocks that otherwise needs to be re-run. Let's load the LoRA weights and see what happens:
11831186

11841187
```py
@@ -1218,6 +1221,8 @@ image = decoder_node(latents=latents_lora, output="images")[0]
12181221
image.save("modular_part2_lora.png")
12191222
```
12201223

1224+
### IP-adapter
1225+
12211226
IP-adapter can also be used as a standalone pipeline. We can generate the embeddings once and reuse them for different workflows.
12221227

12231228
```py
@@ -1247,6 +1252,8 @@ image = decoder_node(latents=latents_ipa, output="images")[0]
12471252
image.save("modular_part2_lora_ipa.png")
12481253
```
12491254

1255+
### ControlNet
1256+
12501257
We can create a new ControlNet workflow by modifying the pipeline blocks, reusing components as much as possible, and see how it affects the generation.
12511258

12521259
We want to use a different ControlNet from the one that's already loaded.
@@ -1287,6 +1294,8 @@ refiner_blocks.sub_blocks.pop("image_encoder")
12871294
refiner_blocks.sub_blocks.pop("decode")
12881295
```
12891296

1297+
### Refiner
1298+
12901299
Create refiner pipeline. refiner has a different unet and use only one text_encoder so it is hosted in a different repo. We pass the same components manager to refiner pipeline, along with a unique "refiner" collection.
12911300

12921301
```py
@@ -1358,3 +1367,29 @@ refined_image = decoder_node(latents=refined_latents, output="images")[0]
13581367
refined_image.save("modular_part2_control_refine_out.png")
13591368
```
13601369

1370+
1371+
### Results
1372+
1373+
Here are the results from our modular pipeline examples. You can find all the generated images in the [Hugging Face dataset](https://huggingface.co/datasets/YiYiXu/testing-images/tree/main/modular_quicktour).
1374+
1375+
#### Base Text-to-Image Generation
1376+
| Base Text-to-Image | Base Text-to-Image (Refined) |
1377+
|-------------------|------------------------------|
1378+
| ![Base T2I](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_t2i.png) | ![Base T2I Refined](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_t2i_refine_out.png) |
1379+
1380+
#### LoRA
1381+
| LoRA | LoRA (Refined) |
1382+
|-------------------|------------------------------|
1383+
| ![LoRA](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_lora.png) | ![LoRA Refined](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_lora_refine_out.png) |
1384+
1385+
#### LoRA + IP-Adapter
1386+
| LoRA + IP-Adapter | LoRA + IP-Adapter (Refined) |
1387+
|-------------------|------------------------------|
1388+
| ![LoRA + IP-Adapter](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_ipa.png) | ![LoRA + IP-Adapter Refined](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_ipa_refine_out.png) |
1389+
1390+
### ControlNet + LoRA + IP-Adapter
1391+
| ControlNet + LoRA + IP-Adapter | ControlNet + LoRA + IP-Adapter (Refined) |
1392+
|-------------------|------------------------------|
1393+
| ![ControlNet + LoRA + IP-Adapter](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_control.png) | ![ControlNet + LoRA + IP-Adapter Refined](https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/modular_quicktour/modular_part2_control_refine_out.png) |
1394+
1395+

0 commit comments

Comments
 (0)