Clear finetuning example

[hamza13-12]

I followed the tutorial for using a different scheduler and after loading all the other SD components, I was successfully able to use that pipeline for inference. However, I am having trouble finding a script that helps me to finetune the Unet module after having loaded the components like this. Particularly, I am struggling with writing the training loop so it would be extremely beneficial if the the tutorial for using a different scheduler could be carried on to show us how then to finetune a stable diffusion model

[tolgacangoz]

Hi @hamza13-12,
Could this page be what you are looking for? If not exactly, you can give details.

[hamza13-12]

Hi @hamza13-12, Could this page be what you are looking for?

This is helpful but how do I how do I set up my own dataset into a format expected by this? I have two column in my dataset, image paths and image captions

[tolgacangoz]

IMHO, as a friendly recommendation, you should figure out how to do this. Try to understand how the code processes the dataset you will give. IMHO, this will be more beneficial for you. You can definitely get help from ChatGPT and/or Gemini.

[hamza13-12]

I have been trying to figure out how to do this.

https://colab.research.google.com/drive/1upKmc7BNkH7f0FPyRozjHDEYYtim8uh9?usp=sharing

You may view my progress on this colab link. But I have been struggling to identify how to finetune a diffusion model given a dataset of captions (prompts) and images. I'm thinking I could utilize ImageFolder on how to achieve this. But again, my question remains the same, as to how I could prepare my dataset for tinetuning suitable for a standard SD pipeline. Any help would be appreciated

[sayakpaul]

This should be an issue. This should be transferred to the Discussions.