About the poor editability of DreamBooth for personal concept images (overfitting?)

[daeunni]

Hi, I am struggling with the editability problem when running DreamBooth + LoRA.
I manually collected the plushy images from Google, and put them to DreamBooth + LoRA. (I utilized default setting)
However, I got pretty poor results like below. Most of the generations don't follow the prompt, and they seem to be overfitted by few-shot user images.

Does anyone know why it happened, and how can I address it?
Also, does any papers address these overfitting issues?

[tolgacangoz]

Hi @daeunni. If you haven't already, I would like to recommend you to read this blog post: Training Stable Diffusion with Dreambooth using 🧨 Diffusers. They say "... but it's hard to select the right set of hyperparameters and it's easy to overfit."

[daeunni]

Hi, @StandardAI, thanks for your recommendation. I will try to modify the learning rate.
However, when I ran CustomDiffusion using the above same few-shot images, CustomDiffusion totally failed (=personalized concept didn't appear at the generated images) to generate personalized images even though I ran many steps. (~800) Do you have any idea also?

[tolgacangoz]

@daeunni I'm wondering what the consequences of hyperparameter tuning of DreamBooth runs were.
Could you share with us what the exact command for running Custom Diffusion was?

[sayakpaul]

Maybe @linoytsaban @apolinario has some suggestions?

[linoytsaban]

Hey @daeunni! just by observing your results it seems that indeed its overfit, so the default settings are not the best for this scenario.
As you mentioned I'd recommend trying to reduce the learning rate (if you're not using an adaptive one) as well as the number of steps and size of the train batch.
it should defeintly be possible to get better results, just need some more hyper-param tweaking IMO :)