train_dreambooth_lora.py loss function cannot converge

[ysw0530]

When I trained Dreambooth using train_dreambooth_lora.py , the loss always keeps fluctuating between 0.01 and 1.0. The setup I used was almost same as the readme.md :

export MODEL_NAME="stable-diffusion-v1-5/stable-diffusion-v1-5"
export INSTANCE_DIR="dreambooth_dataset/dataset/keramer-face"
export CLASS_DIR="regularization_image"
export OUTPUT_DIR="output/lora_text_encoder_face_steps=1400"

accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --with_prior_preservation --prior_loss_weight=1.0 \
  --class_data_dir=$CLASS_DIR \
  --instance_prompt="a sks person" \
  --class_prompt="a person" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=100 \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1400 \
  --validation_prompt="a sks person as the terminator." \
  --validation_epochs=1 \
  --seed="0" \
  --num_class_images=200 \
  --train_text_encoder 

[a-r-r-o-w]

The typical diffusion finetuning loss curves look like they are randomly fluctuating between 0 and 1. You are probably seeing expected behaviour, and your dreambooth lora has learned the concept you're trying to train - you could verify with inference using the lora.

[ysw0530]

I trained for a total of 1400 steps and saved a checkpoint every 100 steps. After inference one by one, it is true that the model learned my instance, but its performance is not stable, and the text encoder seems to have lost its original prior semantics (known by evaluating the Clip-T metric in the Dreambooth paper). So I previously suspected that the model did not fully converge. Maybe my hyper-parameters are set wrong, but I don't find which is wrong.

[a-caycioglu]

If your loss curve looks like this, it is expected behaviour. If it's fluctuating but not decreasing in general, there may be something wrong.

Can you share your loss curve?