sdxl controlnet training example script sudden converge when?

[zhuliyi0]

I am running the example training script with fill50k circle dataset, batch size 4 learning rate 1e-5. Didn't change other training parameters. Didn't notice any sudden converge after more than 10k steps. Would be nice if anyone that knows when it will happen, or if my learning rate is too low.

[tolgacangoz]

As of my understanding, the authors of the controlnet paper observed the sudden convergence at the 6133th step with Stable Diffusion V1.5, and expected to see usually in less than 10K optimization steps. Maybe, SD-XL requires 10K+ steps? How far have you gone beyond 10K steps?
Could you try to set --snr_gamma=5.0 for a likely faster convergence?

[tolgacangoz]

Also, could you look at this previous discussion?

[zhuliyi0]

Thanks for the reply. I added snr=5 and set lr=5e-5 with bs=4, got covergence at around 8600.