More RAM is consumed as training epoch increases

[hongvin]

I have repeated the experiment for many times and eventually at the 14th epoch, the process is killed, mainly due to my CPU RAM is exhausted. My current configuration is 8 core CPUs, 30GB CPU RAM, P5000 16GB GPU.

I have tried to reduce memory bank size, but it's still the same.

[Sudo42b]

reduce num_worker thread and batch size.

[CreatedTRYNA]

I have the same problem when I try to train this model, and anything can I do?