I notice that there seems to be a memory leak issue during training.
On my K80, I set the batch size to 24 and the GPU memory consumption is about 4000MB. However, as the training process goes on, the GPU memory consumptions increases and a RuntimeError: CUDA error: out of memory is raised about 30 minutes later. If I set the batch size to 16 the error would not take place but the memory still increase.
I notice that there seems to be a memory leak issue during training.
On my K80, I set the batch size to 24 and the GPU memory consumption is about 4000MB. However, as the training process goes on, the GPU memory consumptions increases and a
RuntimeError: CUDA error: out of memoryis raised about 30 minutes later. If I set the batch size to 16 the error would not take place but the memory still increase.