train failed when EMA mode is off

[frozenzo]

I found that this is because the model.train() did not open again when evaluation ends.
solution: just move mode.train() to the epoch loop:

• model.train()
  for epoch in range(args.start_epoch, args.epochs):

->

for epoch in range(args.start_epoch, args.epochs):

•    model.train()
  

[frozenzo]

dear author, could you kindly show results that did not use EMA for testing?

[DingYuan0118]

same question here. Considering the long training time, I wanna know did you finish the training for the fixed version?

[kekmodel]

It's late, but I'll check.

[TonyLianLong]

I met exactly the same thing: on the second epoch, the loss becomes nan without modification when EMA is off.

[dahuaxiya]

yes, I met the same thing too. If test the model, the loss will be nan when ema is closed.However, the loss is normal without test step.