CIFAR-10 Evaluation

[harveymannering]

Thanks for this codebase, it is super useful!

I have a question about the CIFAR-10 FID evaluation. If you look at the this line, the FID evaluation for CFM is done using the train split of CIFAR-10. Shouldn't this be done using the test set? Or does it not matter? If not, why not?

[Charley-xiao]

Hey @harveymannering,

It's a common practice in generative modeling, especially GAN and diffusion, to compute FID on the training set. For example, in DDPM, "Our FID score is computed with respect to the training set, as is standard practice; when we compute it with respect to the test set, the score is 5.24..."

However, there are papers like this one that argue train-FID cannot represent generalizability, so you'll also see people computing FID on the test set in tasks like image-to-image translation. So it really depends on how previous works were doing it.

Hope that helps!