Reproducing MLM L 1-16 FID

[Aphoh]

Hello! Thanks for the thorough paper!

I'm trying to reproduce your results for MLM L, which state an FID of 3.7, but I'm getting an FID of around 18...

Here's my modification to the sample_fid.sh script:

TORCH_DISTRIBUTED_DEBUG=INFO torchrun --nnodes=1 --nproc_per_node=4 sample_fid.py \
 --model GPT-L --modeling mlm --sample_dir images_fid_mlm \
 --ckpt /model_home/mlmL-1-16.pth \
 --dataset custom --codebook_size 16 --norm_first --num-classes 1000 \
 --code-dim 16 --token-each 1 --pos_type rope2d --cfg-scale 3.0 --cfg_schedule linear \
 --deterministic --gen_iter_num 256 --temperature 9.0 \
 --num_images 32000 --per_proc_batch_size 16 --global_seed 64

And then using the npz from here to compare I get an FID of 18.66.

I had to fix a few small bugs with some missing arguments being passed into the MLM model, but other than that everything was quite straightforward. Am I missing something? The generated images look not too bad but not great.

[Pepper-lll]

Hello! Thanks for the thorough paper!

I'm trying to reproduce your results for MLM L, which state an FID of 3.7, but I'm getting an FID of around 18...

Here's my modification to the sample_fid.sh script:

TORCH_DISTRIBUTED_DEBUG=INFO torchrun --nnodes=1 --nproc_per_node=4 sample_fid.py \
 --model GPT-L --modeling mlm --sample_dir images_fid_mlm \
 --ckpt /model_home/mlmL-1-16.pth \
 --dataset custom --codebook_size 16 --norm_first --num-classes 1000 \
 --code-dim 16 --token-each 1 --pos_type rope2d --cfg-scale 3.0 --cfg_schedule linear \
 --deterministic --gen_iter_num 256 --temperature 9.0 \
 --num_images 32000 --per_proc_batch_size 16 --global_seed 64

And then using the npz from here to compare I get an FID of 18.66.

I had to fix a few small bugs with some missing arguments being passed into the MLM model, but other than that everything was quite straightforward. Am I missing something? The generated images look not too bad but not great.

You can check to see if there are the same random seeds for the same category, which will result in duplicate images.
Besides, the --gen_iter_num is important for MLM. We found it's best to be around 10 for mlm, we got the best fid when --gen_iter_num equals 9.

[Aphoh]

@Pepper-lll Thanks for your reply! All the images appear unique, so I think seeds are being set ok. I set --gen_iter_num 9 for the MLM-L-1-16 model and the FID score got worse, it's 21.80...

Is there anything I'm missing in this script? I tried to match the reported cfg schedule, temperature, and everything else.

TORCH_DISTRIBUTED_DEBUG=INFO torchrun --nnodes=1 --nproc_per_node=4 sample_fid.py \
 --model GPT-L --modeling mlm --sample_dir images_fid_mlm \
 --ckpt /model_home/mlmL-1-16.pth \
 --dataset custom --codebook_size 16 --norm_first --num-classes 1000 \
 --code-dim 16 --token-each 1 --pos_type rope2d --cfg-scale 3.0 --cfg_schedule linear \
 --deterministic --gen_iter_num 9 --temperature 9.0 \
 --num_images 32000 --per_proc_batch_size 32 --global_seed 64

[Pepper-lll]

@Pepper-lll Thanks for your reply! All the images appear unique, so I think seeds are being set ok. I set --gen_iter_num 9 for the MLM-L-1-16 model and the FID score got worse, it's 21.80...

Is there anything I'm missing in this script? I tried to match the reported cfg schedule, temperature, and everything else.

TORCH_DISTRIBUTED_DEBUG=INFO torchrun --nnodes=1 --nproc_per_node=4 sample_fid.py \
 --model GPT-L --modeling mlm --sample_dir images_fid_mlm \
 --ckpt /model_home/mlmL-1-16.pth \
 --dataset custom --codebook_size 16 --norm_first --num-classes 1000 \
 --code-dim 16 --token-each 1 --pos_type rope2d --cfg-scale 3.0 --cfg_schedule linear \
 --deterministic --gen_iter_num 9 --temperature 9.0 \
 --num_images 32000 --per_proc_batch_size 32 --global_seed 64

Hello, sorry for the late reply. My sampling script for calculating FID is not so clever... you need to make sure that --per_proc_batch_size * --nproc_per_node * --num-classes equals --num_images, then you can generate the same number of images for each category. Your current setting will generate 4*32 pics for classes from 0 to 249, and zero pic for the last classes, which will cause a large fid. If you want to use 4 nodes, you should set --per_proc_batch_size==8.