VAR reconstruction in training

[minrui-liu]

Hi authors, thanks for the great work!

I have a question regarding the image reconstruction pipeline. The paper mentions:

"The predicted representations are then passed through the VAR model to reconstruct pixel observations"

However, in the data preparation code, future_imgs are converted into discrete VAR tokens offline:
for img_path in future_imgs: var_path = ipath_to_varpath(img_path) scale = scale_schedule[0] arr = torch.load(var_path, map_location='cpu')[scale].tolist() var_token = [f"<|{token_id}|>" for token_id in arr] var_token = "".join(var_token)

It looks like the training phase only performs standard next-token prediction on these offline tokens and text. I couldn't find the code where representations are fed back into the VAR decoder for pixel-level reconstruction during training. Could you please clarify this or point me to the relevant code?

[panda7777777]

Thank you for your question.

To accelerate training, we precompute and cache the ground-truth image VAR tokens offline. This is equivalent to decoding representations with the VAR decoder during training, but avoids the additional computational cost.

We will release a version of the code that includes the VAR decoder for completeness.

[minrui-liu]

Thank you for your response！
I have another question regarding the VAR model mentioned in the paper.
In Section 3.3, it is mentioned that the framework uses a pretrained VAR model for CompV-CoT reasoning. Additionally, in the ablation studies in Section 4.3, the results show that using "Scale 4" achieves the best ISR performance.
However, I couldn't find the specific size or version of the pretrained VAR model used in the experiments (e.g., VAR-d16, VAR-d20, VAR-d24, VAR-d30, etc.) in the paper. Could you please let me know which exact VAR model weights were used during the training process?

[panda7777777]

We initialize the VAR model with the var_d16.pth weights. Since there is a domain gap between the images used for VAR pretraining and the images rendered by Habitat-Sim, we fully fine-tune it on all images from the trajectory data in the LH-VLN Bench training set. The VAR tokens used in our pipeline are generated from this fine-tuned model.

[minrui-liu]

Thank you so much for the prompt reply!

I was just wondering if there might be any plans to release the weights of this fine-tuned VAR model in the future? Having access to it would be incredibly valuable for the community to fully reproduce the CompV-CoT setup.