What information is stored in the json file beginning with gdion?

[JiannBai]

For example:
gdino_novel_previous_metric_sunrgbd_novel_oracle_2d.json
gdino_previous_eval_sunrgbd_base_oracle_2d.json
gdino_sunrgbd_base_oracle_2d.json
gdino_sunrgbd_novel_oracle_2d.json
These files are used in the evaluation, but the information stored in these files corresponds to hundreds of instances for a single image, and the amount of information stored in each of these files is different for each of the images, and I don't understand the rationale for using these files as the gt for the evaluation.
How do I properly understand and use the information in these files?

[yaojin17]

These files are the Grounding Dino's 2D detection results. We provide these files to facilitate the evaluation process. They are not used as the gt for evaluation, but instead as the 2D detection input of class-agnostic 3D lifting.

Sorry for the naming confusion. For your examples:

• "gdino_novel_previous_metric_sunrgbd_novel_oracle_2d.json" refers to the Grounding Dino's 2D detection results on sunrgbd's novel categories under standard evaluation protocol.
• "gdino_previous_eval_sunrgbd_base_oracle_2d.json" refers to the Grounding Dino's 2D detection results on sunrgbd's base categories under standard evaluation protocol.
• "gdino_sunrgbd_base_oracle_2d.json" refers to the Grounding Dino's 2D detection results on sunrgbd's base categories under our proposed target-aware evaluation protocol.

[JiannBai]

Thank you for your reply! I am currently learning about this field and your work is very good learning material for me, I think I may need to ask you some more questions later.

[yaojin17]

Thanks for your interest! Feel free to reach out if you have any questions!

[JiannBai]

I'm a little confused about the use of ROI HEADS in training and Evaluation. Specifically, there are two ROI HEADS provided in the code: ROIHeads3DGDINO and ROIHeads3D, and the configuration file OVMono3D_dinov2_SFP.yaml used in the Train and Evaluation commands provided in README.md uses ROIHeads3D by default. Based on the gdino prefixed json file information you provided, the relevant commands in README.md, and the configuration file and code, I'm guessing that you are using ROIHeads3D for Train and Evaluation of OVMono3D-LIFT, and that you will be Grounding Dino's 2D open vocabulary detection is separated from 3D information detection. It is as follows(if it is not correct, can you explain the correct training and Evaluation ideas):

Train: only the 3D regression part is trained, Grounding Dino is not involved in the computation, so ROIHeads3D is used for training.

Evaluation: 1) 2D open vocabulary detection performed by Grounding Dino (according to the description of Target-Aware, the vocabulary and the number of words used in each image are consistent with the category and the number of gt's), and then the detection results are saved as a json file with gdino prefix;
2) Read the json file with gdino prefix as ORACLE2D information, then only use ROIHeads3D to detect category-independent 3D information detection, and finally integrate 2D information and 3D information together.

Also, I would like to know if the Train and Evaluation can be done correctly when ROI HEADS is set to ROIHeads3DGDINO , and if so, how should I set the command parameters?

Actually, I'm a master student in Artificial Intelligence and I'm the first student within my group to do monocular 3D detection of open vocabulary, so I might be asking a little more specific questions. Thank you very much for your kind answers!

[Hanqi7921]

I am currently reproducing and expanding this work, I think you have a very good question. I am confused about some points as well, hope we can have a short discussion if you dont mind~

[JiannBai]

Sure.！I'd be happy to have some discussion with you！

[yaojin17]

Hi XiangDong Bai (@JiannBai), Thanks for your interest!!!

Yes, there are two ROI heads in our code. ROIHeads3D is used during the training and evaluation. ROIHeads3DGDINO is used for the demo.

During training, a 2D box head and the 3D regression part are trained. The 2D box head is to provide noisy 2D detection to make the 3D regression more robust.

Grounding Dino is not involved in the computation, so ROIHeads3D is used for training.

This is correct.

During Evaluation, the 2D box head is discarded and we use the Grounding Dino to provide the open-vocab 2D detection ability.

Evaluation: 1) 2D open vocabulary detection performed by Grounding Dino (according to the description of Target-Aware, the vocabulary and the number of words used in each image are consistent with the category and the number of gt's), and then the detection results are saved as a json file with gdino prefix;
2) Read the json file with gdino prefix as ORACLE2D information, then only use ROIHeads3D to detect category-independent 3D information detection, and finally integrate 2D information and 3D information together.

These are also correct.

Also, I would like to know if the Train and Evaluation can be done correctly when ROI HEADS is set to ROIHeads3DGDINO , and if so, how should I set the command parameters?

In the current implementation, ROIHeads3DGDINO is only used in the demo code to provide in-the-wild open-vocab detection results. You could absolutely integrate ROIHeads3DGDINO into the training and evaluation by adapting the code. However, we don't use ROIHeads3DGDINO in training and evaluation because Ground Dino is slow. It may double or more than double the training and evaluation time.

I hope that could solve your questions. Feel free to ask more questions about our project. Enjoy the open vocabulary monocular 3D detection research!

[JiannBai]

Your reply solved my query perfectly, thanks a lot!!!!

[JiannBai]

Now I need to use Grounding DINO to perform 2D target detection on modified kitti, sunrgbd, and arkitscenes data and compare it with the 2D target detection results you provided, so I would like to know that when you generate gdino_kitti_novel_oracle_2d.json, gdino_ sunrgbd_novel_oracle_2d.json, and gdino_arkitscenes_novel_oracle_2d.json, what values are set for the Grounding DINO parameters “box_threshold” and “text_threshold” respectively for a fair comparison. Looking forward to your reply!

[yaojin17]

"box_threshold" is 0.001. “text_threshold” is 0.25.