Different evaluate result between different VLMEvalKit version

[terry-for-github]

The official LLaVA-v1.5-7B model gets 1362 points in MME Perception scores under the newest code 69b7b5e.

But it turns out to be 1497 under the 027e38c commit.

I was sure about the consistent LLaVA code between this two experiement, why would this happened?

[terry-for-github]

I've check that once again. The problem still exists.

[lyklly]

i also notice this problem, however, when i check the code in 69b7b5e. I found that it only modified three parts：

1. Modify various log records, such as warning to logging
2. Change single quotes to double quotes
3. Modify some code problems
   Have you solved this problem?

[terry-for-github]

i also notice this problem, however, when i check the code in 69b7b5e. I found that it only modified three parts：

1. Modify various log records, such as warning to logging
2. Change single quotes to double quotes
3. Modify some code problems
   Have you solved this problem?

No. I just directly use the 027e38c version lol.

[lyzhongcrd]

I can confirm the issue on commit 9f21ee8.
"perception","reasoning","OCR","artwork","celebrity","code_reasoning","color","commonsense_reasoning","count","existence","landmark","numerical_calculation","position","posters","scene","text_translation" "1342.2684073629453","301.42857142857144","130.0","112.0","125.88235294117646","62.5","151.66666666666669","106.42857142857143","93.33333333333333","185.0","138.5","40.0","115.0","140.1360544217687","150.75","92.5"
The counting score is 93.3.
The counting score is 155 in a reference experiment (haotian-liu/LLaVA#927).
The score of counting is extremely low as Llava-1.5 tends to report the actual count instead of answering yes or no.

The same phenomenon is observed on MME-RealWorld-Lite, the counting score is 0.

This might be related to the system prompt or the instruction following capability of the model. But I am not sure how to address this.

Can @kennymckormick maybe have a look? Thanks!

[terry-for-github]

I can confirm the issue on commit 9f21ee8. "perception","reasoning","OCR","artwork","celebrity","code_reasoning","color","commonsense_reasoning","count","existence","landmark","numerical_calculation","position","posters","scene","text_translation" "1342.2684073629453","301.42857142857144","130.0","112.0","125.88235294117646","62.5","151.66666666666669","106.42857142857143","93.33333333333333","185.0","138.5","40.0","115.0","140.1360544217687","150.75","92.5" The counting score is 93.3. The counting score is 155 in a reference experiment (haotian-liu/LLaVA#927). The score of counting is extremely low as Llava-1.5 tends to report the actual count instead of answering yes or no.

The same phenomenon is observed on MME-RealWorld-Lite, the counting score is 0.

This might be related to the system prompt or the instruction following capability of the model. But I am not sure how to address this.

Can @kennymckormick maybe have a look? Thanks!

Good job! Thanks for the testing. I'm trying to figure out the problem, too.

[ZI-MA]

Anyone have solved it?

[zkril]

Anyone have solved it?

[AZYoung233]

Anyone have solved it?

[yizhilsy]

Anyone have solved it?

[cdllI]

Anyone have solved it?

[cdllI]

I’ve submitted a PR that addresses this issue:PR #1424