Could you please double check the Video-MME-Long score for MA-LMM in your main result table?

[vinlincc]

The score for MA-LMM for Video-MME-Long is 7.2, which seems a big discrepancy compared to its other score in your main result table.

I believe MA-LMM paper does not present the benchmark on Video-MME-Long, therefore, there might be some mistakes in your evaluation pipeline, which makes your result table not as valid as it should.

[NIneeeeeem]

I noticed that the performance of all Socratic Models on VideoMME-Long is lower than the results shown on the leaderboard. Could you please explain why this is the case? @hyc2026

[hyc2026]

The score for MA-LMM for Video-MME-Long is 7.2, which seems a big discrepancy compared to its other score in your main result table.

I believe MA-LMM paper does not present the benchmark on Video-MME-Long, therefore, there might be some mistakes in your evaluation pipeline, which makes your result table not as valid as it should.

We directly use the model and code provided by MA-LMM for testing. The results indicate that it has weak instruction-following ability in multiple-choice questions, as it often fails to explicitly provide option letters and instead outputs short answers. However, the evaluation code of Video-MME-Long strictly requires matching the option letters; this mismatch leads to lower evaluation metrics.

[hyc2026]

I noticed that the performance of all Socratic Models on VideoMME-Long is lower than the results shown on the leaderboard. Could you please explain why this is the case? @hyc2026

Thanks for your question. The difference comes from the experimental settings:

• In the VideoMME paper, the model receives both the entire video and the question as input. This is a standard setup for video question answering benchmarks, but it doesn’t reflect realistic online streaming scenarios, since an agent cannot re-watch the entire history video every time a new question arrives.
• Our Socratic Models are designed for streaming. The video is processed in 30-second clips. The description of each clip is stored as long-term memory. When a question is asked, the agent retrieves relevant memories and then generates an answer.

Our method emphasizes scalability and practicality for streaming, rather than "full-video" access, which is more aligned with real-world usage.