Evaluation, Reproducibility, Benchmarks Meeting 45

Minutes of Meeting 45

Date: 24th June, 2026

Present

• Carole
• Annika
• Lena
• Oliver
• Keyvan
• Alessa

Updates

• Carole
  • Carole is working on the tests for the metrics reloaded implementation
  • Annika will send the segmentation masks from Fabian
• Olivier
  • The CI paper revision is in preparation
  • Goal is to resubmit before summer

Applications / new members

• We have received an application by a master student
• Annika will ask her for her CV and about concrete contributions to the working group
• Generally, we could think about calls / surveys for interns or short membership for specific projects where we need help

MICCAI dynamic benchmark

• Lena suggested the idea of creating a MICCAI dynamic benchmark
• The MICCAI SIG for Challenges will approach all challenge organizers and ask them to contribute (at least part of) their test data for benchmarking of current multimodal frontier models. Their tasks (e.g. disease classification) would have to be translated into VQA pairs (question and reference answer where possible)
• The goal would then be to test models in a zero-shot fashion
• The working group is interested in joining this effort
• It was discussed whether training data should be included as well; the tendency was towards no
• For challenges hosted on Grand Challenge, we could potentially leverage that data was already uploaded
• We would want to do the translation to VQA