Evaluation, Reproducibility, Benchmarks Meeting 43

Minutes of Meeting 43

Date: 6th May, 2026

Present

• Lena
• Rucha
• Carole
• Anne
• Annika
• Michela

Updates

• From Michela

  • Started filling in the shared document on benchmarking.
  • Drafted purposes of benchmarking
  • Drafted criteria a benchmark must meet
  • Additional categories suggested for review (e.g. carbon footprint, energy consumption); group to revisit list to make sure nothing is missed
  • Literature review to integrate with Carole and Annika’s suggestion

• From Annika:

  • BIAS reporting guidelines update
  • Update is being finalised and will be sent shortly; data items in particular have been substantially expanded
  • Michela to be added to the BIAS authors/contributors loop so she can review against her literature review and flag any gaps
  • Agreed approach: use the updated BIAS as the basis for the working group’s data items rather than duplicating effort, and layer additional items on top where needed

• From Carole

  • Work on metrics almost completed (around 90%)

• Next steps:

  • Develop a tool to automatically generate the "identity card" for benchmark datasets (Michela) The tool would extract metadata and quality-relevant parameters from publicly available datasets to populate the identity card automatically Particularly tractable for datasets distributed as DICOM (rich metadata); more limited for NIfTI/JPEG-converted datasets Proposed approach: start with a proof-of-principle on a few key fields, then extend coverage in line with the BIAS-based guidelines

  • Will discuss at the MONAI meeting the possibility/interest of integrating MICCAI datasets into MONAI (Carole/Annika)

  • Michela to share the list of reviewed papers in the GDrive

Administrative Item

• Anne is stepping down from the working group; today was her last meeting. Alessa Hering proposed as Anne’s replacement; she is interested and will contact Annika directly to formalise