Conformance testing

[stefanhahmann]

The purpose of this issue is to make an estimation possible how much features the FIJI plugin resulting from this repo covers.

We have discovered three sources of test OME-Zarr datasets:

• Image Data Repository (IDR)

  • Real world datasets
  • mainly microscopy
  • Web page: https://idr.github.io/ome-ngff-samples/
  • CSV representation: https://raw.githubusercontent.com/IDR/ome-ngff-samples/refs/heads/main/_data/table.csv

• OMEZarr OpenSciVisDatasets

  • Real world datasets
  • medical + material sciences imaging techniques, e.g. MRI, CT
  • https://github.com/InsightSoftwareConsortium/OMEZarrOpenSciVisDatasets/
  • Metadata (readme): https://raw.githubusercontent.com/InsightSoftwareConsortium/OMEZarrOpenSciVisDatasets/refs/heads/main/README.md
  • Metadata (json): http://klacansky.com/open-scivis-datasets/datasets.json

• NGFF spec

  • Official test datasets
  • Artificial datasets
  • Only OME metadata, not actual image data
  • https://github.com/ome/ngff-spec/tree/main/examples

Tasks

• Create a CSV that combines dataset URLs from IDR and OpenSciVis

  • Extract dataset URLs from IDR metadata
  • Extract dataset URLs from OpenSciVis metadata
  • Define a common CSV schema
  • Add the generated CSV as a test resource in this project

• Investigate CI-based conformance testing using the combined CSV

  • Open each dataset URL with the plugin's supported opening methods
  • Compare extracted metadata against expected metadata stored in the CSV (e.g. x,y,z,c,t dimensions, units, scale factor, etc.)
  • Decide whether these tests should run on every CI build or only on demand

• Generate image-backed versions of the NGFF specification example datasets

  • Preserve the metadata structure from the NGFF examples
  • Add a small amount of image data to each dataset
  • Verify that the generated datasets can be opened by the current reader stack
  • Add representative datasets to the repository as test resources
  • Add unit tests covering dataset opening with different scenarios (e.g. wells, plates, transformations, bf2raw, etc.)