Add vision evaluation metrics (exact_match, relaxed_accuracy, word_sort_ratio) (microsoft#2476)

## Summary

Extends Olive's evaluator framework with three vision-oriented accuracy
sub-metrics for VQA, ChartQA, and OCR evaluation, following the existing
pattern used for speech metrics (PR microsoft#2444).

## New Metrics

| Metric | Task Type | Suitable Benchmarks |
|--------|-----------|-------------------|
| `exact_match` | `vision-vqa` | AI2D, ScienceQA, TextVQA, MathVista,
MMMU, InterGPS |
| `relaxed_accuracy` | `vision-chart-qa` | ChartQA (±5% numeric
tolerance for numbers) |
| `word_sort_ratio` | `vision-ocr` | OCR benchmarks (word-level overlap)
|

## Public HuggingFace Datasets

These metrics are designed to work with publicly available datasets:

| Metric | Recommended Dataset | HuggingFace ID |
|--------|-------------------|----------------|
| `exact_match` | TextVQA | `facebook/textvqa` |
| `relaxed_accuracy` | ChartQA | `HuggingFaceM4/ChartQA` |
| `word_sort_ratio` | DocumentVQA | `HuggingFaceM4/DocumentVQA` |

Example configuration snippets are provided in
`docs/source/how-to/configure-workflows/metrics-configuration.md`.

## Changes

- **`olive/evaluator/metric.py`**: Adds `EXACT_MATCH`,
`RELAXED_ACCURACY`, `WORD_SORT_RATIO` to `AccuracySubType` enum
- **`olive/evaluator/accuracy.py`**: Implements the three metric classes
with multi-answer support
- **`olive/evaluator/olive_evaluator.py`**: Adds vision inference path
and task-metric validation
- **`olive/data/component/pre_process_data.py`**: Adds
`vision_vqa_pre_process` component
- **`olive/data/component/dataloader.py`**: Adds `vision_vqa_dataloader`
with custom collate_fn for PIL images
- **`olive/data/container/huggingface_container.py`**: Registers
`vision-vqa`, `vision-chart-qa`, `vision-ocr` task types with
appropriate dataloader
- **`olive/olive_config.json`**: Adds `vision` extras (pillow)
- **`docs/source/how-to/configure-workflows/metrics-configuration.md`**:
Adds vision metrics documentation with public dataset examples
- **`test/evaluator/test_accuracy.py`**: Unit tests covering all new
metrics

## Design

- Vision metrics are text-based (compare predicted answer string to
ground truth), task-dependent
- Multiple valid answers supported via `|` separator (metrics match
against any valid answer)
- Task-metric validation ensures incompatible combinations raise
`ValueError`
- Custom `vision_vqa_dataloader` handles PIL images with a collate_fn
that avoids PyTorch default collation issues
- PyTorch path: model processor handles images natively
- ONNX path: single forward pass assumed (classification-style VQA); for
autoregressive models use PyTorch evaluator with generation loop

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: jiafatom

docs/source/how-to/configure-workflows/metrics-configuration.md

@@ -153,3 +153,131 @@ WER can be used as an accuracy sub-type when your data pipeline returns text pre
 - `wer` (Word Error Rate): Measures transcription errors. Lower is better (defaults to `higher_is_better: false`).
 - `rtfx` (Real-Time Factor): Ratio of audio duration to inference time. Higher means faster (defaults to `higher_is_better: true`).
 ```
+
+## Vision Evaluation Metrics
+
+Olive provides three built-in accuracy sub-types for evaluating vision/multimodal models:
+
+| Metric | Task Type | Description | Suitable Benchmarks |
+|--------|-----------|-------------|---------------------|
+| `exact_match` | `vision-vqa` | Case-insensitive string equality | AI2D, ScienceQA, TextVQA, MMMU |
+| `relaxed_accuracy` | `vision-chart-qa` | ±5% numeric tolerance for numbers | ChartQA |
+| `word_sort_ratio` | `vision-ocr` | Word-level overlap ratio | OCR benchmarks |
+
+### Example: VQA with TextVQA (exact_match)
+
+```json
+{
+    "data_configs": [
+        {
+            "name": "textvqa_data",
+            "type": "HuggingfaceContainer",
+            "load_dataset_config": {
+                "data_name": "facebook/textvqa",
+                "split": "validation"
+            },
+            "pre_process_data_config": {
+                "type": "vision_vqa_pre_process",
+                "image_col": "image",
+                "question_col": "question",
+                "answer_col": "answers",
+                "limit": 100
+            },
+            "dataloader_config": {
+                "batch_size": 1
+            }
+        }
+    ],
+    "metrics": [
+        {
+            "name": "vqa_accuracy",
+            "type": "accuracy",
+            "data_config": "textvqa_data",
+            "sub_types": [
+                {"name": "exact_match", "priority": 1, "goal": {"type": "max-degradation", "value": 0.05}}
+            ]
+        }
+    ]
+}
+```
+
+### Example: ChartQA with relaxed_accuracy
+
+```json
+{
+    "data_configs": [
+        {
+            "name": "chartqa_data",
+            "type": "HuggingfaceContainer",
+            "load_dataset_config": {
+                "data_name": "HuggingFaceM4/ChartQA",
+                "split": "test"
+            },
+            "pre_process_data_config": {
+                "type": "vision_vqa_pre_process",
+                "image_col": "image",
+                "question_col": "question",
+                "answer_col": "answer",
+                "limit": 100
+            },
+            "dataloader_config": {
+                "batch_size": 1
+            }
+        }
+    ],
+    "metrics": [
+        {
+            "name": "chart_accuracy",
+            "type": "accuracy",
+            "data_config": "chartqa_data",
+            "sub_types": [
+                {"name": "relaxed_accuracy", "priority": 1, "goal": {"type": "max-degradation", "value": 0.05}}
+            ]
+        }
+    ]
+}
+```
+
+### Example: OCR with DocumentVQA (word_sort_ratio)
+
+```json
+{
+    "data_configs": [
+        {
+            "name": "docvqa_data",
+            "type": "HuggingfaceContainer",
+            "load_dataset_config": {
+                "data_name": "HuggingFaceM4/DocumentVQA",
+                "split": "validation"
+            },
+            "pre_process_data_config": {
+                "type": "vision_vqa_pre_process",
+                "image_col": "image",
+                "question_col": "question",
+                "answer_col": "answers",
+                "limit": 100
+            },
+            "dataloader_config": {
+                "batch_size": 1
+            }
+        }
+    ],
+    "metrics": [
+        {
+            "name": "ocr_accuracy",
+            "type": "accuracy",
+            "data_config": "docvqa_data",
+            "sub_types": [
+                {"name": "word_sort_ratio", "priority": 1, "goal": {"type": "max-degradation", "value": 0.05}}
+            ]
+        }
+    ]
+}
+```
+
+```{Note}
+- Vision metrics compare predicted answer strings to ground truth. The model's `post_func` must decode model output into text.
+- Use `batch_size: 1` since images have variable sizes.
+- Multiple valid answers (lists) are joined with `|` and the metric matches against any valid answer.
+- For ONNX models, provide a custom pre-process that applies the processor/tokenizer to produce numeric tensors.
+```

olive/data/component/dataloader.py

@@ -55,6 +55,14 @@ def no_auto_batch_dataloader(dataset, **kwargs):
     return DataLoader(dataset, batch_size=None, **kwargs)
 
 
+@Registry.register_dataloader()
+def vision_vqa_dataloader(dataset, batch_size=1, **kwargs):
+    from torch.utils.data import DataLoader
+
+    collate_fn = getattr(dataset, "collate_fn", None)
+    return DataLoader(dataset, batch_size=batch_size, collate_fn=collate_fn, **kwargs)
+
+
 @Registry.register_dataloader()
 def default_calibration_dataloader(
     dataloader,

olive/data/component/pre_process_data.py

@@ -379,3 +379,97 @@ def collate_fn(batch):
             return (audios, texts)
 
     return SpeechTranscriptionDataset(dataset, audio_col, text_col)
+
+
+@Registry.register_pre_process()
+def vision_vqa_pre_process(
+    dataset,
+    image_col: str = "image",
+    question_col: str = "question",
+    answer_col: str = "answer",
+    max_samples: Optional[int] = None,
+    limit: Optional[float] = None,
+    seed: int = 42,
+    **kwargs,
+):
+    """Pre-process data for vision VQA evaluation.
+
+    Loads image, question, and ground truth answer from a HuggingFace dataset.
+    Returns a dataset of ({"image": image, "question": question}, answer) pairs.
+
+    Note: This returns raw PIL images and question strings. For the PyTorch evaluator,
+    the model's own processor/tokenizer should be applied within the model's forward
+    method (or via a custom pre-process component). For the ONNX evaluator, provide a
+    custom pre-process component that applies the appropriate processor/tokenizer to
+    produce numeric tensors matching the model's io_config.
+
+    Args:
+        dataset: HuggingFace dataset with image, question, and answer columns.
+        image_col: Name of the image column. Defaults to "image".
+        question_col: Name of the question column. Defaults to "question".
+        answer_col: Name of the answer column. Defaults to "answer".
+        max_samples: Maximum number of samples (deprecated, use limit). Defaults to None.
+        limit: Sampling limit following Olive convention:
+            If >= 1: use first N samples.
+            If 0 < limit < 1: randomly sample that percentage.
+            If 0 or None: use all samples.
+        seed: Random seed for percentage-based sampling. Defaults to 42.
+        **kwargs: Additional arguments.
+
+    """
+    # Apply sampling: prefer limit over max_samples
+    effective_limit = limit if limit is not None else (max_samples if max_samples else 0)
+    if effective_limit and effective_limit != 0:
+        from random import Random
+
+        total = len(dataset)
+        if 0 < effective_limit < 1:
+            n = max(1, int(total * effective_limit))
+            rng = Random(seed)
+            indices = sorted(rng.sample(range(total), min(n, total)))
+            dataset = dataset.select(indices)
+        elif effective_limit >= 1:
+            n = min(int(effective_limit), total)
+            dataset = dataset.select(range(n))
+
+    class VisionVQADataset:
+        """Dataset that returns (input_dict, answer_text) pairs for VQA evaluation.
+
+        Note: Use batch_size=1 in dataloader config as images have variable sizes.
+        """
+
+        def __init__(self, hf_dataset, image_column, question_column, answer_column):
+            self.dataset = hf_dataset
+            self.image_column = image_column
+            self.question_column = question_column
+            self.answer_column = answer_column
+
+        def __len__(self):
+            return len(self.dataset)
+
+        def __getitem__(self, idx):
+            item = self.dataset[idx]
+            image = item[self.image_column]
+            question = item[self.question_column]
+            answer = item[self.answer_column]
+            # Handle list/tuple answers (some datasets have multiple valid answers)
+            # Join with | separator so metrics can match against any valid answer
+            if isinstance(answer, (list, tuple)):
+                answer = "|".join(str(a) for a in answer) if answer else ""
+            return {"image": image, "question": question}, str(answer)
+
+        @staticmethod
+        def collate_fn(batch):
+            """Collate VQA batches. Use with batch_size=1 for variable-size images.
+
+            Note: answers are always strings at this point (list/tuple answers are
+            joined with "|" in __getitem__), so no list-of-lists issue arises.
+            """
+            if len(batch) == 1:
+                input_dict, answer = batch[0]
+                return (input_dict, [answer])
+            inputs = [item[0] for item in batch]
+            answers = [item[1] for item in batch]
+            return (inputs, answers)
+
+    return VisionVQADataset(dataset, image_col, question_col, answer_col)

olive/data/container/huggingface_container.py

@@ -41,4 +41,16 @@ class HuggingfaceContainer(DataContainer):
         "speech-transcription": {
             DataComponentType.PRE_PROCESS_DATA.value: "speech_transcription_pre_process",
         },
+        "vision-vqa": {
+            DataComponentType.PRE_PROCESS_DATA.value: "vision_vqa_pre_process",
+            DataComponentType.DATALOADER.value: "vision_vqa_dataloader",
+        },
+        "vision-chart-qa": {
+            DataComponentType.PRE_PROCESS_DATA.value: "vision_vqa_pre_process",
+            DataComponentType.DATALOADER.value: "vision_vqa_dataloader",
+        },
+        "vision-ocr": {
+            DataComponentType.PRE_PROCESS_DATA.value: "vision_vqa_pre_process",
+            DataComponentType.DATALOADER.value: "vision_vqa_dataloader",
+        },
     }