Collect results all metrics

[haideraltahan]

collect-results: emit all metrics instead of picking one per task

Closes #75

Problem

• collect_results extracted a single metric per task using a hardcoded priority list (acc,none → acc → acc_norm → f1 → ...) or a per-task override in the task_metrics map in task-groups.yaml. This silently discarded metrics that were already computed and sitting in the result JSONs:

• A flores200 result already contains bleu, bleu_1, bleu_4, chrf++, and all their
  _stderr variants — only one made it to the CSV

• Tasks with both acc and acc_norm (e.g. hellaswag, arc_challenge) only reported one

• task_metrics had to be manually extended every time a new task was added, and could
  easily drift out of sync

Solution

Replace _resolve_metric with _extract_all_metrics: iterate over every key in a result dict and emit one CSV row per numeric value. The lm-eval harness ,{filter} suffix (e.g. acc,none → acc) is stripped for clean metric names; when two raw keys collapse to the same stripped name the first wins. _stderr keys are treated as ordinary metrics and get their own rows — they are not filtered out.

The output schema gains a metric_name column and produces one row per (model, task, n_shot, metric_name) tuple:

model_name	task	n_shot	metric_name	performance
model-a	flores200:eng_Latn-ita_Latn	0	chrf++	57.676
model-a	flores200:eng_Latn-ita_Latn	0	chrf++_stderr	0.422
model-a	flores200:eng_Latn-ita_Latn	0	bleu	21.160
model-a	flores200:eng_Latn-ita_Latn	0	bleu_1	0.512
model-a	hellaswag	10	acc	0.450
model-a	hellaswag	10	acc_norm	0.630
model-a	hellaswag	10	acc_norm_stderr	0.004

Users can filter on metric_name downstream and no pre-computed metric is silently discarded.

Changes

• oellm/main.py: replace _resolve_metric with _extract_all_metrics; remove the yaml/importlib.resources block that loaded task_metrics; fix a pre-existing falsy-zero bug where n_shot=0 fell through to "unknown" in or-chain
• oellm/resources/task-groups.yaml: remove the entire task_metrics section (172 lines) — it is no longer consulted by any code
• scripts/pivot_results.py: add metric_name to required columns; include it in task_label (e.g. "hellaswag (10-shot) [acc_norm]") so pivot columns stay unambiguous when multiple metrics exist for the same task
• tests/test_collect_results.py (new): 17 tests covering multiple metrics per task, correct values, suffix stripping, _stderr as a separate row, non-numeric exclusion, duplicate name deduplication, full flores200 metric set, output schema validation, multi-file merge, and edge cases

[swag2198]

Thanks @haideraltahan! I like the option to fetch all metrics that are computed, but at the same time, I feel it is good to provide only the "standardized metric" per task in the output. For example, if I wanted to do basic comparison between two models, I would care most about which metric is most commonly used for the task and just pick that.
Would it make sense to add a flag to the collect-results utility --fetch-all-metrics for the intended behaviour of this PR?

[haideraltahan]

@swag2198 fair point! Will add that adjustment :)

[kerkathy]

Just a small question. When filter is not none, what does it indicate and is it still safe to remove?

[haideraltahan]

Great question — when the filter is not none, it indicates that a post-processing step was applied to the model outputs before computing the metric (e.g., normalization, strict matching, or task-specific transformations).

In the current implementation, we strip the {metric},{filter} suffix down to {metric}, which can lead to collisions if multiple filters are present (e.g., acc,none and acc,strict both becoming acc). However, for the current evaluation suite, all tasks emit metrics with the default none filter (e.g., acc,none, acc_norm,none, f1,none). Therefore, stripping the filter suffix does not lead to collisions in practice.