Skip to content

Support scalar results in combined evaluations#776

Open
zanvari wants to merge 1 commit into
huggingface:mainfrom
zanvari:investigate-combine-wer-cer
Open

Support scalar results in combined evaluations#776
zanvari wants to merge 1 commit into
huggingface:mainfrom
zanvari:investigate-combine-wer-cer

Conversation

@zanvari

@zanvari zanvari commented Jul 5, 2026

Copy link
Copy Markdown

Summary

Fixes #516.

Some evaluation modules (e.g. wer and cer) return scalar values rather than dictionaries. CombinedEvaluations._merge_results() assumes every module returns a dictionary and attempts to call .keys() on each result, causing combined evaluation to fail.

This change wraps scalar results using the corresponding evaluation module name before merging, allowing scalar-returning evaluation modules to be combined while preserving the existing behavior for dictionary-returning modules.

Tests

Added a regression test covering:

evaluate.combine(["wer", "cer"]).compute(...)

which now returns the expected combined evaluation results without raising an exception.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can't calculate combined metric (WER + CER)

1 participant