Added opensubtitles-eu#70
Merged
Merged
Conversation
geoalgo
approved these changes
May 22, 2026
34 tasks
Vendor the custom opensubtitles_multi40 loader + per-pair tasks and wire two
{lang}-templated groups (English<->EU, both directions) over the 25 EU
languages the dataset ships. Resolve translation-pair task names to their
non-English side (mirroring flores200) and add the hr/no two-letter aliases
so bracket scoping and language filtering work.
haideraltahan
force-pushed
the
OpenSubtitles
branch
from
87fe0f5 to
d5d67b0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I added OpenSubtitles Multi40 translation as two
lm-evaluation-harnessgroups —opensubtitles-eu-en-xx(English→EU) andopensubtitles-eu-xx-en(EU→English) — at 0-shot, scored with BLEU (chrF also computed), using the Helsinki-NLP/OpenSubtitles2024-40-langs-15-movies dataset. OpenSubtitles isn't in lm-eval-harness or lighteval, so I vendored a custom loader + per-pair tasks undercustom_lm_eval_tasks/opensubtitles_multi40/. One of the Evals from #89.Language coverage — EU subset only
The dataset ships 40 languages; 25 of them are in our EU set, and I include all 25 (each in both directions):
EU languages the dataset does not ship are omitted: Irish, Maltese, Catalan, Basque, Galician, Bosnian, Georgian, Macedonian, Albanian, Icelandic.
Both groups are
{lang}templates withvalid_langs. Since these are translation pairs, each task resolves to its non-English side (mirroring theflores200handling intask_groups.py), soopensubtitles-eu-xx-en[deu_Latn]correctly selectsde→en. I also added thehr→hrv_Latnandno→nob_Latnaliases so Croatian and Norwegian resolve.Metric
bleu(sacrebleu), matching the custom task'smetric_list.