Greek basic language modeling benchmark suite

[alexliap]

ILSP Greek evaluation suite

Summary

Adds the ILSP Greek benchmark suite (ilsp_greek), a collection of Greek-language LLM evaluation tasks published by the Institute for Language and Speech Processing (ILSP), Athena Research Center. These are the datasets used to evaluate ILSP's Greek foundation models Meltemi-7B and Llama-Krikri-8B (Voukoutis et al., Krikri: Advancing Open Large Language Models for Greek, arXiv:2505.13772).

The suite bundles 9 task families into a single ilsp_greek group comprising 65 leaf tasks:

Task	Type	Source dataset
arc_challenge_greek	Science QA / reasoning (MCQA)	ilsp/arc_greek
hellaswag_greek	Commonsense NLI (MCQA)	ilsp/hellaswag_greek
truthfulqa_greek_mc1	Truthfulness (MCQA)	ilsp/truthful_qa_greek
truthfulqa_greek_mc2	Truthfulness (MCQA)	ilsp/truthful_qa_greek
mgsm_direct_greek	Grade-school math (generate-until)	ilsp/mgsm_greek
winogrande_greek	Commonsense coreference (MCQA)	ilsp/winogrande_greek
mmlu_greek	Multitask knowledge, 57 subjects (MCQA)	ilsp/mmlu_greek
medical_mcqa_greek	Medical MCQA, ILSP official suite	ilsp/medical_mcqa_greek
mcqa_greek_asep	Greek-native ASEP civil-service exam MCQA	ilsp/mcqa_greek_asep

Design notes

• Translate the data, not the template. The 5 include-based tasks (arc_challenge_greek, hellaswag_greek, truthfulqa_greek_mc1/mc2, mgsm_direct_greek) reuse the upstream English task logic and only override dataset_path (plus per-dataset fixes). The prompt scaffolding stays in English, matching ILSP's Open-LLM-Leaderboard-style setup and the existing arc_challenge_mt_el task. The scored content (questions and answer choices) is Greek from the datasets themselves.

• winogrande_greek: the ILSP dataset keeps sentence/option1/option2 in English and puts the Greek text only in multiple_choice_targets (two full candidate sentences). This task therefore does not reuse the English fill-the-blank template; it scores the two Greek sentences directly (a full-sentence plausibility formulation). Numbers are not directly comparable to the English winogrande task.

• mgsm_direct_greek: ilsp/mgsm_greek ships empty answer/equation_solution fields; the numeric gold is in answer_number. The question field has no prefix, so Question: is prepended. These overrides are required because the shared MGSM direct template assumes juletxara/mgsm's schema.

• truthfulqa_greek_mc1/mc2: ilsp/truthful_qa_greek (multiple_choice config) exposes only a train split (817 items), which is used for evaluation (validation_split: train).

• medical_mcqa_greek: uses the BIG-bench MCQA schema (inputs + multiple_choice_targets + multiple_choice_scores). This is the "Medical MCQA" task from ILSP's official Meltemi/Krikri model-card suite. Evaluates on the validation split; few-shot available from train.

• mcqa_greek_asep: Greek-native (not a translation): Greek civil-service / ASEP exam questions. Standard MCQA schema (question, choices, answer as integer index). Ships a single default split, so it runs 0-shot by default.

• mmlu_greek: ilsp/mmlu_greek preserves the exact MMLU schema and English config names, so the 57 per-subject tasks inherit the base MMLU template unchanged and only swap the dataset path. Mirrors the standard layout: 57 subjects → 4 category groups (_stem, _humanities, _social_sciences, _other) → mmlu_greek, with size-weighted acc and 5-shot from each subject's own dev split. Configs are (re)generated by mmlu/_generate_configs.py.

• Tag detachment. The 5 include-based tasks have tag: [] to detach the inherited English tag (ai2_arc, multiple_choice, mgsm_direct, truthfulqa) so they don't silently join the upstream English tag-groups.

License

The ILSP datasets are released under CC-BY-NC-SA-4.0 (non-commercial).

Test results

Verified with EleutherAI/pythia-160m (0-shot, limit 50):

Task	Metric	Value
arc_challenge_greek	acc / acc_norm	0.26 / 0.36
hellaswag_greek	acc / acc_norm	0.26 / 0.28
truthfulqa_greek_mc1	acc	0.24
truthfulqa_greek_mc2	acc	0.50
mgsm_direct_greek	exact_match (flexible-extract / remove_whitespace)	0.04 / 0.00
winogrande_greek	acc / acc_norm	0.44 / 0.44
mmlu_greek	acc (size-weighted, 2850 samples)	0.2365
mmlu_greek_stem	acc	0.2263
mmlu_greek_humanities	acc	0.2569
mmlu_greek_social_sciences	acc	0.2283
mmlu_greek_other	acc	0.2385
medical_mcqa_greek	acc / acc_norm	0.04 / 0.18
mcqa_greek_asep	acc / acc_norm	0.20 / 0.28

Validation performed

• pre-commit (ruff, check-yaml, trailing-whitespace, end-of-file-fixer, mixed-line-ending): clean
• test_task_manager.py: 67 passed
• test_group.py: 51 passed
• test_tasks.py: 848 passed (65 tasks x 13 checks: dataset download, split existence, doc_to_text/choice/target type-checking, request construction)
• Dummy model end-to-end over all 65 tasks: all produce finite metrics; logged samples verified Greek with correct gold indices
• scripts.write_out few-shot rendering at recommended shot counts: Greek contexts well-formed

[CLAassistant]

All committers have signed the CLA.