Skip to content

Add PolyMath #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
haideraltahan merged 8 commits into from
Jul 8, 2026
Merged

Add PolyMath #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2a65437
feat: add PolyMath multilingual math eval (EU languages) for lm-eval
Jun 24, 2026
1d835e6
refactor: split polymath-eu into one task group per difficulty tier
Jun 24, 2026
6e81468
fix: use --batch_size auto in eval sbatch
Jun 25, 2026
83b3cb0
fix(polymath): score with the official Qwen/PolyMath math_equal judge
Jun 25, 2026
be0e568
fix: set writable MIOpen DB path in eval sbatch (ROCm/MI250X)
Jun 25, 2026
b4889cb
style: keep polymath task-group descriptions under 80 chars, sort uti…
Jul 1, 2026
dd8620e
Merge branch 'main' into PolyMath
swag2198 Jul 7, 2026
2c193cd
Merge branch 'main' into PolyMath
haideraltahan Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Available task groups:
- `flores-200-eu-to-eng` / `flores-200-eng-to-eu` - Translation tasks
- `global-mmlu-eu` - Global MMLU in EU languages
- `mgsm-eu` - Multilingual GSM benchmarks
- `polymath-eu-low` / `polymath-eu-medium` / `polymath-eu-high` / `polymath-eu-top` - PolyMath multilingual math reasoning (EU languages, 0-shot, `\boxed{}` answer extraction), one group per difficulty tier
- `generic-multilingual` - XWinograd, XCOPA, XStoryCloze
- `include` - INCLUDE benchmarks

Expand Down
21 changes: 21 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/_default_template_yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
dataset_path: Qwen/PolyMath
output_type: generate_until
training_split: null
validation_split: null
doc_to_text: !function utils.doc_to_text
doc_to_target: !function utils.doc_to_target
process_results: !function utils.process_results
generation_kwargs:
until:
- "Problem:"
max_gen_toks: 1024
do_sample: false
temperature: 0
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
metadata:
version: 1.0
dataset_kwargs:
trust_remote_code: true
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_de_high.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_de_high
dataset_name: de
test_split: high
tag:
- polymath
- polymath_de
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_de_low.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_de_low
dataset_name: de
test_split: low
tag:
- polymath
- polymath_de
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_de_medium.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_de_medium
dataset_name: de
test_split: medium
tag:
- polymath
- polymath_de
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_de_top.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_de_top
dataset_name: de
test_split: top
tag:
- polymath
- polymath_de
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_en_high.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_en_high
dataset_name: en
test_split: high
tag:
- polymath
- polymath_en
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_en_low.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_en_low
dataset_name: en
test_split: low
tag:
- polymath
- polymath_en
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_en_medium.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_en_medium
dataset_name: en
test_split: medium
tag:
- polymath
- polymath_en
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_en_top.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_en_top
dataset_name: en
test_split: top
tag:
- polymath
- polymath_en
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_es_high.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_es_high
dataset_name: es
test_split: high
tag:
- polymath
- polymath_es
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_es_low.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_es_low
dataset_name: es
test_split: low
tag:
- polymath
- polymath_es
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_es_medium.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_es_medium
dataset_name: es
test_split: medium
tag:
- polymath
- polymath_es
7 changes: 7 additions & 0 deletions oellm/resources/custom_lm_eval_tasks/polymath/polymath_es_top.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include: _default_template_yaml
task: polymath_es_top
dataset_name: es
test_split: top
tag:
- polymath
- polymath_es
Loading
Loading