Add open-r1/OpenR1-Math-220k dataset and nvidia/OpenMathReasoning to RL and fix reward function

[SurbhiJainUSC]

Description

Co-authored with @A9isha

This PR introduces significant improvements to the RL reward and evaluation logic for math-based datasets.

• addition of multiprocessing for math_verify to prevent trainer and evaluation hangs

• support for new datasets (OpenR1-Math-220k, OpenMathReasoning)

• answer extraction and normalization logic handles Multiple Choice Questions by allowing both option letters and values as correct

• Added more evaluation modes like majority voting and pass@1 estimation

• Sync weights to vllm prior to pre RL evaluation

• Added are_equal_under_sympy() for symbolic equality checking as a fallback before math_verify

Tests

Tested on OpenMathInstruct and OpenR1 dataset

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

❌ Patch coverage is 64.75410% with 129 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...maxtext/trainers/post_train/rl/math_verify_pool.py	60.31%	44 Missing and 6 partials ⚠️
src/maxtext/trainers/post_train/rl/utils_rl.py	78.03%	17 Missing and 12 partials ⚠️
src/maxtext/trainers/post_train/rl/evaluate_rl.py	56.14%	21 Missing and 4 partials ⚠️
src/maxtext/trainers/post_train/rl/train_rl.py	50.98%	18 Missing and 7 partials ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

🤖 Hi @SurbhiJainUSC, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

## 📋 Review Summary

This Pull Request introduces significant improvements to the Reinforcement Learning (RL) reward and evaluation logic, specifically for math-based datasets. Key highlights include the addition of multiprocessing for math_verify to prevent trainer hangs, support for new datasets (OpenR1-Math-220k, OpenMathReasoning), and more robust evaluation modes like majority voting and pass@1 estimation. The inclusion of comprehensive unit tests for the new grading logic and multiprocessing pool is a strong positive.

🔍 General Feedback

• Multiprocessing for Math Verification: The move to a process-isolated, timeout-bounded math_verify pool is a critical improvement for stability, especially when dealing with complex symbolic computations that might hang.
• Improved Answer Extraction: The new answer extraction and normalization logic is more robust and handles Multiple Choice Questions (MCQ) better by allowing both option letters and values as correct.
• Evaluation Modes: The addition of maj@K and pass@1 metrics provides a more complete picture of model performance beyond simple pass@K.
• LaTeX Normalization: The fix_latex_escaping utility addresses common issues with improperly escaped LaTeX strings, although it requires some tuning to avoid over-correcting common English words.

[NicoGrande]

Looks good overall! In general, can we add type hints to arguments / return types in the new functions we are adding. It makes it much easier to understand the intent of the function and reason about correctness!

[SurbhiJainUSC]

Looks good overall! In general, can we add type hints to arguments / return types in the new functions we are adding. It makes it much easier to understand the intent of the function and reason about correctness!

That's a great suggestion.

[NicoGrande]

LGTM