How to verify the string answer for MATH dataset?

[yuxiaooye]

Hi! Thanks for your awesome work!

I noticed that evaluate.py only has verify_float() for numerical answers. However, some ground truth in MATH dataset are strings like "p - q". Could you provide a verify function to handle these cases?

Thanks!

[RewindL]

I think the only way is to check whether p-q or $p$-$q$ exists in the output through re, or use a stronger LLM (might be slow and unstable). Hope there is a better way.

[yuxiaooye]

I think the only way is to check whether p-q or p -$q$ exists in the output through re, or use a stronger LLM (might be slow and unstable). Hope there is a better way.

@RewindL I found the evaluation script from original repo of MATH dataset: https://github.com/hendrycks/math/blob/357963a7f5501a6c1708cf3f3fb0cdf525642761/modeling/evaluate_gpt3.py#L106
which works well!

[RewindL]

I think the only way is to check whether p-q or p -$q$ exists in the output through re, or use a stronger LLM (might be slow and unstable). Hope there is a better way.

@RewindL I found the evaluation script from original repo of MATH dataset: https://github.com/hendrycks/math/blob/357963a7f5501a6c1708cf3f3fb0cdf525642761/modeling/evaluate_gpt3.py#L106 which works well!

Wow, it is terrific. Thanks for sharing. I am recently verifying results on math too.