This might be a misunderstanding on my part, please correct me if I'm wrong.
I'm seeing here in the standard GSM8K dataset (http://platinum-bench.csail.mit.edu/inspect?model=gemini-2.5-pro-exp-03-25&dataset=gsm8k) that many of the LLMs are getting this problem wrong:
It looks to me like the problem's solution is incorrect. 80 flagstones would be 80 * 75 = 6,000 pounds, which would require 3 trucks (each carries 2,000 pounds).
However, in GSM8K-Platinum, I also see this problem come up with the same listed solution:
That being said, some of the models (like Gemini 2.5 Pro) are not listed (presumably due to having 0 errors?)
Thanks for your time.
This might be a misunderstanding on my part, please correct me if I'm wrong.
I'm seeing here in the standard GSM8K dataset (http://platinum-bench.csail.mit.edu/inspect?model=gemini-2.5-pro-exp-03-25&dataset=gsm8k) that many of the LLMs are getting this problem wrong:
It looks to me like the problem's solution is incorrect. 80 flagstones would be 80 * 75 = 6,000 pounds, which would require 3 trucks (each carries 2,000 pounds).
However, in
GSM8K-Platinum, I also see this problem come up with the same listed solution:That being said, some of the models (like Gemini 2.5 Pro) are not listed (presumably due to having 0 errors?)
Thanks for your time.