react_gemini-2.5-flash baseline is ~60% empty answers

[rawwerks]

Hi — while analyzing the published ReAct baseline submissions I noticed the gemini-2.5-flash baseline is dominated by empty answers, and wanted to flag it in case it's unintended (or in case leaderboard readers should be aware).

Finding: in submissions/react_gemini-2.5-flash.json, 1,617 of 2,700 runs (59.9%) have an empty answer field (""). For comparison, across the other ReAct baselines:

┌────────────────────────┬─────────┬───────┬─────────┐
│ submission             │ records │ empty │ empty % │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gemini-2.5-flash │ 2700    │ 1617  │ 59.9%   │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_kimi-k2-thinking │ 2600    │ 220   │ 8.5%    │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gemini-3-pro     │ 2700    │ 64    │ 2.4%    │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gpt-5-mini       │ 2700    │ 25    │ 0.9%    │
├────────────────────────┼─────────┼───────┼─────────┤
│ react_gpt-5.2          │ 2700    │ 4     │ 0.1%    │
└────────────────────────┴─────────┴───────┴─────────┘

The Flash empties are spread across all 12 datasets (per-dataset empty rate ranges 31%–87%: agnews 87%, bookreview 80%, stockindex 77%, GITHUB_REPOS 76%, stockmarket 74%, PATENTS 73%, PANCANCER_ATLAS 71%, DEPS_DEV_V1 64%, yelp 53%, googlelocal 50%, crmarenapro 40%, music_brainz_20k 31%) and roughly uniform across the run index 0–49 (~32 ± 8 empties per run, range 24–40). That pattern looks systematic (pipeline-wide) rather than question-specific.

Why it may matter: the official validator returns is_valid: False for llm_answer == "", and accuracy.py counts that as 0 (failure) toward Pass@1, so this baseline's Pass@1 largely reflects non-response rather than incorrect answers — i.e., "the agent produced no final answer" is being counted the same as "the agent answered wrongly." From the submission alone I can't tell whether the empties are from rate-limits/timeouts or the ReAct harness not extracting a final answer from the smaller model (the submission records carry only dataset/query/run/answer — no error or termination reason).

Repro:

import json
d = json.load(open("submissions/react_gemini-2.5-flash.json"))
empty = [r for r in d if not (r.get("answer") or "").strip()]
print(f"{len(empty)}/{len(d)} = {100*len(empty)/len(d):.1f}% empty")

Questions:

1. Is the high empty rate for this baseline expected/known?
2. Should non-responses be retried, scored as a distinct state, or excluded from Pass@1 so they aren't conflated with wrong answers?
3. Minor: react_kimi-k2-thinking shows 8.5% empty (also across all datasets) — same underlying cause?

Happy to share the per-(dataset, run) breakdown if useful.