Claude models are forced to --parallel 1 (SERIAL_ONLY_MODEL_PREFIXES in chunked_eval.py) because the last-resort wall-clock timeout in eval_no_tools._run_request_with_wall_timeout uses SIGALRM, which Python only delivers on the main thread. In worker threads the guard silently disappears, and a provider call that outlives its client timeout wedges the run.
Each chunk already executes as a subprocess (chunked_eval.run_chunk → subprocess.run). Moving the hard bound there makes parallelism safe without signals:
- pass
timeout= to subprocess.run, sized as rows_in_chunk × wall_timeout + slack
- treat
TimeoutExpired as an infrastructure failure: kill the chunk, leave it incomplete, let resume retry it
- drop the SIGALRM path (or keep it main-thread-only as a secondary bound)
- lift the serial restriction so
claude-* can run --parallel 4+
A full Claude model run currently takes ~2–3h serial at chunk size 5; at --parallel 4 it's ~30–40 min. Ran into this again adding Sonnet 5 (2026-07-02).
🤖 Generated with Claude Code
Claude models are forced to
--parallel 1(SERIAL_ONLY_MODEL_PREFIXESin chunked_eval.py) because the last-resort wall-clock timeout ineval_no_tools._run_request_with_wall_timeoutuses SIGALRM, which Python only delivers on the main thread. In worker threads the guard silently disappears, and a provider call that outlives its client timeout wedges the run.Each chunk already executes as a subprocess (
chunked_eval.run_chunk→subprocess.run). Moving the hard bound there makes parallelism safe without signals:timeout=tosubprocess.run, sized asrows_in_chunk × wall_timeout + slackTimeoutExpiredas an infrastructure failure: kill the chunk, leave it incomplete, let resume retry itclaude-*can run--parallel 4+A full Claude model run currently takes ~2–3h serial at chunk size 5; at
--parallel 4it's ~30–40 min. Ran into this again adding Sonnet 5 (2026-07-02).🤖 Generated with Claude Code