fix(v1): cancel the sibling scorer when one scoring handler fails

[dumko2001]

Description

When one scorer failed, the other kept running and could keep writing to a trace that was already being torn down.

Rollout scores a trace by awaiting the task scorer and the harness scorer together with a bare asyncio.gather. gather propagates the first exception right away, but it does not cancel the sibling. That sibling keeps mutating the trace, calling judges, and using a runtime the failure path is already shutting down, which can corrupt the trace or raise confusing follow-on errors.

The fix routes both scorers through a gather_scoring helper. It runs them as explicit tasks, shields them while active, and on any failure cancels the unfinished siblings and drains them before propagating. The original failure wins over errors raised during cleanup, and a caller cancellation that arrives during cleanup stays the final outcome. Result ordering is unchanged.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass when running uv run pytest locally.
• New tests have been added to cover the changes

Verified with a local script covering single failure, timeout, simultaneous failures, repeated cancellation, cleanup raising, empty and singleton call sets, preserved ordering, and no leaked tasks. uv run pytest tests/v1/ — 63 passed, 144 skipped (require PRIME_API_KEY). uv run ruff check . and ruff format --check . clean.

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

Open #2067 edits this exact scoring block and #1939 replaces Rollout; both keep raw gather. The helper is independent and its call site rebases cleanly onto either.

Note

Cancel sibling scorer when one scoring handler fails in v1 rollout

• Introduces gather_scoring in rollout.py, an async helper that runs multiple scoring awaitables concurrently, each wrapped with asyncio.shield to prevent premature cancellation by an outer timeout.
• On any exception, gather_scoring cancels all remaining sibling tasks and drains them via asyncio.gather(..., return_exceptions=True) before re-raising the original error.
• Replaces direct asyncio.gather in the scoring phase with gather_scoring so a failure in one scorer reliably cancels and drains the other.
• Behavioral Change: previously, a failure in one scorer would not cancel its sibling; now the sibling is explicitly cancelled and drained before the error surfaces.

^{Macroscope summarized 3e9bb05.}

---

Note

Medium Risk
Touches concurrent scoring on every rollout teardown path; behavior change is localized to failure/cancellation cleanup but errors in the new helper could affect how scoring failures propagate.

Overview
Fixes a concurrency bug in Rollout’s scoring phase: when one of the two concurrent scorers (task.score and harness.score) failed, a plain asyncio.gather returned immediately while the other kept running and could keep writing to the trace or using a runtime already being torn down.

The PR introduces gather_scoring in rollout.py, which runs scoring awaitables as explicit tasks (with per-task asyncio.shield while gathering), and on any failure cancels unfinished siblings and drains them via run_shielded before re-raising the original error. The scoring wait_for block now calls gather_scoring(...) instead of asyncio.gather(...); result ordering and the outer scoring timeout behavior stay the same.

^{Reviewed by Cursor Bugbot for commit 3e9bb05. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7bb5d4f. Configure here.}