fix: stop submitting truncated reasoning as the browsecomp-plus answer

[shehabyasser-scale]

Stacked on pr3-harness-bench (#46). One file: the BrowseComp+ baseline agent.

The scoring bug

When a completion came back with finish_reason == "length", the agent submitted message.content verbatim as its final answer:

if not calls:
    if (message.content or "").strip():
        await self._submit(environment, message.content)

This target writes its chain-of-thought into ordinary content (it has no separate reasoning_content channel) and routinely runs past the token cap. So what got submitted was not an answer, it was 42k to 52k characters of truncated, mid-sentence reasoning. The judge scores that 0 every time, and the trial is indistinguishable from a genuine wrong answer.

Measured across the smoke runs carrying this change: 6 length-truncated turns over 3 trials, content length min 42,253 / median 46,793 / max 52,104 chars. The pre-fix baseline traces (198 trials over 3 rounds) contain no finish_reason field at all, which is exactly why this went unnoticed. The instrumentation added here is what surfaced it.

Truncated turns are now recoverable rather than final: the model is asked for only the three required labeled lines instead of having its unfinished thinking submitted on its behalf.

Supporting changes

All four target the same failure mode of losing a trial that had real retrieved evidence behind it.

Wall-clock deadline with a final-answer reserve. Turn count is not this agent's binding constraint. Most of a turn is sequential search.py invocations, each booting a fresh pyserini/Lucene JVM against the BM25 index (~13s apiece) against ~3-9s of inference. Trials ran out of clock long before MAX_TURNS, so the existing turn-budget fallback never fired and harbor killed the trial mid-search with no answer recorded. There is now a deadline (BROWSECOMP_AGENT_BUDGET_SEC, default 900) with a reserve (BROWSECOMP_FINAL_RESERVE_SEC, default 120) so the best-effort answer is written before harbor's AgentTimeoutError. Harbor does not tell an agent its own timeout, hence the env var; keep it in step with the run's --agent-timeout-multiplier.

Tool calls past the deadline are stubbed, never skipped, because every tool_call still needs a matching tool message or the forced-final request is rejected for an unanswered call.

Completion shape validation. A gateway can answer 200 with a body the SDK does not turn into a completion, observed once as 'str' object has no attribute 'usage', which crashed the trial on an attribute access two lines later. _complete() now rejects anything without usable .choices in one place.

Empty turns nudge instead of raising. raise RuntimeError("model returned neither a response nor a tool call") became a bounded retry, giving up only after MAX_EMPTY_RETRIES consecutive empty turns. An assistant message carrying neither content nor tool calls is dropped rather than appended, since strict OpenAI-compatible servers reject it.

Bounded forced-final. The last-resort call uses max_retries=1 (the SDK default of 8 could outlast the reserve on its own) and ANSWER_MAX_TOKENS (so the model cannot spend the whole reserve on chain-of-thought and get truncated a second time), and degrades to a stub answer on failure instead of propagating.

Extra trace fields. finish_reason, per-turn llm_sec, left_sec, and a forced_final_reason of deadline vs turns, which lets a post-hoc conservative score treat exactly the trials the pre-fix agent would have lost as zeros.

Rebased onto the current base

pr3-harness-bench moved (and its history was rewritten) after this branch was cut, so the original commit no longer applied. Rebuilt as one commit on 5727a44.

The _is_reasoning_model overlap flagged in the first version of this PR is resolved: #54's version of that gate has landed upstream and is kept verbatim here. All this PR now does to _completion_kwargs is route the already-computed token-limit key through a new max_tokens parameter, which the bounded forced-final call needs:

-        kwargs[_token_limit_key] = 12_000
+        kwargs[_token_limit_key] = max_tokens

Also fixed in the rebase

The first version broke the existing test suite. _complete reaches the client through with_options(...), and the test fake was a bare SimpleNamespace, so test_agent_submits_response_and_populates_context died on AttributeError: 'types.SimpleNamespace' object has no attribute 'with_options'. The fake is now a small FakeClient that implements it. Nothing in CI runs these target test suites (the repo has no workflows; the only checks are Greptile and Socket), which is why a red suite was not visible on the PR.

Three tests added, each verified to fail without its fix:

Test	Guards
test_truncated_reasoning_is_never_submitted_as_the_answer	Without the guard, the 40k-char ramble is submitted verbatim. This is the PR's headline claim, previously untested.
test_repeated_truncation_is_reported_as_empty_retries	Reports turns instead of empty_retries without the new flag.
FakeClient.with_options	The pre-existing test that the first version broke.

Both Greptile findings addressed:

1. Stubbed tool calls produced no trace events. The per-call _trace sat inside the executed branch, so deadline-stubbed calls were invisible in the JSONL exactly when a post-hoc audit needs them. Moved out so both paths are traced.
2. forced_final_reason conflated two causes. empty_turns >= MAX_EMPTY_RETRIES also reported "turns", so a trial that gave up on turn 4 after three truncated turns was indistinguishable from one that used all 32. There is now a third value, empty_retries, and the reason is computed once and used by both the trace and context.metadata.

Verification

• pytest tests/ in browsecomp-plus/baseline/target: 5 passed.
• pytest tests/test_v05_benchmark_configs.py: 13 passed, 6 skipped.
• Each new test confirmed red with its fix reverted and green with it restored.
• Exercised live in the smoke-browsecomp-plus-qwen fix1/fix2 runs cited above.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a scoring bug in the BrowseComp+ baseline agent where a finish_reason == "length" response (truncated mid-sentence reasoning, 42k–52k chars) was submitted verbatim as the final answer, which the judge scores 0 every time. It also adds wall-clock deadline management, _complete() shape validation, bounded empty-turn recovery, and richer trace instrumentation.

• Truncation fix (main loop): finish_reason == \"length\" turns are now treated as recoverable — the agent sends a nudge requesting only the three labeled answer lines, abandoning after MAX_EMPTY_RETRIES = 3 consecutive failures, with forced_final_reason distinguishing this from turn/deadline exhaustion.
• Deadline management: AGENT_BUDGET_SEC (env var, default 900s) with a FINAL_RESERVE_SEC (120s) reserve ensures the forced-final answer is written before harbor's AgentTimeoutError; tool calls past the reserve are stubbed rather than skipped so the message history stays valid for OpenAI-compatible servers.
• Robustness hardening: _complete() validates the gateway response shape in one place; the forced-final call is bounded to max_retries=1 and ANSWER_MAX_TOKENS=3_000 to prevent re-truncation.

Confidence Score: 4/5

Safe to merge; closes a well-documented scoring failure with one remaining gap in the forced-final path.

The forced-final answer extraction at lines 488-489 can still submit truncated reasoning if the model exceeds ANSWER_MAX_TOKENS, reproducing the exact bug the PR fixes in the main loop. Everything else in the change is logically sound and well-tested.

Files Needing Attention: harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py lines 488-489

Important Files Changed

Filename	Overview
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py	Major refactor adding wall-clock deadline management, truncation detection, _complete() shape validation, empty-turn retry logic, and bounded forced-final path — one gap where the forced-final answer can still be truncated reasoning if ANSWER_MAX_TOKENS is exceeded
harness-engineering-bench/browsecomp-plus/baseline/target/tests/test_agent.py	Adds FakeClient wrapper to support with_options, plus two regression tests for the truncation fix and forced_final_reason == 'empty_retries' — well-structured and directly exercises the new code paths

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([run starts]) --> B[set deadline = now + AGENT_BUDGET_SEC]
    B --> C{turn loop 1..MAX_TURNS}
    C --> D{remaining <= FINAL_RESERVE_SEC?}
    D -->|yes| E[out_of_time=True]
    E --> FF
    D -->|no| F[_complete timeout=remaining-reserve/2]
    F --> G{has tool_calls?}
    G -->|yes| H[empty_turns = 0]
    H --> I{deadline mid-turn?}
    I -->|yes| J[stub: search budget exhausted]
    I -->|no| K[execute tool]
    J --> L[append tool message]
    K --> L
    L --> M{submit_response called?}
    M -->|yes| N[completed=True break]
    M -->|no| O{out_of_time?}
    O -->|yes| FF
    O -->|no| C
    G -->|no| P{finish_reason == length?}
    P -->|yes| Q[truncated=True]
    P -->|no| R{content non-empty?}
    R -->|yes| S[_submit content completed=True break]
    R -->|no| Q2[truncated=False]
    Q --> T[empty_turns++]
    Q2 --> T
    T --> U{empty_turns >= MAX_EMPTY_RETRIES?}
    U -->|yes| V[out_of_answers=True break]
    U -->|no| W[append nudge continue]
    W --> C
    V --> FF
    FF([forced-final: if not completed])
    FF --> FF1[reason = deadline / empty_retries / turns]
    FF1 --> FF2[_complete tools=False max_tokens=ANSWER_MAX_TOKENS max_retries=1]
    FF2 --> FF3[submit answer or stub write forced_final_reason]

Loading

_{Reviews (2): Last reviewed commit: "fix: stop submitting truncated reasoning..." | Re-trigger Greptile}