Act on the PR review: benchmark and baseline-agent fixes

Six findings from the reviews on #45/#46 that survived checking against
the current tree. Each was verified against the live litellm endpoint the
gateway proxies to, not inferred from a commit message.

gaia moves to gpt-5.4-mini. It is the one multimodal benchmark and
deepseek-v4-flash is text-only: 5 of the 66 held-out tasks send images and
get 400 "This model does not support image inputs", which caps achievable
reward near 0.92 and reads as ordinary agent failure. gpt-5.4-mini returns
200 for every request shape the gaia agent sends -- image input, hosted
web_search, reasoning.effort, parallel_tool_calls -- confirmed through the
SDK path the agent actually uses. The name is unprefixed on purpose: every
agent sends model_name.removeprefix("openai/"), so an openai/-prefixed
name would be allow-listed in one form and requested in another and the
gateway would deny it. The gaia held-out baseline now needs re-measuring.

tau3 executes every tool call the model returns instead of only the first.
deepseek-v4-flash does return two in a turn, and the usual guard is not
available: litellm rejects parallel_tool_calls for fireworks_ai with
UnsupportedParamsError, so the reviewer's suggested one-liner would have
400'd every tau3 call. Dropping calls[1:] silently skipped actions -- for a
customer-service agent, a verified identity with no message sent.

officeqa and swe-atlas-qna run their tool dispatch inside the try rather
than an else, and catch KeyError/TypeError/ValueError alongside
JSONDecodeError, matching browsecomp-plus. Neither tool schema is strict,
so the model can return valid JSON missing a required key; that ended the
trial instead of feeding the model an error. Widening the except alone
would have caught nothing, since the lookups were in the else.

gaia's client gets max_retries=8, the only one of the five without it. A
within-trial transient failure scores at the failure value for competitive
evaluations, so an unretried 429 cost a candidate a 0.0.

tau3 quotes the MCP session id before interpolating it into a shell
command, for the same reason the payload beside it is base64-encoded.

per_trial_tokens buckets unattributed records under a name rather than
None, which crashed sorted() as soon as one appeared beside an attributed
record.

Not actioned: the browsecomp-plus grader was reported as vulnerable to
format-string injection from curly braces in the agent's response.
str.format does not rescan substituted values, so there is nothing to fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: varunursekar

harness-engineering-bench/CONFIGURATION.md

@@ -38,7 +38,11 @@ benchmark can be checked against the others at a glance.
   grounded-reasoning benchmarks (officeqa/browsecomp 0.60) at roughly gpt-oss
   cost — far cheaper than mini. `swe-atlas-qna` is pinned to
   `fireworks_ai/gpt-oss-120b`, the one benchmark where deepseek is weaker
-  (0.30 vs 0.59 mean rubric) and gpt-oss is both cheaper and stronger.
+  (0.30 vs 0.59 mean rubric) and gpt-oss is both cheaper and stronger. `gaia` is
+  pinned to `gpt-5.4-mini` for a different reason: it is the one multimodal
+  benchmark, 5 of its 66 held-out tasks send image inputs, and deepseek-v4-flash
+  rejects those outright (`This model does not support image inputs`), capping
+  achievable reward near 0.92 and disguising the shortfall as agent failure.
 - **Execution**: `harbor[modal]==0.20.0`, python 3.12, `n_attempts: 1`,
   `max_retries: 1`, 3 infrastructure attempts at 5s, `aggregate_attempts:
   best`, `max_concurrency: 8`, `error_rate_threshold: 0.1`,
@@ -53,7 +57,7 @@ benchmark can be checked against the others at a glance.
 
 | | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
 |---|---|---|---|---|---|
-| target model | deepseek-v4-flash | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
+| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
 | held-out baseline (K=3) ◆ | — | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 |
 | split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
 | dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
@@ -95,6 +99,16 @@ deepseek-v4-flash (gpt-oss-120b on swe-atlas); the three deepseek benchmarks
 logged zero exceptions over 945 trials, swe-atlas lost 5/150 to gpt-oss 128k
 context overflow.
 
+◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and
+that model is text-only. Verified against the same litellm endpoint the gateway
+proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent
+sends (image input, hosted `web_search`, `reasoning.effort`,
+`parallel_tool_calls`), while deepseek-v4-flash returns
+`400 This model does not support image inputs`. Written unprefixed on purpose:
+each agent sends `model_name.removeprefix("openai/")`, so an `openai/`-prefixed
+name would be allow-listed in one form and requested in another and the gateway
+would deny it.
+
 † Sized from stock-agent probes (codex on the target model, 3 development
 tasks each, full declared timeouts): tau3 trials took 202-211s (900s budget
 ≈ 4x headroom) and swe-atlas trials 379-602s (1800s ≈ 3x headroom over the

harness-engineering-bench/gaia/baseline/build.yaml

@@ -45,7 +45,13 @@ score_baseline: true
 rescore_top_k: 3
 rescore_attempts: 1
 
-model: fireworks_ai/deepseek-v4-flash
+# GAIA is multimodal: 5 of the 66 held-out tasks send image inputs, and a
+# text-only target 400s on them ("This model does not support image inputs"),
+# which reads as ordinary agent failure and caps achievable reward near 0.92.
+# Unprefixed on purpose -- the agent sends model_name.removeprefix("openai/"),
+# so an openai/-prefixed name here would be allow-listed in one form and
+# requested in another, and the gateway would deny it.
+model: gpt-5.4-mini
 environment_name: ${inner_env:-modal}
 # inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default
 extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
@@ -85,7 +91,7 @@ inference_gateway:
     allowed_models: ["${optimizer_model:-gpt-5.4}"]
     max_concurrency: 8
   evaluation:
-    allowed_models: [fireworks_ai/deepseek-v4-flash]
+    allowed_models: [gpt-5.4-mini]
     max_requests: 15000
     max_tokens: 100000000
     max_concurrency: 64

harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py

@@ -102,7 +102,10 @@ def __init__(self, *args: Any, **kwargs: Any) -> None:
         if self.model_name is None:
             raise ValueError("GAIA agent requires a Harbor model")
         self._api_model = self.model_name.removeprefix("openai/")
-        self._client = AsyncOpenAI()
+        # Absorb transient 429s in-client: a within-trial infra failure scores
+        # at the failure value for competitive evaluations, so an unretried
+        # rate limit costs a candidate a 0.0.
+        self._client = AsyncOpenAI(max_retries=8)
 
     @override
     async def setup(self, environment: BaseEnvironment) -> None:

harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

@@ -253,15 +253,16 @@ async def run(
 
             submitted = False
             for call in calls:
+                image_url = None
+                # The dispatch runs inside the try rather than an else: the
+                # argument lookups are the likelier failure, since the tool
+                # schemas are not strict and the model can return valid JSON
+                # that omits a required key. Feed that back as a tool error
+                # instead of letting a KeyError end the whole trial.
                 try:
                     arguments = json.loads(call.function.arguments)
-                except json.JSONDecodeError as error:
-                    result: dict[str, Any] = {"error": f"invalid arguments: {error}"}
-                    image_url = None
-                else:
-                    image_url = None
                     if call.function.name == "run_shell":
-                        result = await self._run_shell(
+                        result: dict[str, Any] = await self._run_shell(
                             environment, arguments["command"]
                         )
                     elif call.function.name == "read_image":
@@ -274,6 +275,9 @@ async def run(
                         submitted = True
                     else:
                         result = {"error": f"unknown tool: {call.function.name}"}
+                except (json.JSONDecodeError, KeyError, TypeError, ValueError) as error:
+                    result = {"error": f"invalid arguments: {error}"}
+                    image_url = None
                 self._trace(
                     {"turn": turn, "tool": call.function.name, "result": result}
                 )

harness-engineering-bench/scripts/per_trial_tokens.py

@@ -123,7 +123,9 @@ def load_threads(requests_dir: Path) -> dict[str, dict[str, dict]]:
                     continue
                 if record.get("scope") not in ("evaluation", "finalization"):
                     continue
-                evaluation_id = record.get("attribution")
+                # Salvage copies can lack attribution; a None key would sort
+                # against the real ones and crash the report.
+                evaluation_id = record.get("attribution") or "unattributed"
                 snippet = record.get("root_snippet") or recover_snippet(record)
                 thread_key = record.get("thread_id") or (
                     "snippet:" + normalize(snippet or "")[:160] or "unattributed"

harness-engineering-bench/swe-atlas-qna/baseline/target/src/atlas_agent/agent.py

@@ -234,13 +234,15 @@ async def run(
 
             submitted = False
             for call in calls:
+                # The dispatch runs inside the try rather than an else: the
+                # argument lookups are the likelier failure, since the tool
+                # schemas are not strict and the model can return valid JSON
+                # that omits a required key. Feed that back as a tool error
+                # instead of letting a KeyError end the whole trial.
                 try:
                     arguments = json.loads(call.function.arguments)
-                except json.JSONDecodeError as error:
-                    result: dict[str, Any] = {"error": f"invalid arguments: {error}"}
-                else:
                     if call.function.name == "run_shell":
-                        result = await self._run_shell(
+                        result: dict[str, Any] = await self._run_shell(
                             environment, arguments["command"]
                         )
                     elif call.function.name == "submit_answer":
@@ -249,6 +251,8 @@ async def run(
                         submitted = True
                     else:
                         result = {"error": f"unknown tool: {call.function.name}"}
+                except (json.JSONDecodeError, KeyError, TypeError, ValueError) as error:
+                    result = {"error": f"invalid arguments: {error}"}
                 self._trace({"turn": turn, "tool": call.function.name, "result": result})
                 messages.append(
                     {

harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

@@ -20,6 +20,7 @@
 import json
 import os
 import re
+import shlex
 from typing import Any, override
 
 from harbor.agents.base import BaseAgent
@@ -130,8 +131,13 @@ async def _mcp_raw(
         encoded = base64.b64encode(
             json.dumps(payload, ensure_ascii=False).encode("utf-8")
         ).decode("ascii")
+        # Quoted for the same reason the payload above is base64-encoded: the
+        # session id comes back from the server, and a quote in it would break
+        # out of the surrounding single quotes into the shell.
         session_header = (
-            f"-H 'mcp-session-id: {session_id}' " if session_id is not None else ""
+            f"-H {shlex.quote(f'mcp-session-id: {session_id}')} "
+            if session_id is not None
+            else ""
         )
         command = (
             f"printf '%s' '{encoded}' | base64 -d | "
@@ -341,10 +347,15 @@ async def run(
             )
 
             if calls:
-                # This agent acts one tool at a time; record only the first call
-                # so the assistant turn and its single tool reply stay balanced
-                # (Chat Completions requires a tool message per tool_call id).
-                call = calls[0]
+                # Every call the model made is executed, in order. The target
+                # model does return more than one per turn, and the usual guard
+                # -- parallel_tool_calls: False -- is not available here:
+                # litellm rejects it for fireworks_ai with UnsupportedParamsError.
+                # Acting on only the first would silently skip the rest, which
+                # for a customer-service agent means a verified identity with no
+                # message sent. Chat Completions requires exactly one tool
+                # message per tool_call id, so the assistant turn lists them all
+                # and each gets its reply below.
                 assistant: dict[str, Any] = {
                     "role": "assistant",
                     "tool_calls": [
@@ -356,33 +367,45 @@ async def run(
                                 "arguments": call.function.arguments,
                             },
                         }
+                        for call in calls
                     ],
                 }
                 if message.content:
                     assistant["content"] = message.content
                 messages.append(assistant)
-                try:
-                    arguments = json.loads(call.function.arguments)
-                    result_text = await self._call_tool(
-                        environment, session_id, call.function.name, arguments
+                stop = False
+                for call in calls:
+                    try:
+                        arguments = json.loads(call.function.arguments)
+                        result_text = await self._call_tool(
+                            environment, session_id, call.function.name, arguments
+                        )
+                    except Exception as error:  # noqa: BLE001 - feed failures to model
+                        result_text = json.dumps(
+                            {"error": f"{type(error).__name__}: {error}"}
+                        )
+                    self._trace(
+                        {
+                            "turn": turn,
+                            "tool": call.function.name,
+                            "result": result_text,
+                        }
                     )
-                except Exception as error:  # noqa: BLE001 - feed failures to model
-                    result_text = json.dumps(
-                        {"error": f"{type(error).__name__}: {error}"}
+                    messages.append(
+                        {
+                            "role": "tool",
+                            "tool_call_id": call.id,
+                            "content": result_text,
+                        }
                     )
-                self._trace(
-                    {"turn": turn, "tool": call.function.name, "result": result_text}
-                )
-                messages.append(
-                    {
-                        "role": "tool",
-                        "tool_call_id": call.id,
-                        "content": result_text,
-                    }
-                )
-                if call.function.name == "end_conversation" or _looks_stopped(
-                    result_text
-                ):
+                    # Note the stop but keep going: leaving a tool_call id
+                    # without a reply would make the next request invalid, and
+                    # the remaining calls were ones the model actually asked for.
+                    if call.function.name == "end_conversation" or _looks_stopped(
+                        result_text
+                    ):
+                        stop = True
+                if stop:
                     break
                 continue