fix: only send reasoning.effort to models that support it

Each harness-engineering-bench target agent builds its Responses API request
with a hard-coded "reasoning": {"effort": ...} on every call, regardless of the
configured model. Reasoning models (gpt-5.x, o-series, *-codex) accept it, but
gpt-4o and other non-reasoning models reject the whole request with HTTP 400:
"Unsupported parameter: 'reasoning.effort' is not supported with this model".

On a swe-atlas-qna run pointed at a gpt-4o agent model this produced a wall of
400s (60 cases died with BadRequestError). The gaia and tau3 configs point their
agent model at gpt-4o in the same setup, so they share the latent bug.

Gate reasoning.effort on the model family: send it only when the resolved model
is a reasoning model (gpt-5.x / o1 / o3 / o4 / *-codex), otherwise omit it. Each
agent keeps its own effort level (high for atlas and swe-bench-pro, medium for
gaia, officeqa and tau3). No behavior change for reasoning models; gpt-4o stops
400ing. Applied uniformly to all five bench agents.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: shehabyasser-scale

harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py

@@ -185,10 +185,14 @@ async def run(
                 "instructions": INSTRUCTIONS,
                 "input": next_input,
                 "tools": TOOLS,
-                "reasoning": {"effort": "medium"},
                 "max_output_tokens": 8000,
                 "parallel_tool_calls": False,
             }
+            # gpt-4o and other non-reasoning models reject `reasoning.effort`
+            # with HTTP 400; only send it to reasoning-capable models.
+            _model = self._api_model.lower()
+            if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
+                request["reasoning"] = {"effort": "medium"}
             if previous_response_id is not None:
                 request["previous_response_id"] = previous_response_id
             response = await self._client.responses.create(**request)

harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

@@ -189,10 +189,14 @@ async def run(
                 "instructions": INSTRUCTIONS,
                 "input": next_input,
                 "tools": TOOLS,
-                "reasoning": {"effort": "medium"},
                 "max_output_tokens": 8000,
                 "parallel_tool_calls": False,
             }
+            # gpt-4o and other non-reasoning models reject `reasoning.effort`
+            # with HTTP 400; only send it to reasoning-capable models.
+            _model = self._api_model.lower()
+            if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
+                request["reasoning"] = {"effort": "medium"}
             if previous_response_id is not None:
                 request["previous_response_id"] = previous_response_id
             response = await self._client.responses.create(**request)

harness-engineering-bench/swe-atlas-qna/baseline/target/src/atlas_agent/agent.py

@@ -150,10 +150,14 @@ async def run(
                 "instructions": INSTRUCTIONS,
                 "input": next_input,
                 "tools": TOOLS,
-                "reasoning": {"effort": "high"},
                 "max_output_tokens": 12_000,
                 "parallel_tool_calls": False,
             }
+            # gpt-4o and other non-reasoning models reject `reasoning.effort`
+            # with HTTP 400; only send it to reasoning-capable models.
+            _model = self._api_model.lower()
+            if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
+                request["reasoning"] = {"effort": "high"}
             if previous_response_id is not None:
                 request["previous_response_id"] = previous_response_id
             response = await self._client.responses.create(**request)

harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py

@@ -382,10 +382,14 @@ async def run(
                 "instructions": INSTRUCTIONS,
                 "input": next_input,
                 "tools": TOOLS,
-                "reasoning": {"effort": "high"},
                 "max_output_tokens": 12_000,
                 "parallel_tool_calls": False,
             }
+            # gpt-4o and other non-reasoning models reject `reasoning.effort`
+            # with HTTP 400; only send it to reasoning-capable models.
+            _model = self._api_model.lower()
+            if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
+                request["reasoning"] = {"effort": "high"}
             if previous_response_id is not None:
                 request["previous_response_id"] = previous_response_id
             response = await self._responses_create(**request)

harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

@@ -298,10 +298,14 @@ async def run(
                 "instructions": INSTRUCTIONS,
                 "input": next_input,
                 "tools": openai_tools,
-                "reasoning": {"effort": "medium"},
                 "max_output_tokens": 8_000,
                 "parallel_tool_calls": False,
             }
+            # gpt-4o and other non-reasoning models reject `reasoning.effort`
+            # with HTTP 400; only send it to reasoning-capable models.
+            _model = self._api_model.lower()
+            if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
+                request["reasoning"] = {"effort": "medium"}
             if previous_response_id is not None:
                 request["previous_response_id"] = previous_response_id
             response = await client.responses.create(**request)