ToolCallSuccess: move runtime-status short-circuit from prompt into Python

[mmkawale]

Summary

Follow-up to #5126. Moves the runtime-status short-circuit out of the LLM rubric and into Python, so failed/incomplete tool calls return a deterministic fail result without spending a judge call. The prompty rubric is now consulted only on the success path.

Why

• The LLM was being asked to interpret a [STATUS] failed / [STATUS] incomplete annotation that we already know deterministically at evaluator time.
• That made every failure scorable, judge-model-version dependent, and a token spend.
• The status annotation is also surfaced to the model when the upstream payload is the source of truth, which is back-compat noise in the formatted wire input.

What changed

_tool_call_success.py (both repos):

• _do_eval now calls _collect_failed_tool_calls(eval_input["response"]) after preprocessing. If non-empty, returns _return_short_circuit_failure_result(failed_tools) early — no LLM call, no prompty render.
• New _collect_failed_tool_calls helper: two-pass scanner over messages, treats status in {"failed", "incomplete"} on either a tool_call block or its corresponding tool_result as a failure. Handles flat (name, tool_call_id) and nested (tool_call.function.name, tool_call.id) shapes. Falls back to id or name when one is missing, dedupes by (id, name).
• New _return_short_circuit_failure_result: returns score=0.0, result="fail", status="completed", properties={"failed_tools": ...}. Assets variant additionally includes top-level token-metadata keys for back-compat with the existing LLM path result shape.
• _get_tool_calls_results rewritten — no _format_status_suffix calls; output is byte-identical to the pre-pass-through format.
• _format_status_suffix removed.

tool_call_success.prompty (both repos):

• Removed the [STATUS] failed / [STATUS] incomplete bullet from the rubric.
• Removed step 3 paragraph describing [STATUS] as optional.
• Removed three [STATUS]-flavored examples ("status annotation overrides bland payload", "status completed but payload still indicates an error", "parallel calls in one turn, one annotated failed").

Tests:

• Deleted test_tool_call_success_status_passthrough.py.
• Added test_tool_call_success_status_shortcircuit.py: 19 tests covering both helpers — failed-on-call, failed-on-result, incomplete-treated-as-failure, dedupe, parallel calls, nested function shape, missing-id-falls-back-to-name, malformed-blocks-skipped-silently, non-list-input, no-status-forwarded-in-LLM-input, etc.

Validation

• pytest test_tool_call_success_status_shortcircuit.py — 19/19 passed.
• pytest test_tool_call_success_evaluator_behavior.py — 69/69 passed (no regression).

Related

• SDK side: Enable ToolCallAccuracy / ToolInputAccuracy on restricted-tool conver… azure-sdk-for-python#47462
• Upstream aca status plumbing: Vienna pullrequest/2139056

[github-actions]

Test Results for assets-test

88 tests   88 ✅  2s ⏱️
 1 suites   0 💤
 1 files     0 ❌

Results for commit 89430c0.

♻️ This comment has been updated with latest results.