[NA] [SDK] fix: recover agentic LLM judge verdicts from prose-wrapped output

[alexkuzmik]

Details

The agentic LLM judge's final turn returns free-form content. OpenAI honors response_format strictly, but Anthropic (native and via LiteLLM) treats it as best-effort when tools are in the request — on a false verdict claude-haiku-4-5 narrates in prose and wraps the JSON in a ```json fence, sometimes preceded by a second JSON object (a quoted copy of the agent output). The parser's bare json.loads failed at char 0, so scoring_failed=True, and temperature-0 retries reproduced the same prose — failing the agentic evaluation-metrics tests deterministically (test_false_assertion__fails[anthropic] and [litellm_anthropic]).

• parsers.py: scan every embedded JSON object and return the first that validates against the schema — recovers a prose/fenced verdict and skips the quoted agent-output block. The bare-JSON fast path (OpenAI / strict structured output) is unchanged.
• Agentic judge integration tests now run the Anthropic lanes against claude-sonnet-4-6 instead of claude-haiku-4-5; Sonnet follows the structured-output contract reliably (the one-shot suite already used Sonnet).
• Added unit tests for prose+fence payloads and multi-object candidate selection.

Change checklist

• User facing
• Documentation update

Issues

• N/A — CI test-stability fix for the agentic evaluation-metrics suite; no tracking ticket.

AI-WATERMARK

AI-WATERMARK: yes

• Tools: Claude Code
• Model(s): Claude Opus 4.8 (1M context)
• Scope: full implementation (analysis, fix, tests)
• Human verification: code review + local test run

Testing

Commands run from sdks/python (project venv):

• python -m pytest tests/unit/evaluation/suite_evaluators/test_llm_judge_parsers.py → 23 passed (20 existing + 3 new)
• python -m pytest tests/unit/evaluation/suite_evaluators/ → 213 passed, 2 skipped (no regressions)
• ruff check, ruff format --check, and mypy on the changed files → clean
• Replayed the exact CI raw output (prose + ```json fence + a quoted agent-output object) through ResponseSchema.parse → verdict recovered (scoring_failed=False, value=False).

Not run locally: the two failing tests are library-integration tests that call real provider models (require ANTHROPIC_API_KEY); they are validated by the evaluation_metrics_tests CI lane. Both fixes (tolerant parser + Sonnet) independently target that exact failure.

Documentation

N/A — no user-facing or API changes.

[github-actions]

⏱️ pre-commit per-hook timing

Hook	Description	Result	Duration
🐍 mypy — python sdk	Static type check	✅	1.13s
🐍 fix end of files — python sdk	Ensure files end in a newline	✅	0.03s
🐍 trim trailing whitespace — python sdk	Strip trailing whitespace	✅	0.03s
🐍 ruff-format — python sdk	Format Python code (ruff)	✅	0.01s
🐍 ruff — python sdk	Lint + autofix Python (ruff)	✅	0.01s
Total (5 ran)			1.21s

⏭️ 36 skipped (no matching files changed)

Hook	Description	Result
🤖 trim trailing whitespace — optimizer	Strip trailing whitespace	⏭️
🤖 fix end of files — optimizer	Ensure files end in a newline	⏭️
🤖 check yaml — optimizer	Validate YAML syntax	⏭️
🤖 check json — optimizer	Validate JSON syntax	⏭️
🤖 check toml — optimizer	Validate TOML syntax	⏭️
🤖 check for added large files — optimizer	Block large files (>1MB)	⏭️
🔐 detect private key — optimizer	Block committed private keys	⏭️
🤖 check for merge conflicts — optimizer	Block merge-conflict markers	⏭️
🤖 check for case conflicts — optimizer	Block case-only name clashes	⏭️
🤖 pyupgrade — optimizer	Modernize Python syntax	⏭️
🤖 ruff — optimizer	Lint + autofix Python (ruff)	⏭️
🤖 ruff-format — optimizer	Format Python code (ruff)	⏭️
🤖 mypy — optimizer	Static type check	⏭️
📓 nbstripout — optimizer notebooks	Strip notebook output	⏭️
📝 markdownlint — optimizer	Lint Markdown	⏭️
🔤 codespell — optimizer	Fix common misspellings	⏭️
📊 radon cc — optimizer	Cyclomatic-complexity gate	⏭️
📊 radon raw — optimizer	Raw size metrics gate	⏭️
📊 xenon — optimizer	Fail on complexity thresholds	⏭️
📊 lizard — optimizer	Cyclomatic-complexity gate	⏭️
🧹 vulture — optimizer	Find dead code	⏭️
🛡️ trim trailing whitespace — guardrails	Strip trailing whitespace	⏭️
🛡️ fix end of files — guardrails	Ensure files end in a newline	⏭️
🛡️ ruff — guardrails	Lint + autofix Python (ruff)	⏭️
🛡️ ruff-format — guardrails	Format Python code (ruff)	⏭️
🛡️ mypy — guardrails	Static type check	⏭️
⚓ helm-docs	Regenerate Helm chart README	⏭️
block non-public FE plugins	Block non-public FE plugins	⏭️
☕ spotless — java backend	Format Java code	⏭️
🧪 pre-commit wrapper smoke tests	Self-test the wrapper scripts	⏭️
🌐 eslint — frontend	Lint + autofix JS/TS	⏭️
🌐 typecheck — frontend	Whole-project tsc type check	⏭️
📘 eslint — typescript sdk	Lint + autofix JS/TS	⏭️
📘 typecheck — typescript sdk	Whole-project tsc type check	⏭️
⚙️ actionlint — github workflows	Lint GitHub Actions workflows	⏭️
🐳 hadolint — dockerfiles	Lint Dockerfiles	⏭️

[baz-reviewer]

Duplicate JSON recovery logic

Lines 36-40 duplicate the brace-scanning JSONDecoder.raw_decode fallback already in sdks/python/src/opik/evaluation/metrics/llm_judges/parsing_helpers.py, so any tweak to prose/fenced-output recovery has to be made in two places — should we share that helper or move the loop into a common module?

Want Baz to fix this for you? Activate Fixer

[alexkuzmik]

Superseded by #7405, which consolidates the Sonnet model switch (dropping the parser-leniency change per review) with the genai/adk matrix trim and removal of the broken genai rate-limit retry code.