Skip to content

Commit 41c95f9

Browse files
alexluongclaude
andauthored
fix(agent-eval): bump LLM-judge default model off EOL claude-sonnet-4 (#965)
The LLM-judge default was pinned to claude-sonnet-4-20250514, which is deprecated and now returns a 404 not_found_error. CI doesn't set EVAL_SCORE_MODEL, so it fell back to the dead default and every scenario failed. Bump the default (and the .env.example / help-text references) to claude-sonnet-4-6. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent cc89cf0 commit 41c95f9

3 files changed

Lines changed: 3 additions & 3 deletions

File tree

docs/agent-evaluation/.env.example

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,4 @@ EVAL_TEST_DESTINATION_URL=
3636
# Scoring is ON by default after each scenario (heuristic + LLM). Opt out:
3737
# EVAL_NO_SCORE_HEURISTIC=1
3838
# EVAL_NO_SCORE_LLM=1
39-
# EVAL_SCORE_MODEL=claude-sonnet-4-20250514
39+
# EVAL_SCORE_MODEL=claude-sonnet-4-6

docs/agent-evaluation/src/llm-judge.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ import { basename, dirname, join } from "node:path";
88
import { extractTranscriptScoringText } from "./score-transcript.js";
99

1010
const ANTHROPIC_MESSAGES_URL = "https://api.anthropic.com/v1/messages";
11-
const DEFAULT_SCORE_MODEL = "claude-sonnet-4-20250514";
11+
const DEFAULT_SCORE_MODEL = "claude-sonnet-4-6";
1212
const MAX_TRANSCRIPT_CHARS = 180_000;
1313

1414
export interface LlmCriterionJudgment {

docs/agent-evaluation/src/score-eval.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ Score an eval transcript.
6060
npm run score -- --llm [--write] # Anthropic judge (needs ANTHROPIC_API_KEY)
6161
npm run score -- --llm --no-heuristic # LLM only (no regex heuristic)
6262
63-
Heuristic: src/score-transcript.ts. LLM: reads scenarios/*.md Success criteria + assistant text; model from EVAL_SCORE_MODEL (default claude-sonnet-4-20250514).
63+
Heuristic: src/score-transcript.ts. LLM: reads scenarios/*.md Success criteria + assistant text; model from EVAL_SCORE_MODEL (default claude-sonnet-4-6).
6464
6565
Options:
6666
--run <path> transcript.json, a run directory, or legacy flat *-scenario-NN.json

0 commit comments

Comments
 (0)