Skip to content

Commit 387ee89

Browse files
evekhmevekhm
committed
Add correction analysis with trajectory segmentation, routing failure detection, and parroting detection
- Embed segmented execution trajectories in Correction Analysis section (before/after correction with outcome labels: recovered, parroted, not_recovered) - Detect routing failures (agent answered from LLM knowledge without tools) and surface as Routing Failures subsection - Fix answered_by defaulting to "policy_agent" — now "unknown" with BQ backfill - Tighten correction inference prompt to distinguish genuine recovery from parroting - Add --eval-config flag for external metric/prompt definitions (eval_config.json) - Nest Correction Analysis under Sample Sessions with dynamic heading levels - Update sample report and README
1 parent a2fd41e commit 387ee89

6 files changed

Lines changed: 1396 additions & 444 deletions

File tree

‎scripts/README.md‎

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ EVAL_MODEL_ID=gemini-2.5-flash
6161
./scripts/quality_report.sh --session-ids-file ids.json # evaluate specific sessions
6262
./scripts/quality_report.sh --output-json report.json # write structured JSON output
6363
./scripts/quality_report.sh --threshold 15 # unhelpful rate warning at 15%
64-
./scripts/quality_report.sh --config config.json # scope-aware eval with config
64+
./scripts/quality_report.sh --agent-context agent_context.json # scope-aware eval
6565
./scripts/quality_report.sh --session <session_id> # evaluate single session (verbose)
6666
```
6767

@@ -124,7 +124,7 @@ The evaluation scores each session on **7 dimensions** using LLM-as-a-judge.
124124
| `task_grounding` | `grounded`, `ungrounded`, `no_tool_needed` | Whether the response is based on tool-retrieved data or fabricated |
125125

126126
The **`declined`** category is only included when scope context is provided
127-
(via `--config` or auto-discovered `agent_context.json`). Without scope
127+
(via `--agent-context` or auto-discovered `agent_context.json`). Without scope
128128
context, the judge has no basis for distinguishing intentional declines
129129
from failures, so only `meaningful`, `unhelpful`, and `partial` are used.
130130

@@ -172,18 +172,18 @@ Evaluate a single session and see all 7 metrics with full justifications:
172172
This is useful for verifying whether the LLM judge scored a specific
173173
session correctly, or for debugging individual conversations.
174174

175-
### Scope-Aware Evaluation (`--config`)
175+
### Scope-Aware Evaluation (`--agent-context`)
176176

177-
For more accurate scope evaluation, provide a config file that tells the
177+
For more accurate scope evaluation, provide a context file that tells the
178178
LLM judge exactly which topics your agent intentionally does not handle:
179179

180180
```bash
181-
./scripts/quality_report.sh --config agent_context.json --report
181+
./scripts/quality_report.sh --agent-context agent_context.json --report
182182
```
183183

184184
The script also auto-discovers `eval/data/agent_context.json` relative to
185-
the repo root or script directory, so `--config` is only needed to point
186-
at a non-default location.
185+
the repo root or script directory, so `--agent-context` is only needed to
186+
point at a non-default location.
187187

188188
A sample config is provided at `scripts/eval/data/agent_context.example.json`.
189189
Copy it and customize for your agent:
@@ -223,6 +223,31 @@ the config, the judge is told exactly which topics are out of scope, so it
223223
can correctly classify polite refusals as `declined` (correct behavior)
224224
rather than `unhelpful` (a bug).
225225

226+
### Custom Metrics (`--eval-config`)
227+
228+
Override the built-in metric definitions with your own:
229+
230+
```bash
231+
./scripts/quality_report.sh --eval-config scripts/eval/eval_config.json --report
232+
```
233+
234+
The eval config file is a JSON file with a `metrics` key — a list of metric
235+
definitions that replace the built-in 7 dimensions. Each metric has a `name`,
236+
`definition`, and a list of `categories` with scoring criteria. Metrics with
237+
`scope_aware: true` are automatically enriched with scope context when
238+
`--agent-context` is provided.
239+
240+
A complete example is provided at `scripts/eval/eval_config.json`. Copy it
241+
and customize for your evaluation needs:
242+
243+
```bash
244+
cp scripts/eval/eval_config.json my_eval_config.json
245+
# Edit metric definitions, add/remove dimensions, adjust categories
246+
./scripts/quality_report.sh --eval-config my_eval_config.json
247+
```
248+
249+
When `--eval-config` is not specified, the built-in metrics are used.
250+
226251
### A2A Support
227252

228253
The script automatically detects and resolves responses from remote A2A

‎scripts/eval/eval_config.json‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
{
2+
"metrics": [
3+
{
4+
"name": "response_usefulness",
5+
"definition": "Whether the agent final response provides a genuinely useful, substantive answer to the user question. A response that apologizes, says it cannot help, returns no data, provides only generic filler, or loops without resolving the question is NOT useful.",
6+
"categories": [
7+
{
8+
"name": "meaningful",
9+
"definition": "The response directly and substantively addresses the user question with specific, actionable information."
10+
},
11+
{
12+
"name": "unhelpful",
13+
"definition": "The response does NOT meaningfully answer the user question. This includes: (1) The agent said 'I don't have that information', gave generic advice, or directed the user elsewhere instead of using its tools. (2) The agent apologized without answering. (3) Empty data results or generic filler text. (4) The agent looped without resolution."
14+
},
15+
{
16+
"name": "partial",
17+
"definition": "The response partially addresses the question but is incomplete, missing key details, or only tangentially relevant."
18+
}
19+
],
20+
"required": true,
21+
"scope_aware": true,
22+
"declined_category": {
23+
"name": "declined",
24+
"definition": "The TOPIC of the question is explicitly listed as out of scope (see AGENT SCOPE CONTEXT above) and the agent correctly declined. Use this ONLY when the topic itself is out of scope -- NOT when the agent simply failed to find an answer for an in-scope topic.",
25+
"insert_after": "meaningful"
26+
},
27+
"scope_suffix": " UNLESS the question is outside the agent's defined scope, in which case a polite decline IS a correct and meaningful response."
28+
},
29+
{
30+
"name": "task_grounding",
31+
"definition": "Whether the agent response is grounded in actual data retrieved from its tools, or is fabricated / hallucinated general knowledge.",
32+
"categories": [
33+
{
34+
"name": "grounded",
35+
"definition": "The response is clearly based on data retrieved from the agent tools (search results, database lookups, API calls)."
36+
},
37+
{
38+
"name": "ungrounded",
39+
"definition": "The response appears to be fabricated or based on the LLM general knowledge rather than actual tool results. The tool may have returned empty data and the agent filled in anyway."
40+
},
41+
{
42+
"name": "no_tool_needed",
43+
"definition": "The question did not require tool usage and a direct LLM response was appropriate."
44+
}
45+
],
46+
"required": true
47+
},
48+
{
49+
"name": "correctness",
50+
"definition": "Whether the facts stated in the agent response are accurate. Evaluate based on the information the agent retrieved from its tools and whether it was conveyed faithfully.",
51+
"categories": [
52+
{
53+
"name": "correct",
54+
"definition": "All facts stated by the agent are accurate and consistent with the tool results retrieved."
55+
},
56+
{
57+
"name": "mostly_correct",
58+
"definition": "The response is mostly correct but contains a minor inaccuracy, omission, or imprecise wording."
59+
},
60+
{
61+
"name": "incorrect",
62+
"definition": "The response contains wrong facts, hallucinated information, or claims contradicted by the tool results."
63+
}
64+
],
65+
"required": true
66+
},
67+
{
68+
"name": "tool_usage",
69+
"definition": "Whether the agent used its available tools correctly to answer the question, rather than relying on general knowledge.",
70+
"categories": [
71+
{
72+
"name": "proper",
73+
"definition": "The agent used its tools and based the answer on the tool results. Tools were called with appropriate parameters."
74+
},
75+
{
76+
"name": "partial",
77+
"definition": "The agent partially used tools, or tool usage was unclear or incomplete. Some information may not be tool-derived."
78+
},
79+
{
80+
"name": "none",
81+
"definition": "The agent answered from general knowledge without looking up information via tools, even though tools were available and the question warranted their use."
82+
}
83+
],
84+
"required": true
85+
},
86+
{
87+
"name": "specificity",
88+
"definition": "Whether the agent response provides specific, concrete details (numbers, dates, dollar amounts, limits) rather than vague or generic statements.",
89+
"categories": [
90+
{
91+
"name": "specific",
92+
"definition": "The response includes specific and complete details: exact numbers, percentages, dollar amounts, dates, or limits."
93+
},
94+
{
95+
"name": "somewhat_specific",
96+
"definition": "The response is somewhat specific but missing some key details that would make it fully actionable."
97+
},
98+
{
99+
"name": "vague",
100+
"definition": "The response is vague, generic, or missing key specifics that the user needs to act on the information."
101+
}
102+
],
103+
"required": true
104+
},
105+
{
106+
"name": "scope_compliance",
107+
"definition": "Whether the agent correctly handled the scope of the question. An agent should answer in-scope questions and politely decline out-of-scope ones.",
108+
"categories": [
109+
{
110+
"name": "compliant",
111+
"definition": "The agent correctly answered an in-scope question OR correctly declined an out-of-scope question."
112+
},
113+
{
114+
"name": "partially_compliant",
115+
"definition": "The agent answered but with unnecessary caveats, excessive hedging, or was partially out of scope."
116+
},
117+
{
118+
"name": "non_compliant",
119+
"definition": "The agent tried to answer an out-of-scope question it should have declined, OR refused to answer an in-scope question it should have handled."
120+
}
121+
],
122+
"required": true,
123+
"scope_aware": true
124+
},
125+
{
126+
"name": "first_time_right",
127+
"definition": "Whether the agent's FIRST response in the conversation was satisfactory, without needing user corrections or follow-ups to fix errors. For single-turn conversations, evaluate the only response. For multi-turn, focus on whether the first substantive answer was correct.",
128+
"categories": [
129+
{
130+
"name": "correct",
131+
"definition": "The first response was correct and complete. No correction or significant clarification was needed from the user."
132+
},
133+
{
134+
"name": "clarification_needed",
135+
"definition": "The first response was mostly right but needed minor clarification or a follow-up to be fully useful."
136+
},
137+
{
138+
"name": "correction_needed",
139+
"definition": "The first response was wrong, vague, or incomplete enough that the user had to push back or correct the agent."
140+
}
141+
],
142+
"required": true
143+
}
144+
]
145+
}

‎scripts/eval/quality_metrics.json‎

Lines changed: 0 additions & 135 deletions
This file was deleted.

0 commit comments

Comments
 (0)