Skip to content

Commit e4af59f

Browse files
committed
Add parroting penalty to eval and auto-discover eval_config.json
Penalize parroted recoveries (agent echoes user's correction without re-verifying via tool) as unhelpful in both the usefulness definition and the unhelpful category. Refactor quality_report.py to auto-discover eval/eval_config.json from the repo root, removing 290 lines of hardcoded metric definitions in favor of the config-driven approach.
1 parent 387ee89 commit e4af59f

2 files changed

Lines changed: 74 additions & 327 deletions

File tree

scripts/eval/eval_config.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@
22
"metrics": [
33
{
44
"name": "response_usefulness",
5-
"definition": "Whether the agent final response provides a genuinely useful, substantive answer to the user question. A response that apologizes, says it cannot help, returns no data, provides only generic filler, or loops without resolving the question is NOT useful.",
5+
"definition": "Whether the agent final response provides a genuinely useful, substantive answer to the user question. A response that apologizes, says it cannot help, returns no data, provides only generic filler, or loops without resolving the question is NOT useful. If the conversation contains a user correction and the agent merely repeated or acknowledged the correction without independently verifying it (e.g. re-querying a tool, citing a new source), the response is NOT useful — the user did the agent's work.",
66
"categories": [
77
{
88
"name": "meaningful",
99
"definition": "The response directly and substantively addresses the user question with specific, actionable information."
1010
},
1111
{
1212
"name": "unhelpful",
13-
"definition": "The response does NOT meaningfully answer the user question. This includes: (1) The agent said 'I don't have that information', gave generic advice, or directed the user elsewhere instead of using its tools. (2) The agent apologized without answering. (3) Empty data results or generic filler text. (4) The agent looped without resolution."
13+
"definition": "The response does NOT meaningfully answer the user question. This includes: (1) The agent said 'I don't have that information', gave generic advice, or directed the user elsewhere instead of using its tools. (2) The agent apologized without answering. (3) Empty data results or generic filler text. (4) The agent looped without resolution. (5) The agent only became correct after the user provided the right answer and the agent repeated it without independent verification (e.g. re-querying a tool)."
1414
},
1515
{
1616
"name": "partial",

0 commit comments

Comments
 (0)