Commit e4af59f
committed
Add parroting penalty to eval and auto-discover eval_config.json
Penalize parroted recoveries (agent echoes user's correction without
re-verifying via tool) as unhelpful in both the usefulness definition
and the unhelpful category.
Refactor quality_report.py to auto-discover eval/eval_config.json
from the repo root, removing 290 lines of hardcoded metric
definitions in favor of the config-driven approach.1 parent 387ee89 commit e4af59f
2 files changed
Lines changed: 74 additions & 327 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
0 commit comments