11# Evaluation + Optimization Closed Loop
22
3- This example implements a reproducible evaluation and prompt-optimization loop
4- that runs without ` TRPC_AGENT_API_KEY ` or any external model provider. It is an
5- example-local implementation: the code mirrors the evaluator/optimizer workflow
6- with a thin adapter so the behavior is deterministic and easy to review.
3+ This example implements issue #91 as a reproducible evaluation + optimization
4+ loop. The default path is deterministic fake mode, so it runs in CI and on a
5+ fresh checkout without ` TRPC_AGENT_API_KEY ` or any external model provider. A
6+ real SDK adapter path is also present in ` eval_loop/backends.py ` for
7+ ` AgentOptimizer ` / ` TargetPrompt ` integration.
78
89## Architecture
910
1011``` text
11- train.evalset.json val.evalset.json baseline_system_prompt.txt
12- | | |
13- v v v
14- loader.py --------------> evaluator.py <----------- fake_model.py
15- | | |
16- | v |
17- | fake_judge.py |
18- | | |
19- v v |
20- attribution.py <------ baseline + candidate results |
21- | | |
22- v v |
23- optimizer.py --------> candidate prompts --------------+
24- | |
25- v v
26- gate.py ------------> accept/reject decisions
12+ train.evalset.json + val.evalset.json + optimizer.json + baseline prompt
2713 |
2814 v
29- report.py ----------> optimization_report.json/.md
15+ loader.py / config.py --> validated cases, gate config, input hashes
16+ |
17+ v
18+ backends.py
19+ |-- FakeBackend -> FakeModel + FakeJudge + FakeOptimizer
20+ `-- SDKBackend -> AgentOptimizer + TargetPrompt + user call_agent
21+ |
22+ v
23+ attribution.py -> evaluator.py -> gate.py -> report.py
24+ |
25+ v
26+ optimization_report.json / optimization_report.md / runs/<run_id>/ audit files
3027```
3128
3229## Quick Start
3330
34- Short deterministic command :
31+ One-command fake mode :
3532
3633``` bash
3734python examples/optimization/eval_optimize_loop/run_pipeline.py --fake-model --fake-judge --trace
3835```
3936
40- Full command with explicit inputs:
37+ Equivalent new form:
38+
39+ ``` bash
40+ python examples/optimization/eval_optimize_loop/run_pipeline.py --mode fake --trace
41+ ```
42+
43+ Full fake command:
4144
4245``` bash
4346python examples/optimization/eval_optimize_loop/run_pipeline.py \
@@ -46,86 +49,94 @@ python examples/optimization/eval_optimize_loop/run_pipeline.py \
4649 --optimizer-config examples/optimization/eval_optimize_loop/data/optimizer.json \
4750 --prompt examples/optimization/eval_optimize_loop/prompts/baseline_system_prompt.txt \
4851 --output-dir /tmp/eval-optimize-loop \
49- --fake-model \
50- --fake-judge \
52+ --mode fake \
5153 --trace
5254```
5355
54- The default output directory is the system temp directory,
55- ` eval-optimize-loop ` . The fixed checked-in examples are:
56+ SDK adapter command shape:
5657
57- - ` outputs/optimization_report.example.json `
58- - ` outputs/optimization_report.example.md `
58+ ``` bash
59+ python examples/optimization/eval_optimize_loop/run_pipeline.py \
60+ --mode sdk \
61+ --train path/to/sdk_train.evalset.json \
62+ --val path/to/sdk_val.evalset.json \
63+ --optimizer-config path/to/sdk_optimizer.json \
64+ --prompt path/to/system_prompt.txt \
65+ --sdk-call-agent your_package.your_module:call_agent \
66+ --output-dir /tmp/eval-optimize-loop-sdk
67+ ```
5968
60- ## What The Example Demonstrates
69+ ` --sdk-call-agent ` must point to an async callable compatible with
70+ ` AgentOptimizer.optimize(call_agent=...) ` . Configure any real model credentials
71+ needed by that callable. SDK mode never silently falls back to fake mode.
6172
62- - Baseline evaluation on train and validation evalsets.
63- - Rule-based failure attribution for failed cases.
64- - A fake optimizer that proposes two candidates:
65- - ` candidate_001_overfit ` : improves train score but regresses validation.
66- - ` candidate_002_safe ` : improves validation without protected regression.
67- - Candidate validation on the full validation set.
68- - A configurable gate that rejects train-only gains and protected-case
69- regressions.
70- - Audit persistence with seed, cost, deterministic duration, config hash, and
71- candidate prompt text.
73+ ## Source Prompt Writes
7274
73- The six eval cases cover:
75+ The default is ** no source write-back** . The baseline prompt file is not modified
76+ by fake mode, and ` SDKBackend ` calls ` AgentOptimizer.optimize(update_source=False) `
77+ unless ` --update-source ` is explicitly passed. The report records
78+ ` run.update_source ` and the Markdown report states whether source write-back was
79+ enabled.
7480
75- - optimization succeeds: ` candidate_002_safe ` is accepted;
76- - optimization has no effect: rubric cases stay unchanged;
77- - optimization regresses/overfits: ` candidate_001_overfit ` forces JSON on
78- validation prose and protected exact-answer cases.
81+ ## Candidate Behavior
7982
80- ## Fake Model, Fake Judge, And Trace Mode
83+ The fake optimizer proposes exactly two candidates:
8184
82- ` --fake-model ` uses deterministic prompt markers to simulate baseline, overfit,
83- and safe candidate behavior. ` --fake-judge ` scores JSON, exact-answer, and
84- rubric cases locally. ` --trace ` stores per-case fake model and judge decisions
85- inside ` optimization_report.json ` , which makes failures reviewable without
86- calling an external service .
85+ - ` candidate_001_overfit ` : fixes train formatting but forces JSON too broadly;
86+ it improves train score and regresses validation, so the gate rejects it.
87+ - ` candidate_002_safe ` : applies strict JSON/exact-answer behavior only when
88+ requested; it improves validation without protected-case regression, so the
89+ gate may accept it .
8790
88- This example currently requires ` --fake-model --fake-judge ` . That is deliberate:
89- it keeps the PR deterministic and ensures the example runs in CI without API
90- keys.
91+ The fake model is driven by ` EvalCase.expectation ` , ` tags ` , ` protected ` , and
92+ optional ` simulated_outputs ` ; it does not depend on sample ` case_id ` names.
9193
92- ## Inspecting Reports
94+ ## Reports
9395
94- Open ` optimization_report.md ` first for the human-readable decision :
96+ ` optimization_report.json ` includes :
9597
96- - final selected candidate;
97- - why the overfit candidate was rejected;
98- - why the safe candidate was accepted;
99- - baseline vs candidate score table;
100- - per-case delta table;
101- - failure attribution summary;
102- - prompt diffs;
103- - reproducibility command.
98+ - ` schema_version ` ;
99+ - ` run ` metadata: mode, fake flags, trace flag, case counts, update-source flag,
100+ and input paths;
101+ - ` baseline ` plus compatibility fields ` baseline_train ` and
102+ ` baseline_validation ` ;
103+ - all candidate train/validation results, rationale, and prompt diff;
104+ - per-case deltas with ` delta_type ` (` new_pass ` , ` new_fail ` , ` score_up ` ,
105+ ` score_down ` , ` unchanged ` );
106+ - failure attribution summary and attribution accuracy when expected labels are
107+ present;
108+ - gate decisions with overfit detection, protected regressions, new hard
109+ failures, excessive drops, and cost fields;
110+ - audit data: seed, duration, config hash, input hashes, candidate prompt hashes,
111+ cost, prompt diffs, and reproducibility command.
104112
105- Use ` optimization_report.json ` for automation and audit. It includes the same
106- decision data plus full case outputs, traces, costs, config hash, and candidate
107- prompt snapshots .
113+ ` optimization_report.md ` includes final decision, gate reasons, score table,
114+ per- case delta table, failure attribution summary, cost/audit details, prompt
115+ diffs, and the reproducibility command .
108116
109- ## Run Tests
117+ ` report.py ` also writes audit artifacts under ` output_dir/runs/<run_id>/ ` :
110118
111- ``` bash
112- python -m pytest examples/optimization/eval_optimize_loop/tests
113- ```
119+ - ` config.snapshot.json ` ;
120+ - ` input_hashes.json ` ;
121+ - ` candidate_prompts/<candidate_id>/system_prompt.txt ` ;
122+ - ` case_results/<candidate_id>_<split>.json ` ;
123+ - ` prompt_diffs/<candidate_id>.diff ` .
114124
115- The tests cover gate rejection/acceptance, failure attribution, fake-mode report
116- generation, and deterministic output with the same seed.
125+ The repository keeps only stable examples:
117126
118- ## Switching To Real Evaluator/Optimizer Later
127+ - ` outputs/optimization_report.example.json `
128+ - ` outputs/optimization_report.example.md `
129+
130+ Runtime ` optimization_report.json ` , ` optimization_report.md ` , and ` runs/ `
131+ directories are not committed.
119132
120- The example intentionally isolates integration points:
133+ ## Run Tests
121134
122- - replace ` ExampleEvaluator ` in ` eval_loop/evaluator.py ` with an adapter around
123- ` AgentEvaluator ` when real agent execution and SDK evalsets are desired;
124- - replace ` FakeOptimizer ` in ` eval_loop/optimizer.py ` with ` AgentOptimizer ` or a
125- remote optimizer;
126- - keep ` gate.py ` , ` attribution.py ` , and ` report.py ` as reviewable policy and
127- audit layers around the real components.
135+ ``` bash
136+ python -m pytest examples/optimization/eval_optimize_loop/tests
137+ ```
128138
129- When switching to real mode, keep train and validation files distinct, preserve
130- the protected-case gate, and continue writing both JSON and Markdown reports so
131- optimization decisions remain reviewable.
139+ The tests cover fake hidden-sample generalization, config validation, gate
140+ rejection paths, protected-case behavior, failure attribution, tool/knowledge
141+ judge paths, SDK adapter wiring through monkeypatching, deterministic report
142+ generation, and both CLI forms.
0 commit comments