|
| 1 | +# autoresearch: optimize-anything |
| 2 | + |
| 3 | +You are an autonomous code improvement agent. Your job is to iteratively improve the optimize-anything Python codebase by making small, focused changes, measuring their impact, and keeping improvements while discarding regressions. |
| 4 | + |
| 5 | +## Setup |
| 6 | + |
| 7 | +Before starting experiments, complete these steps: |
| 8 | + |
| 9 | +1. **Read the codebase.** Read every file in `src/optimize_anything/` to understand the architecture. This is a CLI tool for LLM-guided text optimization using GEPA (evolutionary prompt algorithm). Key modules: |
| 10 | + - `cli.py` (1,439 lines) — all 8 CLI commands. This is the god module. CC=54 in `_cmd_optimize`. |
| 11 | + - `evaluator_generator.py` (601 lines) — generates evaluator scripts from objectives |
| 12 | + - `llm_judge.py` (467 lines) — LLM-as-judge evaluator |
| 13 | + - `spec_loader.py` (165 lines) — loads optimization specs. `_normalize_spec` has CC=40. |
| 14 | + - `result_contract.py` (228 lines) — result formatting and analysis |
| 15 | + - `intake.py` (188 lines) — intake/explain pipeline |
| 16 | + - `evaluators.py` (173 lines) — evaluator types and dispatch |
| 17 | + - `dataset.py` (55 lines) — dataset/valset loading |
| 18 | + - `stop.py` (40 lines) — early stopping logic |
| 19 | + |
| 20 | +2. **Read the tests.** Skim `tests/` to understand what's covered. 308 tests, 85% coverage. The tests are READ-ONLY — you cannot modify them. They are the safety net. |
| 21 | + |
| 22 | +3. **Verify the eval harness.** Run `./autoresearch/eval.sh` and confirm you get a score around 58.18. This is the baseline. |
| 23 | + |
| 24 | +4. **Check git state.** You should be on branch `autoresearch/round1`. Confirm with `git branch --show-current`. |
| 25 | + |
| 26 | +5. **Read `autoresearch/results.tsv`** to see the baseline entry. |
| 27 | + |
| 28 | +6. **Confirm setup** by saying what branch you're on, what the baseline score is, and that you're ready to begin. Then immediately start experimenting. |
| 29 | + |
| 30 | +## Rules |
| 31 | + |
| 32 | +### What you CAN edit |
| 33 | +- Any file in `src/optimize_anything/` (the 11 source files listed above) |
| 34 | + |
| 35 | +### What you CANNOT edit |
| 36 | +- `tests/` — read-only. These are the safety net. Never modify, delete, or add tests. |
| 37 | +- `autoresearch/eval.sh` — read-only. This is the ground truth scorer. |
| 38 | +- `pyproject.toml` — no dependency changes. |
| 39 | +- `README.md`, `CHANGELOG.md`, docs — not in scope. |
| 40 | + |
| 41 | +### What you CANNOT do |
| 42 | +- Add new dependencies or imports from packages not already in use |
| 43 | +- Delete or rename any public API function (anything called from tests or CLI entry points) |
| 44 | +- Change CLI argument names or behavior (users depend on these) |
| 45 | +- Split `cli.py` into multiple files (this is a round 1 constraint — too risky) |
| 46 | +- Add `# type: ignore` comments to suppress mypy errors (that's gaming, not fixing) |
| 47 | +- Create trivial wrapper functions just to reduce average complexity |
| 48 | + |
| 49 | +## The Eval Harness |
| 50 | + |
| 51 | +Run: `./autoresearch/eval.sh` |
| 52 | + |
| 53 | +It outputs: |
| 54 | +``` |
| 55 | +tests=308 coverage=85.0% complexity_avg=7.03 mypy_errors=23 loc=3378 |
| 56 | +sub_scores: cov=85.0 cc=39.7 type=8.0 loc=100.0 |
| 57 | +score: 58.18 |
| 58 | +``` |
| 59 | + |
| 60 | +**Scoring:** |
| 61 | +- Tests are a HARD GATE. Any test failure = score 0. No exceptions. |
| 62 | +- Composite score (0-100) from 4 equally-weighted metrics: |
| 63 | + - **Coverage (25%):** `coverage_pct` directly (85% = 85 points in this bucket) |
| 64 | + - **Complexity (25%):** `max(0, 100 - (avg_cc - 1) * 10)` — lower average complexity = higher score |
| 65 | + - **Type safety (25%):** `max(0, 100 - mypy_errors * 4)` — each mypy error costs 4 points |
| 66 | + - **LOC efficiency (25%):** `min(130, 100 * baseline_loc / current_loc)` — rewards shrinking, capped |
| 67 | + |
| 68 | +**Current baseline: 58.18.** The biggest opportunity is type safety (only 8/100) and complexity (39.7/100). |
| 69 | + |
| 70 | +## Experiment Discipline |
| 71 | + |
| 72 | +### One change per experiment |
| 73 | +Each experiment should make ONE logical change. Not "fix types AND refactor a function AND remove dead code." One thing. This keeps diffs reviewable and makes keep/discard decisions clean. |
| 74 | + |
| 75 | +Examples of good single experiments: |
| 76 | +- Fix the 2 float coercion mypy errors in `llm_judge.py` |
| 77 | +- Extract a helper function from `_cmd_optimize` to reduce its complexity |
| 78 | +- Remove unused imports across all files |
| 79 | +- Add type annotations to `_normalize_spec` parameters |
| 80 | + |
| 81 | +Examples of bad experiments (too broad): |
| 82 | +- "Refactor cli.py" (too vague, too many changes) |
| 83 | +- "Fix all mypy errors" (23 errors across 4 files — break this into per-file experiments) |
| 84 | +- "Improve everything" (meaningless) |
| 85 | + |
| 86 | +### Scope limit |
| 87 | +Never change more than 100 lines in a single experiment. If your change requires more, break it into smaller experiments. |
| 88 | + |
| 89 | +### Commit messages |
| 90 | +Format: `exp-NNN: <what you changed> (<files touched>)` |
| 91 | +Example: `exp-003: extract _build_gepa_config from _cmd_optimize (cli.py)` |
| 92 | + |
| 93 | +## The Experiment Loop |
| 94 | + |
| 95 | +``` |
| 96 | +BASELINE_SHA = current HEAD |
| 97 | +BASELINE_SCORE = score from eval.sh |
| 98 | +
|
| 99 | +LOOP FOREVER: |
| 100 | + 1. Decide what to try next (see priority order below) |
| 101 | + 2. Make the change in source files |
| 102 | + 3. git add -A && git commit -m "exp-NNN: <description>" |
| 103 | + 4. Run: ./autoresearch/eval.sh > run.log 2>&1 |
| 104 | + 5. Read the score: grep "^score:" run.log |
| 105 | + 6. Read sub-scores: grep "^sub_scores:" run.log |
| 106 | +
|
| 107 | + IF score > BASELINE_SCORE: |
| 108 | + - Log to results.tsv with status "keep" |
| 109 | + - Update BASELINE_SHA and BASELINE_SCORE |
| 110 | + - Continue to next experiment |
| 111 | +
|
| 112 | + IF score <= BASELINE_SCORE: |
| 113 | + - Log to results.tsv with status "discard" |
| 114 | + - git reset --hard $BASELINE_SHA |
| 115 | + - Verify: run eval.sh again, confirm score matches BASELINE_SCORE |
| 116 | + - If verification fails, STOP and report the problem |
| 117 | + - Continue to next experiment |
| 118 | +
|
| 119 | + IF eval.sh crashes or tests fail (score = 0): |
| 120 | + - Log to results.tsv with status "crash" |
| 121 | + - git reset --hard $BASELINE_SHA |
| 122 | + - Try to understand what went wrong |
| 123 | + - Continue to next experiment |
| 124 | +``` |
| 125 | + |
| 126 | +## Results Logging |
| 127 | + |
| 128 | +Append each experiment to `autoresearch/results.tsv` (tab-separated): |
| 129 | + |
| 130 | +``` |
| 131 | +exp_id score cov cc_avg mypy_errs loc status description |
| 132 | +exp-000 58.18 85.0 7.03 23 3378 baseline initial state |
| 133 | +``` |
| 134 | + |
| 135 | +Log EVERY experiment — keeps, discards, and crashes. This is the science notebook. |
| 136 | + |
| 137 | +## Priority Order for Improvements |
| 138 | + |
| 139 | +Start with the highest-impact, lowest-risk changes: |
| 140 | + |
| 141 | +### Tier 1: Type Safety Fixes (highest ROI) |
| 142 | +There are 23 mypy errors. Each one fixed = +4 points on the type sub-score (which is currently 8/100). These are concrete, mechanical fixes: |
| 143 | +- Float coercion from `Any | None` in `llm_judge.py` and `evaluator_generator.py` |
| 144 | +- Untyped dict lookups in `cli.py` (use TypedDict or explicit type narrowing) |
| 145 | +- Fix these file by file, 1-3 errors per experiment |
| 146 | + |
| 147 | +### Tier 2: Complexity Reduction |
| 148 | +- `_cmd_optimize` has CC=54 (rated F). Extract logical chunks into helper functions. |
| 149 | +- `_normalize_spec` has CC=40 (rated E). Same approach. |
| 150 | +- `_apply_spec_to_args` has CC=27 (rated D). |
| 151 | +- Each extraction should be one experiment. |
| 152 | + |
| 153 | +### Tier 3: Dead Code and Simplification |
| 154 | +- Look for unused imports, unreachable branches, redundant conditionals |
| 155 | +- Simplify overly-nested logic |
| 156 | +- These often improve both complexity and LOC scores |
| 157 | + |
| 158 | +### Tier 4: Coverage Improvements |
| 159 | +- `evaluator_generator.py` is at 72% — the lowest coverage |
| 160 | +- You can't add tests, but you CAN restructure code so existing tests cover more paths |
| 161 | +- Extract untested private logic into tested public functions (if it makes architectural sense) |
| 162 | + |
| 163 | +### Tier 5: Architectural Cleanup |
| 164 | +- Reduce coupling between modules |
| 165 | +- Improve function signatures (explicit params over **kwargs where possible) |
| 166 | +- Add return type annotations to functions missing them |
| 167 | + |
| 168 | +## Anti-Gaming |
| 169 | + |
| 170 | +You are optimizing for REAL code quality, not for score. The human reviewer will read every diff. |
| 171 | + |
| 172 | +**Things that look like gaming:** |
| 173 | +- Adding `# type: ignore` to suppress mypy errors |
| 174 | +- Splitting functions into trivial one-line wrappers just to lower average CC |
| 175 | +- Deleting real functionality to lower LOC |
| 176 | +- Moving code between files without actually simplifying it |
| 177 | +- Adding dead branches to increase coverage of other paths |
| 178 | + |
| 179 | +**If you catch yourself doing any of these, stop and try a different approach.** The goal is code that's genuinely better — simpler, more correct, more maintainable. |
| 180 | + |
| 181 | +## Drift Check |
| 182 | + |
| 183 | +After every 10 experiments, do a full sanity check: |
| 184 | +1. Run the full test suite (all tests, including integration markers): `uv run python -m pytest --tb=short -q` |
| 185 | +2. Verify the score is still improving or stable |
| 186 | +3. Re-read the files you've been editing to make sure the code still reads well |
| 187 | +4. If anything seems off, stop and document what happened |
| 188 | + |
| 189 | +## NEVER STOP |
| 190 | + |
| 191 | +Once the experiment loop begins, do NOT pause to ask if you should continue. Do NOT ask "should I keep going?" or "is this a good stopping point?" The human may be away from the computer and expects you to continue working autonomously. |
| 192 | + |
| 193 | +If you run out of ideas in one tier, move to the next. If you've exhausted all tiers, go back to earlier tiers with fresh eyes — re-read the source files, look for patterns you missed, try combining approaches from previous near-misses. |
| 194 | + |
| 195 | +The loop runs until the human interrupts you. Period. |
0 commit comments