Skip to content

Commit efdb8e7

Browse files
Fix all P0/P1/P2 review findings in autoresearch plugin
SKILL.md: - P0: Add init subcommand call in setup, include JSONL in initial commit - P0: Reorder keep flow: log to JSONL before git commit (prevents entry loss) - P0: Replace git clean -fd with backup/restore pattern for state files - P1: Add concrete baseline commands in Step 4 - P1: Add --asi flag to all log invocation examples - P1: Handle existing branch on resume (checkout || checkout -b) - P2: Fix evaluate text (compare against best-kept, not baseline) - P2: Skip user confirmation in mission worker finalization - P2: Reference status subcommand in resume section - P2: Show --metrics flag in log examples autoresearch_helper.py: - P0: Replace > 0 metric filter with status-based filter (supports zero/negative metrics) - P1: Use find_baseline() consistently in compute_confidence - P1: Add init-before-log guard (exit 1 if no config) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
1 parent 91141e4 commit efdb8e7

2 files changed

Lines changed: 76 additions & 28 deletions

File tree

plugins/autoresearch/skills/autoresearch/SKILL.md

Lines changed: 66 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Ask the user (or infer from context) for:
4141
### Step 2: Create Branch and State Files
4242

4343
```bash
44-
git checkout -b autoresearch/<goal>-<date>
44+
git checkout autoresearch/<goal>-<date> 2>/dev/null || git checkout -b autoresearch/<goal>-<date>
4545
```
4646

4747
Read the source files thoroughly. Understand the workload deeply before writing anything.
@@ -115,16 +115,41 @@ pnpm test --run --reporter=dot 2>&1 | tail -50
115115
pnpm typecheck 2>&1 | grep -i error || true
116116
```
117117

118-
### Step 3: Commit State Files
118+
### Step 3: Initialize JSONL and Commit State Files
119+
120+
Initialize the experiment log:
121+
122+
```bash
123+
python3 autoresearch_helper.py init --jsonl autoresearch.jsonl --name '<goal>' --metric-name '<metric_name>' --direction <lower|higher>
124+
```
125+
126+
Commit all state files:
119127

120128
```bash
121-
git add autoresearch.md autoresearch.sh
129+
git add autoresearch.md autoresearch.sh autoresearch.jsonl
122130
git commit -m "autoresearch: initialize experiment session"
123131
```
124132

125133
### Step 4: Run Baseline
126134

127-
Run the experiment command and record the baseline result. This is experiment #1 — it establishes the starting point.
135+
Run the benchmark and record the baseline result:
136+
137+
```bash
138+
bash autoresearch.sh
139+
```
140+
141+
Parse the METRIC lines from the output, then log the baseline as a keep:
142+
143+
```bash
144+
python3 autoresearch_helper.py log --jsonl autoresearch.jsonl \
145+
--commit $(git rev-parse --short=7 HEAD) \
146+
--metric <baseline_value> \
147+
--status keep \
148+
--description "baseline" \
149+
--asi '{"hypothesis": "baseline measurement"}'
150+
```
151+
152+
This is experiment #1 — it establishes the starting point for all future comparisons.
128153

129154
## The Experiment Loop
130155

@@ -163,7 +188,7 @@ If checks fail, log as `checks_failed` and revert.
163188

164189
#### 4. Evaluate Results
165190

166-
Compare the primary metric against the baseline using the helper script:
191+
Compare the primary metric against the current best (or baseline if no keeps yet) using the helper script:
167192

168193
```bash
169194
python3 autoresearch_helper.py evaluate --jsonl autoresearch.jsonl --metric <value> --direction <lower|higher>
@@ -182,44 +207,58 @@ Decision rules:
182207

183208
**On keep:**
184209

185-
```bash
186-
# Commit the changes
187-
git add -A
188-
git commit -m "<description>
189-
190-
Result: {\"status\": \"keep\", \"<metric_name>\": <value>}"
191-
```
192-
193-
Append to `autoresearch.jsonl`:
210+
Log to JSONL first (so the entry is included in the commit):
194211
```bash
195212
python3 autoresearch_helper.py log --jsonl autoresearch.jsonl \
196213
--commit $(git rev-parse --short=7 HEAD) \
197214
--metric <value> \
198215
--status keep \
199216
--description "<what was tried>" \
217+
--asi '{"hypothesis": "<what you tried>"}' \
218+
# --metrics '{"compile_us": <value>, "render_us": <value>}' # optional secondary metrics
200219
--direction <lower|higher>
201220
```
202221

203-
**On discard/crash/checks_failed:**
204-
222+
Then commit all changes (including the JSONL entry):
205223
```bash
206-
# Revert changes (preserve autoresearch state files)
207-
git checkout -- .
208-
git clean -fd 2>/dev/null
209-
# Re-stage state files in case they were unstaged
210-
git add autoresearch.jsonl autoresearch.md autoresearch.ideas.md autoresearch.sh autoresearch.checks.sh 2>/dev/null || true
224+
git add -A
225+
git commit -m "<description>
226+
227+
Result: {\"status\": \"keep\", \"<metric_name>\": <value>}"
211228
```
212229

213-
Append to `autoresearch.jsonl`:
230+
**On discard/crash/checks_failed:**
231+
232+
Log to JSONL first (before reverting, so the entry is preserved):
214233
```bash
215234
python3 autoresearch_helper.py log --jsonl autoresearch.jsonl \
216235
--commit "0000000" \
217236
--metric <value_or_0> \
218237
--status <discard|crash|checks_failed> \
219238
--description "<what was tried>" \
239+
--asi '{"hypothesis": "<what you tried>", "rollback_reason": "<why it failed>"}' \
240+
# --metrics '{"compile_us": <value>, "render_us": <value>}' # optional secondary metrics
220241
--direction <lower|higher>
221242
```
222243

244+
Then revert changes, backing up state files so `git clean -fd` doesn't destroy them:
245+
```bash
246+
# Backup state files
247+
cp autoresearch.jsonl autoresearch.jsonl.bak 2>/dev/null || true
248+
cp autoresearch.md autoresearch.md.bak 2>/dev/null || true
249+
cp autoresearch.ideas.md autoresearch.ideas.md.bak 2>/dev/null || true
250+
251+
# Revert all changes
252+
git checkout -- .
253+
git clean -fd 2>/dev/null
254+
255+
# Restore state files
256+
cp autoresearch.jsonl.bak autoresearch.jsonl 2>/dev/null || true
257+
cp autoresearch.md.bak autoresearch.md 2>/dev/null || true
258+
cp autoresearch.ideas.md.bak autoresearch.ideas.md 2>/dev/null || true
259+
rm -f autoresearch.jsonl.bak autoresearch.md.bak autoresearch.ideas.md.bak
260+
```
261+
223262
#### 6. Update Research Journal
224263

225264
After every few experiments (or after significant findings), update the "What's Been Tried" section in `autoresearch.md`. Include:
@@ -296,7 +335,10 @@ Droid sessions have finite context. To handle this gracefully:
296335
1. **Track experiment count** in the current session. After ~15 experiments, context is getting heavy.
297336
2. **Save state proactively** — all state lives in files (jsonl, md), so a new session can resume immediately.
298337
3. **When context is getting exhausted**: update `autoresearch.md` with current findings, commit state files, and stop. The next session reads the files and continues.
299-
4. **On resume**: read `autoresearch.md`, `autoresearch.jsonl`, and `git log --oneline -20` to understand where things stand.
338+
4. **On resume**: read `autoresearch.md`, `autoresearch.jsonl`, and `git log --oneline -20` to understand where things stand. Check current status:
339+
```bash
340+
python3 autoresearch_helper.py status --jsonl autoresearch.jsonl
341+
```
300342

301343
## Loop Rules Summary
302344

@@ -347,7 +389,7 @@ Group 2: "Switch to cosine LR schedule"
347389
Experiments: #15, #18
348390
```
349391

350-
Wait for user confirmation before proceeding.
392+
Wait for user confirmation before proceeding. In mission worker mode, proceed with the best grouping without waiting for confirmation.
351393

352394
### Step 3: Resolve File Conflicts
353395

plugins/autoresearch/skills/autoresearch/autoresearch_helper.py

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ def compute_confidence(results, segment, direction):
7676
7777
Returns None if fewer than 3 data points or MAD is 0.
7878
"""
79-
cur = [r for r in current_segment_results(results, segment) if r.get("metric", 0) > 0]
79+
cur = [r for r in current_segment_results(results, segment) if r.get("status") not in ("crash", "checks_failed")]
8080
if len(cur) < 3:
8181
return None
8282

@@ -85,11 +85,13 @@ def compute_confidence(results, segment, direction):
8585
if mad == 0:
8686
return None
8787

88-
baseline = cur[0]["metric"]
88+
baseline = find_baseline(results, segment)
89+
if baseline is None:
90+
return None
8991

9092
best_kept = None
9193
for r in cur:
92-
if r.get("status") == "keep" and r.get("metric", 0) > 0:
94+
if r.get("status") == "keep":
9395
val = r["metric"]
9496
if best_kept is None:
9597
best_kept = val
@@ -116,7 +118,7 @@ def find_best_kept(results, segment, direction):
116118
cur = current_segment_results(results, segment)
117119
best = None
118120
for r in cur:
119-
if r.get("status") == "keep" and r.get("metric", 0) > 0:
121+
if r.get("status") == "keep":
120122
val = r["metric"]
121123
if best is None:
122124
best = val
@@ -150,6 +152,10 @@ def cmd_log(args):
150152
"""Append an experiment result to the JSONL file."""
151153
config, results = read_jsonl(args.jsonl)
152154

155+
if config is None:
156+
print("Error: No config found. Run 'init' first.", file=sys.stderr)
157+
sys.exit(1)
158+
153159
segment = config.get("_segment", 0) if config else 0
154160
direction = args.direction or (config.get("bestDirection", "lower") if config else "lower")
155161

0 commit comments

Comments
 (0)