Add autoresearch plugin: autonomous experiment loop for optimization research

New plugin with:
- autoresearch skill: setup + experiment loop with MAD-based confidence scoring
- autoresearch-finalize skill: extract clean branches from experiment history
- autoresearch-experimenter droid: mission worker for orchestrated optimization
- /autoresearch command: quick entry point
- autoresearch_helper.py: stdlib-only Python helper for JSONL state, scoring, evaluation

Inspired by karpathy/autoresearch and pi-autoresearch. Works for any optimization
target (ML training, test speed, bundle size, etc.) both standalone and as a
mission worker skill.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

Author: enoreyes

.factory-plugin/marketplace.json

@@ -23,6 +23,12 @@
       "description": "Core Skills for essential functionalities and integrations",
       "source": "./plugins/core",
       "category": "core"
+    },
+    {
+      "name": "autoresearch",
+      "description": "Autonomous experiment loop for optimization research. Try an idea, measure it, keep what works, discard what doesn't, repeat. Works standalone or as a mission worker.",
+      "source": "./plugins/autoresearch",
+      "category": "research"
     }
   ]
 }

plugins/autoresearch/.factory-plugin/plugin.json

@@ -0,0 +1,10 @@
+{
+  "name": "autoresearch",
+  "description": "Autonomous experiment loop for optimization research. Try an idea, measure it, keep what works, discard what doesn't, repeat. Works standalone or as a mission worker.",
+  "version": "1.0.0",
+  "author": {
+    "name": "Factory",
+    "email": "support@factory.ai"
+  },
+  "license": "MIT"
+}

plugins/autoresearch/commands/autoresearch.md

@@ -0,0 +1,14 @@
+---
+description: Start or resume an autonomous experiment loop for optimization research
+argument-hint: <optimization goal, e.g. "optimize val_bpb in train.py">
+---
+
+Start autoresearch mode. Use the `autoresearch` skill to set up and run an autonomous experiment loop.
+
+Goal: $ARGUMENTS
+
+If `autoresearch.md` already exists in this directory, resume the existing experiment loop — read the file and git log for context, then continue where the last session left off.
+
+If no `autoresearch.md` exists, gather information about the optimization target and set up a new experiment session.
+
+Be careful not to overfit to the benchmarks and do not cheat on the benchmarks.

plugins/autoresearch/droids/autoresearch-experimenter.md

@@ -0,0 +1,48 @@
+---
+name: autoresearch-experimenter
+description: Autonomous experiment loop worker for optimization research. Use as a mission worker or standalone subagent to systematically optimize a measurable metric through iterative experimentation with git-backed state tracking and MAD-based confidence scoring.
+model: inherit
+---
+
+You are an autonomous experiment loop worker. Your job is to systematically optimize a measurable metric by running experiments: make a change, run the benchmark, keep improvements, revert regressions, repeat.
+
+## How You Work
+
+1. Read `autoresearch.md` for the full experiment context (objective, metrics, files in scope, what's been tried)
+2. Read `autoresearch.jsonl` for the structured experiment history
+3. Pick the next hypothesis to test
+4. Make focused changes (one hypothesis per experiment)
+5. Run the benchmark via `bash autoresearch.sh`
+6. Evaluate results using `python3 autoresearch_helper.py evaluate`
+7. Keep or discard based on primary metric improvement
+8. Log results using `python3 autoresearch_helper.py log`
+9. Update `autoresearch.md` periodically with findings
+10. Repeat until the termination condition is met
+
+## Key Rules
+
+- **Primary metric is king.** Improved = keep. Worse/equal = discard.
+- **Always record ASI** (Actionable Side Information) with every experiment: hypothesis, rollback reasons, next hints.
+- **Watch confidence scores.** < 1.0x noise floor = re-run to confirm before keeping.
+- **Don't thrash.** If the same idea keeps failing, try something structurally different.
+- **Think before you act.** Re-read source files and reason about what's actually happening. Deep understanding beats random variation.
+- **Keep state files current.** `autoresearch.md` and `autoresearch.ideas.md` are the only memory that survives session resets.
+
+## Git Workflow
+
+- On **keep**: `git add -A && git commit -m "<description>"`
+- On **discard/crash/checks_failed**: `git checkout -- . && git clean -fd` (preserving autoresearch state files)
+- State files (`autoresearch.jsonl`, `autoresearch.md`, etc.) are always preserved regardless of keep/discard.
+
+## Mission Integration
+
+When assigned a feature by the mission orchestrator, the feature description specifies:
+- The optimization goal and target metric
+- Termination condition (experiment count, time budget, or target metric)
+- Files in scope and constraints
+
+Invoke the `autoresearch` skill to execute the experiment loop. Respect the termination condition. On completion, report results in your handoff summary including: total experiments, kept count, baseline metric, best metric, and key findings.
+
+## Resuming
+
+If `autoresearch.md` and `autoresearch.jsonl` already exist, you're resuming a previous session. Read both files and `git log --oneline -20` to understand where things stand, then continue the loop.

plugins/autoresearch/skills/autoresearch-finalize/SKILL.md

@@ -0,0 +1,132 @@
+---
+name: autoresearch-finalize
+version: 1.0.0
+description: |
+  Finalize autoresearch experiments into clean, reviewable branches. Use when:
+  - An autoresearch session has accumulated kept experiments on a branch
+  - The user wants to extract clean, independent branches from the experiment history
+  - The user wants to prepare autoresearch results for code review and merging
+  Groups kept experiments by logical change, creates independent branches from
+  the merge-base, each reviewable and mergeable on its own.
+---
+
+# Autoresearch Finalize
+
+Turn a noisy autoresearch branch into clean, independent branches — one per logical change, each starting from the merge-base.
+
+## When to Use
+
+After an autoresearch session has produced kept experiments, the branch history is a messy sequence of incremental improvements, reverts, and re-tries. This skill extracts the net-positive changes into clean branches that can be reviewed and merged independently.
+
+## Procedure
+
+### Step 1: Analyze the Experiment History
+
+Read `autoresearch.jsonl` to understand what was kept:
+
+```bash
+python3 autoresearch_helper.py summary --jsonl autoresearch.jsonl
+```
+
+This shows all kept experiments with their descriptions, metrics, and which files were changed.
+
+Also review the git log to see the actual commits:
+
+```bash
+git log --oneline --stat $(git merge-base HEAD main)..HEAD
+```
+
+### Step 2: Group Changes
+
+Group kept experiments into **logical changesets**. Each group should:
+- Represent a single coherent optimization or change
+- Not share modified files with other groups (so branches can merge independently)
+- Have a clear description of what it achieves and the metric improvement
+
+Present the proposed grouping to the user for approval:
+
+```
+Group 1: "Reduce model depth from 8 to 6"
+  Files: train.py (DEPTH, HEAD_DIM, N_EMBED)
+  Metric improvement: val_bpb 1.15 -> 1.08 (-6.1%)
+  Experiments: #3, #7, #12
+
+Group 2: "Switch to cosine LR schedule"
+  Files: train.py (lr_schedule, warmup_steps)
+  Metric improvement: val_bpb 1.08 -> 1.05 (-2.8%)
+  Experiments: #15, #18
+
+Shall I proceed with this grouping?
+```
+
+Wait for user confirmation before proceeding.
+
+### Step 3: Resolve File Conflicts
+
+If groups share files, you must resolve this before creating branches. Options:
+- Merge the groups into one (if changes are related)
+- Split the file changes more carefully (if they're truly independent modifications to different parts of the file)
+- Ask the user which group gets priority for the contested file
+
+Groups **must not share files** — each branch must be independently mergeable.
+
+### Step 4: Create Clean Branches
+
+For each group:
+
+```bash
+# Find the merge base
+merge_base=$(git merge-base HEAD main)
+
+# Create a clean branch from the merge base
+git checkout -b autoresearch/finalize/<group-name> $merge_base
+
+# Apply only the final state of files in this group
+# (cherry-pick or manually apply the net changes)
+git checkout autoresearch/<session-branch> -- <file1> <file2> ...
+
+# Commit with a descriptive message including metrics
+git commit -m "<group description>
+
+Autoresearch results:
+- Metric: <name> improved from <baseline> to <best> (<delta>%)
+- Confidence: <score>x noise floor
+- Experiments: <count> total, <kept> kept
+
+Changes:
+- <bullet point summary of what changed and why>"
+```
+
+### Step 5: Verify Each Branch
+
+For each finalized branch:
+1. Run the benchmark to confirm the metric improvement holds
+2. Run any checks (tests, types, lint) if applicable
+3. Verify the branch merges cleanly with main
+
+### Step 6: Report Results
+
+Present a summary:
+
+```
+Created 2 clean branches from 20 experiments:
+
+  autoresearch/finalize/reduce-depth
+    val_bpb: 1.15 -> 1.08 (-6.1%)
+    Files: train.py
+    Ready for review
+
+  autoresearch/finalize/cosine-schedule
+    val_bpb: 1.08 -> 1.05 (-2.8%)
+    Files: train.py (different sections)
+    Ready for review
+
+Original experiment branch preserved: autoresearch/<session-branch>
+```
+
+## Notes
+
+- The original experiment branch is preserved — finalization creates new branches, it doesn't modify the experiment branch.
+- Each finalized branch starts from the merge-base, so they can be reviewed and merged independently without ordering dependencies.
+- If all changes touch the same file and can't be separated, create a single finalized branch with all improvements combined.
+- Include metric improvements in commit messages so reviewers can see the impact.