Skip to content

docs: add experiment framework design document#121

Merged
abrichr merged 1 commit into
mainfrom
docs/experiment-framework-design
Mar 17, 2026
Merged

docs: add experiment framework design document#121
abrichr merged 1 commit into
mainfrom
docs/experiment-framework-design

Conversation

@abrichr

@abrichr abrichr commented Mar 16, 2026

Copy link
Copy Markdown
Member

Summary

  • Reframes OpenAdapt as a general-purpose computer use framework, not a single-method system
  • Defines experiment tracks: demo-conditioning, LoRA-per-task, GRPO, SFT, API baselines, UI-Venus base model
  • Maps autoresearch pattern to OpenAdapt (mutable config, immutable eval, git-as-ratchet)
  • Differentiates wright (Level 2: code changes) vs autoresearch (Level 1: config optimization)
  • Tiered oracle architecture for trustworthy evaluation (static constraints → offline proxies → replay evals → live evals → adjudication)
  • Multi-objective lexicographic scoring (not single scalar)
  • Mutation surface ordering by signal-to-blast-radius (evaluator first, then narrow agent surfaces)
  • Reproducibility requirements and experiment lineage
  • Phased action plan: oracle quality → narrow agent optimization → broader search → training loops

Test plan

  • Review doc for accuracy and strategic alignment
  • Validate referenced codebase components exist

🤖 Generated with Claude Code

Frames OpenAdapt as a general-purpose computer use framework with
multiple experiment tracks (demo-conditioning, LoRA-per-task, GRPO,
SFT, API baselines, UI-Venus base model). Covers autoresearch pattern
adaptation, wright+autoresearch composition, tiered oracle architecture,
multi-objective scoring, mutation surface ordering, and reproducibility
requirements.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@abrichr abrichr merged commit f71c81f into main Mar 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant