feat(experiment): fan-out + compare + crown#240
Open
SDSLeon wants to merge 1 commit into
Open
Conversation
Introduce experiments that fan one prompt across multiple agent/model candidates, each in its own worktree, for side-by-side diff comparison. - add experiment contract, IPC procedure, and supervisor judge with one-shot prompt runner and response parsing - add experiment store slice, launcher store, and actions to create candidate threads in batch and crown a winner - add experiment app view with candidate cards and NewExperimentModal - wire experiment view into routing, hydration, overlays, and sidebar - persist experiments through dbStorage and project/view selectors
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds experiments: fan one prompt across multiple agent/model candidates, each in its own git worktree, then compare diffs side-by-side and merge the winner — with an optional LLM-judge crown that pre-selects the best diff and a one-line rationale (user-overridable).
This is mostly orchestration glue on top of existing primitives (threads, worktrees, diffs, the one-shot prompt runner). An experiment is a thread group (candidates share
groupId === experiment.id) plus anexperimentsregistry holding the orchestration metadata.How to use
+/−, files), and per-card Open / Crown / Merge winner.What's included
experiment.tscontract,judgeExperimentIPC procedure, supervisor judge (experimentJudge.ts) built on the existingrunOneShotPromptWithFallback, with robust response parsing.experimentSlice(persisted),experimentLauncherStore,createThreadsBatch, andopenExperiment/openThreadStandalone/closeExperimentview actions; newAppViewkind"experiment".experimentActions.ts): per-candidate worktree creation (sequential — avoidsindex.lockraces), headlessstartThreadlaunch,mergeExperimentWinner(viagitMergeToSource+performWorktreeRemoval),discardExperiment,crownExperiment.ExperimentViewboard +ExperimentCandidateCard,NewExperimentModal, composer trigger, sidebar experiment-group affordance.Design notes (informed by prior art: cmux, Vibe Kanban, uzi, Crystal)
Testing
pnpm run typecheck✅pnpm run lint(touched files) ✅pnpm exec vitest runforexperimentJudge,appStore,dbStorage,threadActions, and the app integration test — 99 passing ✅Follow-ups (not in this PR)
🤖 Generated with Claude Code