|
| 1 | +# Agent Profile Routing Implementation Plan |
| 2 | + |
| 3 | +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | +
|
| 5 | +**Goal:** Replace fragmented model-only selection with a single, versioned agent-profile routing boundary while retaining compatibility for existing Maestro modes and clients. |
| 6 | + |
| 7 | +**Architecture:** A new profile module owns user-facing capability levels and complete invocation bundles. Existing mode and intelligent-router entry points adapt to that module during migration, so every surface receives the same model, reasoning, oracle, specialist, and budget configuration. Routing promotion remains conservative until verified outcome evidence meets an explicit sample threshold. |
| 8 | + |
| 9 | +**Tech Stack:** TypeScript, Bun, Vitest, existing Maestro model registry and routing services. |
| 10 | + |
| 11 | +## Global Constraints |
| 12 | + |
| 13 | +- Preserve `smart`, `rush`, `free`, `custom`, `frontier`, and `replay` as accepted compatibility inputs. |
| 14 | +- Expose `low`, `medium`, `high`, and `ultra` as the canonical user-facing capability levels. |
| 15 | +- Keep provider/model identifiers in the model registry layer; profiles reference registered identifiers and capabilities. |
| 16 | +- No routing promotion based solely on an assistant message completing without an error. |
| 17 | +- Every behavior change follows a failing-test, passing-test cycle. |
| 18 | + |
| 19 | +--- |
| 20 | + |
| 21 | +### Task 1: Versioned agent profiles and compatibility aliases |
| 22 | + |
| 23 | +**Files:** |
| 24 | +- Create: `src/agent/profiles.ts` |
| 25 | +- Modify: `src/agent/modes.ts` |
| 26 | +- Modify: `src/agent/index.ts` |
| 27 | +- Test: `test/agent/profiles.test.ts` |
| 28 | +- Test: `test/agent/modes.test.ts` |
| 29 | + |
| 30 | +**Interfaces:** |
| 31 | +- Produces: `AgentProfileLevel`, `AgentProfile`, `AGENT_PROFILES`, `resolveAgentProfile(input, provider)`, and `parseAgentProfileLevel(input)`. |
| 32 | +- Consumes: existing `ModelProvider`, `ReasoningEffort`, `SubagentType`, and model-tier resolution during the compatibility period. |
| 33 | + |
| 34 | +- [ ] **Step 1: Write failing profile tests** |
| 35 | + |
| 36 | +```ts |
| 37 | +expect(parseAgentProfileLevel("rush")).toBe("low"); |
| 38 | +expect(parseAgentProfileLevel("smart")).toBe("medium"); |
| 39 | +expect(resolveAgentProfile("high", "openai-codex")).toMatchObject({ |
| 40 | + id: "high-v1", |
| 41 | + level: "high", |
| 42 | + primary: { provider: "openai-codex", reasoningEffort: "xhigh" }, |
| 43 | + oracle: { provider: "anthropic" }, |
| 44 | +}); |
| 45 | +``` |
| 46 | + |
| 47 | +- [ ] **Step 2: Run the tests and verify missing exports fail** |
| 48 | + |
| 49 | +Run: `bunx vitest --run test/agent/profiles.test.ts test/agent/modes.test.ts` |
| 50 | +Expected: FAIL because `src/agent/profiles.ts` and its exports do not exist. |
| 51 | + |
| 52 | +- [ ] **Step 3: Implement immutable profile definitions and alias parsing** |
| 53 | + |
| 54 | +Define `low-v1`, `medium-v1`, `high-v1`, and `ultra-v1` profiles. Each profile contains primary and oracle invocation configurations, specialist dispatch, fallback levels, and task budgets. Adapt `parseMode`, `getModelForMode`, and subagent dispatch through canonical levels without removing legacy inputs. |
| 55 | + |
| 56 | +- [ ] **Step 4: Run focused tests** |
| 57 | + |
| 58 | +Run: `bunx vitest --run test/agent/profiles.test.ts test/agent/modes.test.ts test/codex/subagent-dispatch-table.test.ts` |
| 59 | +Expected: PASS. |
| 60 | + |
| 61 | +### Task 2: Route profiles through the intelligent router |
| 62 | + |
| 63 | +**Files:** |
| 64 | +- Modify: `src/services/intelligent-router/types.ts` |
| 65 | +- Modify: `src/services/intelligent-router/normalize.ts` |
| 66 | +- Modify: `src/services/intelligent-router/service.ts` |
| 67 | +- Modify: `src/services/intelligent-router/recorder.ts` |
| 68 | +- Modify: `src/server/handlers/chat.ts` |
| 69 | +- Test: `test/services/intelligent-router.test.ts` |
| 70 | +- Test: `test/web/chat-handler-routing.test.ts` |
| 71 | + |
| 72 | +**Interfaces:** |
| 73 | +- Consumes: `AgentProfile` and `resolveAgentProfile` from Task 1. |
| 74 | +- Produces: routing decisions containing `selectedProfile`, `fallbackProfiles`, and the resolved invocation profile while retaining `selectedModel` for wire compatibility. |
| 75 | + |
| 76 | +- [ ] **Step 1: Write failing decision-contract tests** |
| 77 | + |
| 78 | +```ts |
| 79 | +expect(decision.selectedProfile).toMatchObject({ level: "medium", id: "medium-v1" }); |
| 80 | +expect(decision.fallbackProfiles.map(profile => profile.level)).toContain("low"); |
| 81 | +``` |
| 82 | + |
| 83 | +- [ ] **Step 2: Run focused tests and verify the new contract is absent** |
| 84 | + |
| 85 | +Run: `bunx vitest --run test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts` |
| 86 | +Expected: FAIL because routing decisions contain models only. |
| 87 | + |
| 88 | +- [ ] **Step 3: Implement the profile adapter and compatibility projection** |
| 89 | + |
| 90 | +Resolve a profile before candidate scoring, constrain candidates to profile policy, return the selected immutable profile, and project its primary invocation into the legacy model fields. |
| 91 | + |
| 92 | +- [ ] **Step 4: Run focused tests** |
| 93 | + |
| 94 | +Run: `bunx vitest --run test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts` |
| 95 | +Expected: PASS. |
| 96 | + |
| 97 | +### Task 3: Verified outcome evidence and conservative promotion |
| 98 | + |
| 99 | +**Files:** |
| 100 | +- Modify: `src/services/intelligent-router/types.ts` |
| 101 | +- Modify: `src/services/intelligent-router/normalize.ts` |
| 102 | +- Modify: `src/services/intelligent-router/service.ts` |
| 103 | +- Modify: `src/services/intelligent-router/recorder.ts` |
| 104 | +- Create: `src/services/intelligent-router/outcome.ts` |
| 105 | +- Test: `test/services/intelligent-router.test.ts` |
| 106 | +- Create: `test/services/intelligent-router-outcome.test.ts` |
| 107 | + |
| 108 | +**Interfaces:** |
| 109 | +- Produces: `RoutingOutcomeEvidence`, `deriveRoutingOutcome(evidence)`, `MIN_VERIFIED_SAMPLES`, and confidence metadata on routing scores. |
| 110 | +- Consumes: verifier results, user acceptance/retry signals, task cost, and terminal assistant status. |
| 111 | + |
| 112 | +- [ ] **Step 1: Write failing outcome tests** |
| 113 | + |
| 114 | +```ts |
| 115 | +expect(deriveRoutingOutcome({ assistantCompleted: true })).toEqual({ verified: false }); |
| 116 | +expect(deriveRoutingOutcome({ assistantCompleted: true, verificationPassed: true })) |
| 117 | + .toMatchObject({ verified: true, success: true }); |
| 118 | +``` |
| 119 | + |
| 120 | +- [ ] **Step 2: Run tests and verify the outcome module is missing** |
| 121 | + |
| 122 | +Run: `bunx vitest --run test/services/intelligent-router-outcome.test.ts test/services/intelligent-router.test.ts` |
| 123 | +Expected: FAIL because outcome evidence is not implemented. |
| 124 | + |
| 125 | +- [ ] **Step 3: Implement verified metrics and promotion guards** |
| 126 | + |
| 127 | +Do not record production quality or success from stop reason alone. Record unverified completions separately, require at least 20 verified samples for automatic promotion, expose sample sufficiency in routing reasons, and include total attempts plus total task cost in aggregates. |
| 128 | + |
| 129 | +- [ ] **Step 4: Run focused tests** |
| 130 | + |
| 131 | +Run: `bunx vitest --run test/services/intelligent-router-outcome.test.ts test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts` |
| 132 | +Expected: PASS. |
| 133 | + |
| 134 | +### Task 4: Complementary oracle selection |
| 135 | + |
| 136 | +**Files:** |
| 137 | +- Modify: `src/tools/oracle.ts` |
| 138 | +- Modify: `src/agent/profiles.ts` |
| 139 | +- Test: `test/tools/oracle.test.ts` |
| 140 | + |
| 141 | +**Interfaces:** |
| 142 | +- Consumes: profile oracle invocation configuration and active primary provider. |
| 143 | +- Produces: `selectComplementaryOracleModel` that prefers a different provider family and never silently falls back to an arbitrary non-reasoning model. |
| 144 | + |
| 145 | +- [ ] **Step 1: Write failing oracle-selection tests** |
| 146 | + |
| 147 | +```ts |
| 148 | +expect(selectComplementaryOracleModel(models, { primaryProvider: "openai-codex" })) |
| 149 | + .toMatchObject({ provider: "anthropic", reasoning: true }); |
| 150 | +expect(() => selectComplementaryOracleModel(nonReasoningModels, options)).toThrow(); |
| 151 | +``` |
| 152 | + |
| 153 | +- [ ] **Step 2: Run the test and verify current arbitrary fallback fails it** |
| 154 | + |
| 155 | +Run: `bunx vitest --run test/tools/oracle.test.ts` |
| 156 | +Expected: FAIL because current selection accepts the first configured model. |
| 157 | + |
| 158 | +- [ ] **Step 3: Implement complementary selection and profile defaults** |
| 159 | + |
| 160 | +Select an explicit override first, then the active profile's oracle, then a reasoning-capable model from another provider. If none exists, use a same-provider reasoning model and record the degraded choice; never choose a non-reasoning model. |
| 161 | + |
| 162 | +- [ ] **Step 4: Run focused tests** |
| 163 | + |
| 164 | +Run: `bunx vitest --run test/tools/oracle.test.ts test/agent/profiles.test.ts` |
| 165 | +Expected: PASS. |
| 166 | + |
| 167 | +### Task 5: Profile packages, telemetry, documentation, and full verification |
| 168 | + |
| 169 | +**Files:** |
| 170 | +- Create: `src/agent/profile-loader.ts` |
| 171 | +- Modify: `src/cli/commands/modes.ts` |
| 172 | +- Modify: `docs/MODELS.md` |
| 173 | +- Modify: `docs/FEATURES.md` |
| 174 | +- Create: `docs/AGENT_PROFILES.md` |
| 175 | +- Create: `test/agent/profile-loader.test.ts` |
| 176 | + |
| 177 | +**Interfaces:** |
| 178 | +- Consumes: the Task 1 profile schema. |
| 179 | +- Produces: project/user profile loading, validation, versioned telemetry fields, and CLI descriptions of resolved complete profiles. |
| 180 | + |
| 181 | +- [ ] **Step 1: Write failing loader tests** |
| 182 | + |
| 183 | +```ts |
| 184 | +expect(loadAgentProfiles(fixtureDir)).toContainEqual( |
| 185 | + expect.objectContaining({ id: "security-review-v1", level: "custom" }), |
| 186 | +); |
| 187 | +expect(() => loadAgentProfiles(invalidFixtureDir)).toThrow(/oracle.model/); |
| 188 | +``` |
| 189 | + |
| 190 | +- [ ] **Step 2: Run tests and verify loader absence** |
| 191 | + |
| 192 | +Run: `bunx vitest --run test/agent/profile-loader.test.ts` |
| 193 | +Expected: FAIL because the loader does not exist. |
| 194 | + |
| 195 | +- [ ] **Step 3: Implement declarative loading and document the schema** |
| 196 | + |
| 197 | +Load `.maestro/agent-profiles/*.yaml` and user profiles with project precedence, validate complete invocation bundles, show resolved primary/oracle/specialists/budgets in `maestro modes describe`, and document migration aliases. |
| 198 | + |
| 199 | +- [ ] **Step 4: Run repository verification** |
| 200 | + |
| 201 | +Run: `bun run bun:lint` |
| 202 | +Expected: PASS. |
| 203 | + |
| 204 | +Run: `npx nx run maestro:test --skip-nx-cache` |
| 205 | +Expected: PASS. |
| 206 | + |
| 207 | +Run: `npx nx run maestro:evals --skip-nx-cache` |
| 208 | +Expected: PASS. |
| 209 | + |
| 210 | +Run: `npx nx run maestro:build --skip-nx-cache` |
| 211 | +Expected: PASS. |
0 commit comments