Skip to content

Commit d49fc5b

Browse files
chore: sync public mirror from internal
1 parent 9a8010c commit d49fc5b

36 files changed

Lines changed: 2111 additions & 63 deletions

docs/AGENT_PROFILES.md

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Agent Profiles
2+
3+
Agent profiles are versioned runtime bundles. A profile selects the primary model invocation, reasoning effort, complementary read-only Oracle, specialist invocations, fallbacks, and task budgets together. This prevents model comparisons from accidentally comparing different prompts, tools, or reasoning settings.
4+
5+
## Capability dial
6+
7+
| Level | Use it for |
8+
| --- | --- |
9+
| `low` | Bounded, obvious, reversible changes |
10+
| `medium` | Ordinary repository work with moderate uncertainty |
11+
| `high` | Ambiguous or cross-cutting work where a miss is expensive |
12+
| `ultra` | Migrations, architecture, and discovery-heavy work |
13+
14+
Existing names remain accepted during migration: `free` and `rush` map to `low`, `smart` and `custom` map to `medium`, and `frontier` maps to `ultra`.
15+
16+
## Project profiles
17+
18+
Place YAML files in `.maestro/agent-profiles/`. A complete profile looks like:
19+
20+
```yaml
21+
id: security-review-v1
22+
version: 1
23+
level: high
24+
description: Cross-provider security review
25+
primary:
26+
provider: openai-codex
27+
model: gpt-5.5
28+
reasoningEffort: high
29+
oracle:
30+
provider: anthropic
31+
model: claude-opus-4-6
32+
reasoningEffort: high
33+
readOnly: true
34+
specialists:
35+
reviewer:
36+
provider: anthropic
37+
model: claude-opus-4-6
38+
reasoningEffort: high
39+
fallbackLevels: [medium, low]
40+
budgets:
41+
maxAttempts: 2
42+
maxToolCalls: 40
43+
maxCostUsd: 5
44+
```
45+
46+
Oracle configurations must be read-only and reasoning-capable at runtime. The built-in profiles prefer a different provider family for the Oracle on higher levels.
47+
48+
## Routing evidence
49+
50+
An assistant response completing without an error is not a verified success. Automatic promotion requires at least 20 verified outcomes for the workload. Verification results, explicit user acceptance, rejection, and retries can supply outcome evidence; unverified runs remain useful for latency and cost observations only.
51+
52+
Clients can request a profile through `X-Maestro-Agent-Profile` (or the compatibility `X-Composer-Agent-Profile`) and inspect the versioned resolved profile on the routing decision.

docs/FEATURES.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Feature Guide
22

3+
Maestro provides a capability dial (`low`, `medium`, `high`, and `ultra`) backed by versioned agent profiles. Legacy `free`, `rush`, `smart`, `custom`, and `frontier` names remain available as migration aliases. See [Agent Profiles](AGENT_PROFILES.md).
4+
35
Audience: users exploring TUI/CLI flows; skim first.
46
Nav: [Docs index](README.md) · [Quickstart](QUICKSTART.md) · [Tools Reference](TOOLS_REFERENCE.md) · [Web UI](WEB_UI.md)
57

docs/MODELS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Providers & Factory Integration
22

3+
For task-level selection, prefer the `low`, `medium`, `high`, and `ultra` agent profiles over selecting a model alone. Profiles keep the model, reasoning effort, Oracle, specialists, fallbacks, and budgets reproducible as one versioned unit. See [Agent Profiles](AGENT_PROFILES.md).
4+
35
Audience: contributors/operator tweaking model registry and provider configs.
46
Nav: [Docs index](README.md) · [Quickstart](QUICKSTART.md) · [Safety](SAFETY.md) · [AI SDK](../packages/ai/README.md)
57

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# Agent Profile Routing Implementation Plan
2+
3+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4+
5+
**Goal:** Replace fragmented model-only selection with a single, versioned agent-profile routing boundary while retaining compatibility for existing Maestro modes and clients.
6+
7+
**Architecture:** A new profile module owns user-facing capability levels and complete invocation bundles. Existing mode and intelligent-router entry points adapt to that module during migration, so every surface receives the same model, reasoning, oracle, specialist, and budget configuration. Routing promotion remains conservative until verified outcome evidence meets an explicit sample threshold.
8+
9+
**Tech Stack:** TypeScript, Bun, Vitest, existing Maestro model registry and routing services.
10+
11+
## Global Constraints
12+
13+
- Preserve `smart`, `rush`, `free`, `custom`, `frontier`, and `replay` as accepted compatibility inputs.
14+
- Expose `low`, `medium`, `high`, and `ultra` as the canonical user-facing capability levels.
15+
- Keep provider/model identifiers in the model registry layer; profiles reference registered identifiers and capabilities.
16+
- No routing promotion based solely on an assistant message completing without an error.
17+
- Every behavior change follows a failing-test, passing-test cycle.
18+
19+
---
20+
21+
### Task 1: Versioned agent profiles and compatibility aliases
22+
23+
**Files:**
24+
- Create: `src/agent/profiles.ts`
25+
- Modify: `src/agent/modes.ts`
26+
- Modify: `src/agent/index.ts`
27+
- Test: `test/agent/profiles.test.ts`
28+
- Test: `test/agent/modes.test.ts`
29+
30+
**Interfaces:**
31+
- Produces: `AgentProfileLevel`, `AgentProfile`, `AGENT_PROFILES`, `resolveAgentProfile(input, provider)`, and `parseAgentProfileLevel(input)`.
32+
- Consumes: existing `ModelProvider`, `ReasoningEffort`, `SubagentType`, and model-tier resolution during the compatibility period.
33+
34+
- [ ] **Step 1: Write failing profile tests**
35+
36+
```ts
37+
expect(parseAgentProfileLevel("rush")).toBe("low");
38+
expect(parseAgentProfileLevel("smart")).toBe("medium");
39+
expect(resolveAgentProfile("high", "openai-codex")).toMatchObject({
40+
id: "high-v1",
41+
level: "high",
42+
primary: { provider: "openai-codex", reasoningEffort: "xhigh" },
43+
oracle: { provider: "anthropic" },
44+
});
45+
```
46+
47+
- [ ] **Step 2: Run the tests and verify missing exports fail**
48+
49+
Run: `bunx vitest --run test/agent/profiles.test.ts test/agent/modes.test.ts`
50+
Expected: FAIL because `src/agent/profiles.ts` and its exports do not exist.
51+
52+
- [ ] **Step 3: Implement immutable profile definitions and alias parsing**
53+
54+
Define `low-v1`, `medium-v1`, `high-v1`, and `ultra-v1` profiles. Each profile contains primary and oracle invocation configurations, specialist dispatch, fallback levels, and task budgets. Adapt `parseMode`, `getModelForMode`, and subagent dispatch through canonical levels without removing legacy inputs.
55+
56+
- [ ] **Step 4: Run focused tests**
57+
58+
Run: `bunx vitest --run test/agent/profiles.test.ts test/agent/modes.test.ts test/codex/subagent-dispatch-table.test.ts`
59+
Expected: PASS.
60+
61+
### Task 2: Route profiles through the intelligent router
62+
63+
**Files:**
64+
- Modify: `src/services/intelligent-router/types.ts`
65+
- Modify: `src/services/intelligent-router/normalize.ts`
66+
- Modify: `src/services/intelligent-router/service.ts`
67+
- Modify: `src/services/intelligent-router/recorder.ts`
68+
- Modify: `src/server/handlers/chat.ts`
69+
- Test: `test/services/intelligent-router.test.ts`
70+
- Test: `test/web/chat-handler-routing.test.ts`
71+
72+
**Interfaces:**
73+
- Consumes: `AgentProfile` and `resolveAgentProfile` from Task 1.
74+
- Produces: routing decisions containing `selectedProfile`, `fallbackProfiles`, and the resolved invocation profile while retaining `selectedModel` for wire compatibility.
75+
76+
- [ ] **Step 1: Write failing decision-contract tests**
77+
78+
```ts
79+
expect(decision.selectedProfile).toMatchObject({ level: "medium", id: "medium-v1" });
80+
expect(decision.fallbackProfiles.map(profile => profile.level)).toContain("low");
81+
```
82+
83+
- [ ] **Step 2: Run focused tests and verify the new contract is absent**
84+
85+
Run: `bunx vitest --run test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts`
86+
Expected: FAIL because routing decisions contain models only.
87+
88+
- [ ] **Step 3: Implement the profile adapter and compatibility projection**
89+
90+
Resolve a profile before candidate scoring, constrain candidates to profile policy, return the selected immutable profile, and project its primary invocation into the legacy model fields.
91+
92+
- [ ] **Step 4: Run focused tests**
93+
94+
Run: `bunx vitest --run test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts`
95+
Expected: PASS.
96+
97+
### Task 3: Verified outcome evidence and conservative promotion
98+
99+
**Files:**
100+
- Modify: `src/services/intelligent-router/types.ts`
101+
- Modify: `src/services/intelligent-router/normalize.ts`
102+
- Modify: `src/services/intelligent-router/service.ts`
103+
- Modify: `src/services/intelligent-router/recorder.ts`
104+
- Create: `src/services/intelligent-router/outcome.ts`
105+
- Test: `test/services/intelligent-router.test.ts`
106+
- Create: `test/services/intelligent-router-outcome.test.ts`
107+
108+
**Interfaces:**
109+
- Produces: `RoutingOutcomeEvidence`, `deriveRoutingOutcome(evidence)`, `MIN_VERIFIED_SAMPLES`, and confidence metadata on routing scores.
110+
- Consumes: verifier results, user acceptance/retry signals, task cost, and terminal assistant status.
111+
112+
- [ ] **Step 1: Write failing outcome tests**
113+
114+
```ts
115+
expect(deriveRoutingOutcome({ assistantCompleted: true })).toEqual({ verified: false });
116+
expect(deriveRoutingOutcome({ assistantCompleted: true, verificationPassed: true }))
117+
.toMatchObject({ verified: true, success: true });
118+
```
119+
120+
- [ ] **Step 2: Run tests and verify the outcome module is missing**
121+
122+
Run: `bunx vitest --run test/services/intelligent-router-outcome.test.ts test/services/intelligent-router.test.ts`
123+
Expected: FAIL because outcome evidence is not implemented.
124+
125+
- [ ] **Step 3: Implement verified metrics and promotion guards**
126+
127+
Do not record production quality or success from stop reason alone. Record unverified completions separately, require at least 20 verified samples for automatic promotion, expose sample sufficiency in routing reasons, and include total attempts plus total task cost in aggregates.
128+
129+
- [ ] **Step 4: Run focused tests**
130+
131+
Run: `bunx vitest --run test/services/intelligent-router-outcome.test.ts test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts`
132+
Expected: PASS.
133+
134+
### Task 4: Complementary oracle selection
135+
136+
**Files:**
137+
- Modify: `src/tools/oracle.ts`
138+
- Modify: `src/agent/profiles.ts`
139+
- Test: `test/tools/oracle.test.ts`
140+
141+
**Interfaces:**
142+
- Consumes: profile oracle invocation configuration and active primary provider.
143+
- Produces: `selectComplementaryOracleModel` that prefers a different provider family and never silently falls back to an arbitrary non-reasoning model.
144+
145+
- [ ] **Step 1: Write failing oracle-selection tests**
146+
147+
```ts
148+
expect(selectComplementaryOracleModel(models, { primaryProvider: "openai-codex" }))
149+
.toMatchObject({ provider: "anthropic", reasoning: true });
150+
expect(() => selectComplementaryOracleModel(nonReasoningModels, options)).toThrow();
151+
```
152+
153+
- [ ] **Step 2: Run the test and verify current arbitrary fallback fails it**
154+
155+
Run: `bunx vitest --run test/tools/oracle.test.ts`
156+
Expected: FAIL because current selection accepts the first configured model.
157+
158+
- [ ] **Step 3: Implement complementary selection and profile defaults**
159+
160+
Select an explicit override first, then the active profile's oracle, then a reasoning-capable model from another provider. If none exists, use a same-provider reasoning model and record the degraded choice; never choose a non-reasoning model.
161+
162+
- [ ] **Step 4: Run focused tests**
163+
164+
Run: `bunx vitest --run test/tools/oracle.test.ts test/agent/profiles.test.ts`
165+
Expected: PASS.
166+
167+
### Task 5: Profile packages, telemetry, documentation, and full verification
168+
169+
**Files:**
170+
- Create: `src/agent/profile-loader.ts`
171+
- Modify: `src/cli/commands/modes.ts`
172+
- Modify: `docs/MODELS.md`
173+
- Modify: `docs/FEATURES.md`
174+
- Create: `docs/AGENT_PROFILES.md`
175+
- Create: `test/agent/profile-loader.test.ts`
176+
177+
**Interfaces:**
178+
- Consumes: the Task 1 profile schema.
179+
- Produces: project/user profile loading, validation, versioned telemetry fields, and CLI descriptions of resolved complete profiles.
180+
181+
- [ ] **Step 1: Write failing loader tests**
182+
183+
```ts
184+
expect(loadAgentProfiles(fixtureDir)).toContainEqual(
185+
expect.objectContaining({ id: "security-review-v1", level: "custom" }),
186+
);
187+
expect(() => loadAgentProfiles(invalidFixtureDir)).toThrow(/oracle.model/);
188+
```
189+
190+
- [ ] **Step 2: Run tests and verify loader absence**
191+
192+
Run: `bunx vitest --run test/agent/profile-loader.test.ts`
193+
Expected: FAIL because the loader does not exist.
194+
195+
- [ ] **Step 3: Implement declarative loading and document the schema**
196+
197+
Load `.maestro/agent-profiles/*.yaml` and user profiles with project precedence, validate complete invocation bundles, show resolved primary/oracle/specialists/budgets in `maestro modes describe`, and document migration aliases.
198+
199+
- [ ] **Step 4: Run repository verification**
200+
201+
Run: `bun run bun:lint`
202+
Expected: PASS.
203+
204+
Run: `npx nx run maestro:test --skip-nx-cache`
205+
Expected: PASS.
206+
207+
Run: `npx nx run maestro:evals --skip-nx-cache`
208+
Expected: PASS.
209+
210+
Run: `npx nx run maestro:build --skip-nx-cache`
211+
Expected: PASS.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Revision-Aware Thread Retrieval Implementation Plan
2+
3+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4+
5+
**Goal:** Make `thread/read` able to return original, compacted, active, and superseded thread entries with enough lineage to identify later revisions and reverts.
6+
7+
**Architecture:** Extend the existing session graph projection rather than creating a second history model. The projection will retain its compacted replay window, add the complete authoritative active path and sibling revision groups, while `thread/read` exposes raw tree items only when `includeHistory: true` is requested.
8+
9+
**Tech Stack:** TypeScript, TypeBox contracts, Vitest, Bun, Nx.
10+
11+
## Global Constraints
12+
13+
- Changes originate in `evalops/maestro-internal` and reach the public repository through mirror automation.
14+
- Existing `thread/read` responses remain backward-compatible when `includeHistory` is absent.
15+
- Compaction summaries are navigation aids; original entries remain authoritative.
16+
- Production behavior is implemented only after its failing test has been observed.
17+
18+
---
19+
20+
### Task 1: Project authoritative history and revisions
21+
22+
**Files:**
23+
- Modify: `src/session/session-graph-projection.ts`
24+
- Test: `test/session/session-graph-projection.test.ts`
25+
26+
**Interfaces:**
27+
- Consumes: `SessionEntry[]` and the existing active-leaf traversal.
28+
- Produces: `authoritativeEntryIds: string[]`, `supersededEntryIds: string[]`, and `revisionGroups: Array<{ parentEntryId: string | null; childEntryIds: string[]; activeChildEntryId?: string }>` on `SessionGraphProjection`.
29+
30+
- [ ] **Step 1: Write failing projection tests** proving pre-compaction entries remain in `authoritativeEntryIds` and sibling branches appear in a revision group with only the selected child active.
31+
- [ ] **Step 2: Run** `bunx vitest --run test/session/session-graph-projection.test.ts` **and confirm failures are missing fields.**
32+
- [ ] **Step 3: Implement the minimal projection fields** by retaining the unwindowed active path and grouping tree-entry siblings by `parentId`; do not change `activeEntries` replay semantics.
33+
- [ ] **Step 4: Re-run the targeted test and confirm it passes.**
34+
- [ ] **Step 5: Commit** `test(session): specify revision-aware graph projection` and the minimal implementation atomically.
35+
36+
### Task 2: Add opt-in raw history to thread retrieval
37+
38+
**Files:**
39+
- Modify: `packages/contracts/src/maestro-app-server.ts`
40+
- Modify: `src/app-server/session-api.ts`
41+
- Test: `test/app-server/session-api.test.ts`
42+
43+
**Interfaces:**
44+
- Consumes: `thread/read` params `{ includeHistory?: boolean }` and `SessionGraphProjection` from Task 1.
45+
- Produces: optional `history: MaestroAppServerThreadItem[]` and graph lineage fields in the contract and response.
46+
47+
- [ ] **Step 1: Write a failing API test** requesting `includeHistory: true` from a compacted branched thread and asserting that old originals and superseded branch items are returned in persistence order.
48+
- [ ] **Step 2: Run** `bunx vitest --run test/app-server/session-api.test.ts -t "revision-aware history"` **and confirm the response lacks history.**
49+
- [ ] **Step 3: Extend TypeBox contracts and the response mapper** so history is omitted by default and populated from every `SessionTreeEntry` only when requested.
50+
- [ ] **Step 4: Re-run the targeted API and projection tests and confirm they pass.**
51+
- [ ] **Step 5: Regenerate contract artifacts using the repository’s existing schema-generation target, then commit** `feat(session): expose revision-aware thread history`.
52+
53+
### Task 3: Document and verify the public contract
54+
55+
**Files:**
56+
- Modify: `packages/contracts/README.md`
57+
- Modify generated schema files only through the repository generator.
58+
- Test: `test/openapi-spec.test.ts`
59+
60+
**Interfaces:**
61+
- Consumes: the additive contract from Task 2.
62+
- Produces: documented client behavior and schema regression coverage.
63+
64+
- [ ] **Step 1: Add a failing contract assertion** that revision-aware graph fields and optional history are present in generated schemas.
65+
- [ ] **Step 2: Run the contract/OpenAPI test and confirm the expected schema assertion fails before regeneration.**
66+
- [ ] **Step 3: Document compacted replay versus authoritative raw history, including the cost of `includeHistory: true`.**
67+
- [ ] **Step 4: Run** `bun run bun:lint`, `npx nx run maestro:test --skip-nx-cache`, and the touched package builds.
68+
- [ ] **Step 5: Commit** `docs(session): describe authoritative thread retrieval`, push the branch, open `[maestro] add revision-aware thread retrieval`, address review/CI, merge, and verify internal deploy plus public mirror workflows.

packages/contracts/README.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,3 +67,24 @@ assertMaestroChatRequest(payload);
6767
```
6868

6969
Consumers outside the monorepo can depend on `@evalops/contracts` once the package has been built/published; only the compiled `dist` folder is distributed.
70+
71+
## Revision-aware thread retrieval
72+
73+
`thread/read` keeps compacted reads inexpensive by default. Pass
74+
`includeTurns: true` for the replayable active window, and pass
75+
`includeHistory: true` when an inspector needs every persisted tree entry,
76+
including entries hidden by compaction or superseded by a later branch.
77+
78+
When either option is requested, `thread.graph` distinguishes:
79+
80+
- `activeEntryIds`: the compacted replay window used to resume execution.
81+
- `authoritativeEntryIds`: the complete selected branch, including original
82+
pre-compaction entries.
83+
- `supersededEntryIds`: persisted entries outside the selected branch.
84+
- `revisionGroups`: sibling revisions and the child selected by the active
85+
branch.
86+
87+
Treat compaction summaries as orientation metadata. For audits, evaluations,
88+
or decisions that depend on an earlier tool attempt, request `includeHistory`
89+
and inspect the original items plus any later sibling revisions. This opt-in
90+
response can be substantially larger than a normal thread read.

0 commit comments

Comments
 (0)