Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/AGENT_PROFILES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Agent Profiles

Agent profiles are versioned runtime bundles. A profile selects the primary model invocation, reasoning effort, complementary read-only Oracle, specialist invocations, fallbacks, and task budgets together. This prevents model comparisons from accidentally comparing different prompts, tools, or reasoning settings.

## Capability dial

| Level | Use it for |
| --- | --- |
| `low` | Bounded, obvious, reversible changes |
| `medium` | Ordinary repository work with moderate uncertainty |
| `high` | Ambiguous or cross-cutting work where a miss is expensive |
| `ultra` | Migrations, architecture, and discovery-heavy work |

Existing names remain accepted during migration: `free` and `rush` map to `low`, `smart` and `custom` map to `medium`, and `frontier` maps to `ultra`.

## Project profiles

Place YAML files in `.maestro/agent-profiles/`. A complete profile looks like:

```yaml
id: security-review-v1
version: 1
level: high
description: Cross-provider security review
primary:
provider: openai-codex
model: gpt-5.5
reasoningEffort: high
oracle:
provider: anthropic
model: claude-opus-4-6
reasoningEffort: high
readOnly: true
specialists:
reviewer:
provider: anthropic
model: claude-opus-4-6
reasoningEffort: high
fallbackLevels: [medium, low]
budgets:
maxAttempts: 2
maxToolCalls: 40
maxCostUsd: 5
```

Oracle configurations must be read-only and reasoning-capable at runtime. The built-in profiles prefer a different provider family for the Oracle on higher levels.

## Routing evidence

An assistant response completing without an error is not a verified success. Automatic promotion requires at least 20 verified outcomes for the workload. Verification results, explicit user acceptance, rejection, and retries can supply outcome evidence; unverified runs remain useful for latency and cost observations only.

Clients can request a profile through `X-Maestro-Agent-Profile` (or the compatibility `X-Composer-Agent-Profile`) and inspect the versioned resolved profile on the routing decision.
2 changes: 2 additions & 0 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Feature Guide

Maestro provides a capability dial (`low`, `medium`, `high`, and `ultra`) backed by versioned agent profiles. Legacy `free`, `rush`, `smart`, `custom`, and `frontier` names remain available as migration aliases. See [Agent Profiles](AGENT_PROFILES.md).

Audience: users exploring TUI/CLI flows; skim first.
Nav: [Docs index](README.md) · [Quickstart](QUICKSTART.md) · [Tools Reference](TOOLS_REFERENCE.md) · [Web UI](WEB_UI.md)

Expand Down
2 changes: 2 additions & 0 deletions docs/MODELS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Providers & Factory Integration

For task-level selection, prefer the `low`, `medium`, `high`, and `ultra` agent profiles over selecting a model alone. Profiles keep the model, reasoning effort, Oracle, specialists, fallbacks, and budgets reproducible as one versioned unit. See [Agent Profiles](AGENT_PROFILES.md).

Audience: contributors/operator tweaking model registry and provider configs.
Nav: [Docs index](README.md) · [Quickstart](QUICKSTART.md) · [Safety](SAFETY.md) · [AI SDK](../packages/ai/README.md)

Expand Down
211 changes: 211 additions & 0 deletions docs/superpowers/plans/2026-07-13-agent-profile-routing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
# Agent Profile Routing Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace fragmented model-only selection with a single, versioned agent-profile routing boundary while retaining compatibility for existing Maestro modes and clients.

**Architecture:** A new profile module owns user-facing capability levels and complete invocation bundles. Existing mode and intelligent-router entry points adapt to that module during migration, so every surface receives the same model, reasoning, oracle, specialist, and budget configuration. Routing promotion remains conservative until verified outcome evidence meets an explicit sample threshold.

**Tech Stack:** TypeScript, Bun, Vitest, existing Maestro model registry and routing services.

## Global Constraints

- Preserve `smart`, `rush`, `free`, `custom`, `frontier`, and `replay` as accepted compatibility inputs.
- Expose `low`, `medium`, `high`, and `ultra` as the canonical user-facing capability levels.
- Keep provider/model identifiers in the model registry layer; profiles reference registered identifiers and capabilities.
- No routing promotion based solely on an assistant message completing without an error.
- Every behavior change follows a failing-test, passing-test cycle.

---

### Task 1: Versioned agent profiles and compatibility aliases

**Files:**
- Create: `src/agent/profiles.ts`
- Modify: `src/agent/modes.ts`
- Modify: `src/agent/index.ts`
- Test: `test/agent/profiles.test.ts`
- Test: `test/agent/modes.test.ts`

**Interfaces:**
- Produces: `AgentProfileLevel`, `AgentProfile`, `AGENT_PROFILES`, `resolveAgentProfile(input, provider)`, and `parseAgentProfileLevel(input)`.
- Consumes: existing `ModelProvider`, `ReasoningEffort`, `SubagentType`, and model-tier resolution during the compatibility period.

- [ ] **Step 1: Write failing profile tests**

```ts
expect(parseAgentProfileLevel("rush")).toBe("low");
expect(parseAgentProfileLevel("smart")).toBe("medium");
expect(resolveAgentProfile("high", "openai-codex")).toMatchObject({
id: "high-v1",
level: "high",
primary: { provider: "openai-codex", reasoningEffort: "xhigh" },
oracle: { provider: "anthropic" },
});
```

- [ ] **Step 2: Run the tests and verify missing exports fail**

Run: `bunx vitest --run test/agent/profiles.test.ts test/agent/modes.test.ts`
Expected: FAIL because `src/agent/profiles.ts` and its exports do not exist.

- [ ] **Step 3: Implement immutable profile definitions and alias parsing**

Define `low-v1`, `medium-v1`, `high-v1`, and `ultra-v1` profiles. Each profile contains primary and oracle invocation configurations, specialist dispatch, fallback levels, and task budgets. Adapt `parseMode`, `getModelForMode`, and subagent dispatch through canonical levels without removing legacy inputs.

- [ ] **Step 4: Run focused tests**

Run: `bunx vitest --run test/agent/profiles.test.ts test/agent/modes.test.ts test/codex/subagent-dispatch-table.test.ts`
Expected: PASS.

### Task 2: Route profiles through the intelligent router

**Files:**
- Modify: `src/services/intelligent-router/types.ts`
- Modify: `src/services/intelligent-router/normalize.ts`
- Modify: `src/services/intelligent-router/service.ts`
- Modify: `src/services/intelligent-router/recorder.ts`
- Modify: `src/server/handlers/chat.ts`
- Test: `test/services/intelligent-router.test.ts`
- Test: `test/web/chat-handler-routing.test.ts`

**Interfaces:**
- Consumes: `AgentProfile` and `resolveAgentProfile` from Task 1.
- Produces: routing decisions containing `selectedProfile`, `fallbackProfiles`, and the resolved invocation profile while retaining `selectedModel` for wire compatibility.

- [ ] **Step 1: Write failing decision-contract tests**

```ts
expect(decision.selectedProfile).toMatchObject({ level: "medium", id: "medium-v1" });
expect(decision.fallbackProfiles.map(profile => profile.level)).toContain("low");
```

- [ ] **Step 2: Run focused tests and verify the new contract is absent**

Run: `bunx vitest --run test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts`
Expected: FAIL because routing decisions contain models only.

- [ ] **Step 3: Implement the profile adapter and compatibility projection**

Resolve a profile before candidate scoring, constrain candidates to profile policy, return the selected immutable profile, and project its primary invocation into the legacy model fields.

- [ ] **Step 4: Run focused tests**

Run: `bunx vitest --run test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts`
Expected: PASS.

### Task 3: Verified outcome evidence and conservative promotion

**Files:**
- Modify: `src/services/intelligent-router/types.ts`
- Modify: `src/services/intelligent-router/normalize.ts`
- Modify: `src/services/intelligent-router/service.ts`
- Modify: `src/services/intelligent-router/recorder.ts`
- Create: `src/services/intelligent-router/outcome.ts`
- Test: `test/services/intelligent-router.test.ts`
- Create: `test/services/intelligent-router-outcome.test.ts`

**Interfaces:**
- Produces: `RoutingOutcomeEvidence`, `deriveRoutingOutcome(evidence)`, `MIN_VERIFIED_SAMPLES`, and confidence metadata on routing scores.
- Consumes: verifier results, user acceptance/retry signals, task cost, and terminal assistant status.

- [ ] **Step 1: Write failing outcome tests**

```ts
expect(deriveRoutingOutcome({ assistantCompleted: true })).toEqual({ verified: false });
expect(deriveRoutingOutcome({ assistantCompleted: true, verificationPassed: true }))
.toMatchObject({ verified: true, success: true });
```

- [ ] **Step 2: Run tests and verify the outcome module is missing**

Run: `bunx vitest --run test/services/intelligent-router-outcome.test.ts test/services/intelligent-router.test.ts`
Expected: FAIL because outcome evidence is not implemented.

- [ ] **Step 3: Implement verified metrics and promotion guards**

Do not record production quality or success from stop reason alone. Record unverified completions separately, require at least 20 verified samples for automatic promotion, expose sample sufficiency in routing reasons, and include total attempts plus total task cost in aggregates.

- [ ] **Step 4: Run focused tests**

Run: `bunx vitest --run test/services/intelligent-router-outcome.test.ts test/services/intelligent-router.test.ts test/web/chat-handler-routing.test.ts`
Expected: PASS.

### Task 4: Complementary oracle selection

**Files:**
- Modify: `src/tools/oracle.ts`
- Modify: `src/agent/profiles.ts`
- Test: `test/tools/oracle.test.ts`

**Interfaces:**
- Consumes: profile oracle invocation configuration and active primary provider.
- Produces: `selectComplementaryOracleModel` that prefers a different provider family and never silently falls back to an arbitrary non-reasoning model.

- [ ] **Step 1: Write failing oracle-selection tests**

```ts
expect(selectComplementaryOracleModel(models, { primaryProvider: "openai-codex" }))
.toMatchObject({ provider: "anthropic", reasoning: true });
expect(() => selectComplementaryOracleModel(nonReasoningModels, options)).toThrow();
```

- [ ] **Step 2: Run the test and verify current arbitrary fallback fails it**

Run: `bunx vitest --run test/tools/oracle.test.ts`
Expected: FAIL because current selection accepts the first configured model.

- [ ] **Step 3: Implement complementary selection and profile defaults**

Select an explicit override first, then the active profile's oracle, then a reasoning-capable model from another provider. If none exists, use a same-provider reasoning model and record the degraded choice; never choose a non-reasoning model.

- [ ] **Step 4: Run focused tests**

Run: `bunx vitest --run test/tools/oracle.test.ts test/agent/profiles.test.ts`
Expected: PASS.

### Task 5: Profile packages, telemetry, documentation, and full verification

**Files:**
- Create: `src/agent/profile-loader.ts`
- Modify: `src/cli/commands/modes.ts`
- Modify: `docs/MODELS.md`
- Modify: `docs/FEATURES.md`
- Create: `docs/AGENT_PROFILES.md`
- Create: `test/agent/profile-loader.test.ts`

**Interfaces:**
- Consumes: the Task 1 profile schema.
- Produces: project/user profile loading, validation, versioned telemetry fields, and CLI descriptions of resolved complete profiles.

- [ ] **Step 1: Write failing loader tests**

```ts
expect(loadAgentProfiles(fixtureDir)).toContainEqual(
expect.objectContaining({ id: "security-review-v1", level: "custom" }),
);
expect(() => loadAgentProfiles(invalidFixtureDir)).toThrow(/oracle.model/);
```

- [ ] **Step 2: Run tests and verify loader absence**

Run: `bunx vitest --run test/agent/profile-loader.test.ts`
Expected: FAIL because the loader does not exist.

- [ ] **Step 3: Implement declarative loading and document the schema**

Load `.maestro/agent-profiles/*.yaml` and user profiles with project precedence, validate complete invocation bundles, show resolved primary/oracle/specialists/budgets in `maestro modes describe`, and document migration aliases.

- [ ] **Step 4: Run repository verification**

Run: `bun run bun:lint`
Expected: PASS.

Run: `npx nx run maestro:test --skip-nx-cache`
Expected: PASS.

Run: `npx nx run maestro:evals --skip-nx-cache`
Expected: PASS.

Run: `npx nx run maestro:build --skip-nx-cache`
Expected: PASS.
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Revision-Aware Thread Retrieval Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make `thread/read` able to return original, compacted, active, and superseded thread entries with enough lineage to identify later revisions and reverts.

**Architecture:** Extend the existing session graph projection rather than creating a second history model. The projection will retain its compacted replay window, add the complete authoritative active path and sibling revision groups, while `thread/read` exposes raw tree items only when `includeHistory: true` is requested.

**Tech Stack:** TypeScript, TypeBox contracts, Vitest, Bun, Nx.

## Global Constraints

- Changes originate in `evalops/maestro-internal` and reach the public repository through mirror automation.
- Existing `thread/read` responses remain backward-compatible when `includeHistory` is absent.
- Compaction summaries are navigation aids; original entries remain authoritative.
- Production behavior is implemented only after its failing test has been observed.

---

### Task 1: Project authoritative history and revisions

**Files:**
- Modify: `src/session/session-graph-projection.ts`
- Test: `test/session/session-graph-projection.test.ts`

**Interfaces:**
- Consumes: `SessionEntry[]` and the existing active-leaf traversal.
- Produces: `authoritativeEntryIds: string[]`, `supersededEntryIds: string[]`, and `revisionGroups: Array<{ parentEntryId: string | null; childEntryIds: string[]; activeChildEntryId?: string }>` on `SessionGraphProjection`.

- [ ] **Step 1: Write failing projection tests** proving pre-compaction entries remain in `authoritativeEntryIds` and sibling branches appear in a revision group with only the selected child active.
- [ ] **Step 2: Run** `bunx vitest --run test/session/session-graph-projection.test.ts` **and confirm failures are missing fields.**
- [ ] **Step 3: Implement the minimal projection fields** by retaining the unwindowed active path and grouping tree-entry siblings by `parentId`; do not change `activeEntries` replay semantics.
- [ ] **Step 4: Re-run the targeted test and confirm it passes.**
- [ ] **Step 5: Commit** `test(session): specify revision-aware graph projection` and the minimal implementation atomically.

### Task 2: Add opt-in raw history to thread retrieval

**Files:**
- Modify: `packages/contracts/src/maestro-app-server.ts`
- Modify: `src/app-server/session-api.ts`
- Test: `test/app-server/session-api.test.ts`

**Interfaces:**
- Consumes: `thread/read` params `{ includeHistory?: boolean }` and `SessionGraphProjection` from Task 1.
- Produces: optional `history: MaestroAppServerThreadItem[]` and graph lineage fields in the contract and response.

- [ ] **Step 1: Write a failing API test** requesting `includeHistory: true` from a compacted branched thread and asserting that old originals and superseded branch items are returned in persistence order.
- [ ] **Step 2: Run** `bunx vitest --run test/app-server/session-api.test.ts -t "revision-aware history"` **and confirm the response lacks history.**
- [ ] **Step 3: Extend TypeBox contracts and the response mapper** so history is omitted by default and populated from every `SessionTreeEntry` only when requested.
- [ ] **Step 4: Re-run the targeted API and projection tests and confirm they pass.**
- [ ] **Step 5: Regenerate contract artifacts using the repository’s existing schema-generation target, then commit** `feat(session): expose revision-aware thread history`.

### Task 3: Document and verify the public contract

**Files:**
- Modify: `packages/contracts/README.md`
- Modify generated schema files only through the repository generator.
- Test: `test/openapi-spec.test.ts`

**Interfaces:**
- Consumes: the additive contract from Task 2.
- Produces: documented client behavior and schema regression coverage.

- [ ] **Step 1: Add a failing contract assertion** that revision-aware graph fields and optional history are present in generated schemas.
- [ ] **Step 2: Run the contract/OpenAPI test and confirm the expected schema assertion fails before regeneration.**
- [ ] **Step 3: Document compacted replay versus authoritative raw history, including the cost of `includeHistory: true`.**
- [ ] **Step 4: Run** `bun run bun:lint`, `npx nx run maestro:test --skip-nx-cache`, and the touched package builds.
- [ ] **Step 5: Commit** `docs(session): describe authoritative thread retrieval`, push the branch, open `[maestro] add revision-aware thread retrieval`, address review/CI, merge, and verify internal deploy plus public mirror workflows.
21 changes: 21 additions & 0 deletions packages/contracts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,24 @@ assertMaestroChatRequest(payload);
```

Consumers outside the monorepo can depend on `@evalops/contracts` once the package has been built/published; only the compiled `dist` folder is distributed.

## Revision-aware thread retrieval

`thread/read` keeps compacted reads inexpensive by default. Pass
`includeTurns: true` for the replayable active window, and pass
`includeHistory: true` when an inspector needs every persisted tree entry,
including entries hidden by compaction or superseded by a later branch.

When either option is requested, `thread.graph` distinguishes:

- `activeEntryIds`: the compacted replay window used to resume execution.
- `authoritativeEntryIds`: the complete selected branch, including original
pre-compaction entries.
- `supersededEntryIds`: persisted entries outside the selected branch.
- `revisionGroups`: sibling revisions and the child selected by the active
branch.

Treat compaction summaries as orientation metadata. For audits, evaluations,
or decisions that depend on an earlier tool attempt, request `includeHistory`
and inspect the original items plus any later sibling revisions. This opt-in
response can be substantially larger than a normal thread read.
Loading
Loading