Skip to content

Commit 9b9939a

Browse files
SmithSmith
authored andcommitted
feat: Enhance drift detection and validation mechanisms
- Added tests for frontmatter model defaults validation in drift-detection.test.mjs, ensuring that frontmatter models align with role defaults and do not conflict with tier-specific overrides. - Implemented validation for payload model description semantics, ensuring that payload models are correctly defined and do not override frontmatter models. - Updated orchestration-handoff-contract.test.mjs to include checks for outer dispatch contracts, ensuring separation of outer tool-call fields and payload-level models. - Enhanced validation logic in validate.mjs to incorporate checks for frontmatter model defaults and payload model semantics. - Revised project context documentation to reflect updated model routing roles for agents. - Documented lessons learned in session outcomes regarding model routing and frontmatter defaults. - Updated orchestrator delegation protocol schema to clarify payload model descriptions and their intended use. - Introduced a new structural reference fixture in model-routing-alignment.json to ensure alignment across governance role defaults, direct-invocation frontmatter, and payload-level delegation schema.
1 parent 846e69f commit 9b9939a

15 files changed

Lines changed: 684 additions & 79 deletions

NOTES.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22

33
Repo-persistent active-objective state only. See `docs/agent-engineering/MEMORY-ARCHITECTURE.md` for the three-layer memory model; task-episodic history lives under `plans/artifacts/<task-slug>/`.
44

5-
- Active objective: Claude Code plugin implementation complete; awaiting user review.
5+
- Active objective: model-routing mismatch root-cause fix complete; awaiting user review of completion summary.
66
- Blockers: none.
7-
- Pending: user review of final summary and any follow-up Claude CLI validation when available.
7+
- Pending: user review before any commit or merge action.

Orchestrator.agent.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -166,13 +166,13 @@ Before every `agent/runSubagent` call, regardless of dispatch context, apply thi
166166
4. Pass the resolved `primary` model string as the `model` parameter to `agent/runSubagent`. Never omit `model`.
167167
5. For initial planning dispatches before any plan `complexity_tier` exists, use the target role's top-level `primary` model. For replan/planning dispatches after a plan exists, use the active plan's `complexity_tier`. Never omit `model` because tier context is missing.
168168

169-
This rule covers all dispatch paths without exception: Plan Review Gate reviewers (PlanAuditor, AssumptionVerifier, ExecutabilityVerifier), phase CodeReviewer dispatch, final CodeReviewer dispatch, needs_replan Planner dispatch, and Implementation Loop executor dispatch.
169+
This rule covers all dispatch paths without exception: Plan Review Gate reviewers (PlanAuditor, AssumptionVerifier, ExecutabilityVerifier), phase CodeReviewer dispatch, final CodeReviewer dispatch, failure-classification retry dispatch, needs_replan Planner dispatch, and Implementation Loop executor dispatch.
170170

171171
### Dispatch Tool-Call Contract (Required Fields)
172172

173173
Every `agent/runSubagent` call must include these outer tool-call fields:
174174
- **`agentName`** — the verified target-agent field (string). Placing the agent name only inside prompt prose or a delegation payload is non-compliant.
175-
- **`model`** — the resolved primary model string from the Universal Model Resolution Rule. Never omit.
175+
- **`model`** — the resolved primary model string from the Universal Model Resolution Rule, passed as the outer `model` field at the tool-call boundary. Never omit. A payload-level `model` inside the prompt/delegation payload remains useful for schema validation and audit context, but it does not by itself select the runtime model and is not a substitute for this outer field.
176176
- **Prompt/context payload** — scope, deliverables, and relevant context references.
177177

178178
#### Capable-Reviewer Model Routing

Planner.agent.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ For all other scopes, record applicability, impact, evidence source, and disposi
5050
4. Include selected skill file paths in each applicable phase's `skill_references` array.
5151
Implementation agents load referenced skills before executing phase tasks.
5252
6. Research (delegate CodeMapper-subagent/Researcher-subagent when scope is large).
53-
- **Model Resolution:** For every `agent/runSubagent` dispatch to `CodeMapper-subagent` or `Researcher-subagent`, load `governance/model-routing.json`, resolve the subagent role via the top-level `agent_role_index`, then apply `roles[role].by_tier[complexity_tier]`. If the tier entry is `{ "inherit_from": "default" }`, inherit the role's default `primary` model and default `fallbacks`; otherwise use the tier-specific `primary` and tier-specific `fallbacks` when present. Pass the resolved `primary` as the `model` parameter. Only pass a fallback list if/when `agent/runSubagent` explicitly supports one; otherwise pass only the resolved primary model string.
53+
- **Model Resolution:** For every `agent/runSubagent` dispatch to `CodeMapper-subagent` or `Researcher-subagent`, load `governance/model-routing.json`, resolve the subagent role via the top-level `agent_role_index`, then apply `roles[role].by_tier[complexity_tier]`. If the tier entry is `{ "inherit_from": "default" }`, inherit the role's default `primary` model and default `fallbacks`; otherwise use the tier-specific `primary` and tier-specific `fallbacks` when present. Every research dispatch must pass the exact target as the outer `agentName` field and the resolved `primary` as the outer `model` field. Only pass a fallback list if/when `agent/runSubagent` explicitly supports one; otherwise pass only the resolved primary model string. A payload-level `model` in the prompt/delegation payload is contract and audit context; it does not by itself select the runtime model and is not a substitute for the outer `model` field.
5454
7. Design (structured design decisions and diagram selection):
5555
- **Design Decisions Checklist** — Before proceeding to Planning (Step 8), explicitly address four dimensions:
5656
1. **Boundary changes** — Does the task change system boundaries, add new actors, or modify integration points? If no boundary changes, state "No boundary changes."

docs/agent-engineering/MODEL-ROUTING.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Model Routing
22

33
**Status:** Phase 4 spike (single-agent rollout)
4-
**File:** `governance/model-routing.json`
4+
**File:** `governance/model-routing.json` is the canonical source of truth for internal model selection.
55
**trace_id:** 7d3f5a2e-1b4c-4e9f-9a8b-2c5d8e1f3a7b
66

77
## Purpose
@@ -25,7 +25,7 @@ Decouple agent definitions from hard-pinned model strings. Today every `*.agent.
2525
The 10 roles are:
2626

2727
| Role | Consumers |
28-
|---|---|
28+
| --- | --- |
2929
| `orchestration-capable` | Orchestrator |
3030
| `capable-planner` | Planner |
3131
| `capable-implementer` | CoreImplementer, PlatformEngineer |
@@ -72,7 +72,11 @@ When Orchestrator or Planner dispatch a subagent via `agent/runSubagent`, they a
7272
1. They load `governance/model-routing.json`.
7373
2. They look up the target agent in `agent_role_index`.
7474
3. They apply the `by_tier` complexity rule to determine the required model string.
75-
4. They pass the resolved `primary` explicitly as the `model` parameter to `agent/runSubagent`, and MUST supply the verified target-agent field `agentName` at the tool-call boundary, overriding the agent's frontmatter at call time.
75+
4. They pass the resolved `primary` explicitly as the outer `model` parameter to `agent/runSubagent`, and MUST supply the verified target-agent field `agentName` at the outer tool-call boundary, overriding the agent's frontmatter at call time.
76+
77+
### Delegation Payload Contract
78+
79+
The schema definition in `schemas/orchestrator.delegation-protocol.schema.json` requires a nested payload-level `model` field in all delegation objects. These payload-level fields carry resolved model context for validation, audit, and prompt-visible traceability, but they do not by themselves enforce the runtime model override. The outer `model` parameter on the tool call is the strict enforcement point.
7680

7781
While global VS Code Copilot execution (e.g., triggering an agent directly from chat) still relies on the frontmatter fallback, all internal orchestrated pipeline dispatches strictly enforce the logical routing graph dynamically. It is important to note that offline evals do not prove live `runSubagent` execution; we distinguish structural tests and tool/API-shape evidence from real live subagent dispatch (as proven by the existing model override spike).
7882

@@ -114,17 +118,19 @@ fallback chain at every complexity tier.
114118
The prompt-driven runtime resolution module (Orchestrator/Planner dynamic lookup) is complete.
115119

116120
Remaining prerequisites for Stage D (auto-tuning observability):
121+
117122
- (a) accumulate ≥50 task telemetry entries via the NDJSON sink at `plans/artifacts/observability/`
118123
- (b) expand `governance/model-routing.json` schema with `inherit_from` targets beyond `"default"` (e.g., other roles or tier mixes)
119124

120125
### Stage C Cross-references
126+
121127
- Phase 1 spike artifact: `plans/artifacts/model-routing-stage-c/phase-1-spike-result.md`
122128
- Validation helper: `evals/drift-checks.mjs``validateByTierShape`
123129

124130
## Cost/latency tier meanings
125131

126132
| Tier | `cost_tier` | `latency_tier` |
127-
|---|---|---|
133+
| --- | --- | --- |
128134
| `low` | Inexpensive per-call; suitable for high-volume read-only or smoke tasks. | Sub-second to a few seconds typical first-token. |
129135
| `medium` | Mid-range per-call; default for implementer and review-readonly work. | A few seconds typical first-token. |
130136
| `high` | Expensive per-call; reserve for planning, deep review, or research. | `slow` — multi-second first-token; long completions expected. |

docs/tutorial-en/10-governance.md

Lines changed: 26 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Understand **which governance files regulate agent behavior** and how to change
1313
## Governance Files
1414

1515
| File | Purpose |
16-
|------|---------|
16+
| --- | --- |
1717
| `agent-grants.json` | Which agents can access which tools |
1818
| `tool-grants.json` | Which tools are exposed and their properties |
1919
| `runtime-policy.json` | Operational knobs: retry budgets, tier routing, approval policies |
@@ -45,46 +45,54 @@ The most important governance file. Orchestrator reads it as the **authoritative
4545
### Key Sections
4646

4747
**`approval_actions`** — which operations require user approval before execution:
48-
```
48+
49+
```text
4950
["delete_file", "drop_table", "git_push_force", "git_reset_hard", "run_destructive_migration"]
5051
```
5152

5253
**`review_pipeline_by_tier`** — which reviewers run for each tier:
53-
```
54+
55+
```text
5456
TRIVIAL: []
5557
SMALL: [PlanAuditor]
5658
MEDIUM: [PlanAuditor, AssumptionVerifier]
5759
LARGE: [PlanAuditor, AssumptionVerifier, ExecutabilityVerifier]
5860
```
61+
5962
(plus `code_review: always` on all tiers)
6063

6164
**`max_iterations_by_tier`** — review loop iteration cap:
62-
```
65+
66+
```text
6367
TRIVIAL: 0, SMALL: 2, MEDIUM: 5, LARGE: 5
6468
```
6569

6670
**`retry_budgets`** — cumulative retry limit per phase and ceiling on consecutive identical failures:
67-
```
71+
72+
```text
6873
per_phase: 5
6974
same_classification_ceiling: 3
7075
```
7176

7277
**`stagnation_detection`** — when to call a review loop "stuck":
73-
```
78+
79+
```text
7480
min_iterations_before_check: 3
7581
min_improvement_percentage: 5
7682
```
7783

7884
**`plan_review_gate_trigger_conditions`** — PLAN_REVIEW activation conditions:
79-
```
85+
86+
```text
8087
min_phases: 3
8188
confidence_threshold: 0.8
8289
destructive_operations: true
8390
unresolved_high_risks: true
8491
```
8592

8693
**`final_review_gate`** — optional final CodeReviewer pass:
87-
```
94+
95+
```text
8896
enabled_by_default: true / false
8997
auto_trigger_tiers: ["LARGE"]
9098
max_fix_cycles: 1
@@ -95,7 +103,7 @@ max_fix_cycles: 1
95103
Controls memory quality enforcement thresholds and content standards:
96104

97105
| Key | Value | Purpose |
98-
|-----|-------|---------|
106+
| --- | --- | --- |
99107
| `notes_md_max_lines` | 20 | NOTES.md size cap (CI-enforced by Pass 7) |
100108
| `archive_completed_plans_threshold_days` | 14 | Days before a closed plan is eligible for archival |
101109
| `archive_eligible_statuses` | `["DONE","SUPERSEDED","DEFERRED"]` | Plan statuses eligible for auto-archive |
@@ -109,17 +117,19 @@ Controls memory quality enforcement thresholds and content standards:
109117
Defines which LLM model handles which type of task. Does **not** contain API keys — only model names / capabilities.
110118

111119
**Typical sections:**
112-
- `task_type_routing` — by task type (research, implementation, review, planning).
113-
- `tier_routing` — by complexity tier.
114-
- `fallback` — default model.
115120

116-
**Who uses it:** Orchestrator and individual agents via the `model_role` field in schema outputs. See also [docs/agent-engineering/MODEL-ROUTING.md](../agent-engineering/MODEL-ROUTING.md).
121+
- `roles` — by role ID (e.g. `capable-planner`, `capable-implementer`), explicitly handling complexity tiers.
122+
- `by_tier` — explicit capability overrides for routing models dynamically by complexity tier.
123+
- `agent_role_index` — reverse-mapping index from agent name to active model role.
124+
125+
**Who uses it:** Orchestrator and Planner actively read `model-routing.json` for internal dispatch resolution (e.g. `agent/runSubagent`) to pass the resolved model as an outer tool-call parameter overriding the fallback frontmatter. See [docs/agent-engineering/MODEL-ROUTING.md](../agent-engineering/MODEL-ROUTING.md).
117126

118127
## rename-allowlist.json
119128

120129
Contains permitted file renames. Prevents accidental drift: if an agent tries to rename a file not on the allowlist — the drift check fails.
121130

122131
**Typical structure:**
132+
123133
```json
124134
[{"from": "old-name.md", "to": "new-name.md", "reason": "..."}]
125135
```
@@ -146,28 +156,28 @@ flowchart TD
146156
Schema --> Eval[cd evals && npm test]
147157
Eval -->|passed| Done[Change accepted]
148158
Eval -->|failed| Fix[Fix the issue]
149-
| What taxonomy types does repo memory use? | `runtime-policy.json → memory_hygiene.memory_content_types` |
150159
Fix --> Eval
151160
```
152161

153162
## Knowledge Location Table
154163

155164
| Question | Where to look |
156-
|----------|--------------|
165+
| --- | --- |
157166
| Which tools can an agent use? | `agent-grants.json` |
158167
| Which callers can use a tool? | `tool-grants.json` |
159168
| How many retries are allowed? | `runtime-policy.json → retry_budgets` |
160169
| When does PLAN_REVIEW fire? | `runtime-policy.json → plan_review_gate_trigger_conditions` |
161170
| Which model to use for which tier? | `model-routing.json` |
162171
| Can a file be renamed? | `rename-allowlist.json` |
163172
| What approval is required? | `runtime-policy.json → approval_actions` |
173+
| What taxonomy types does repo memory use? | `runtime-policy.json → memory_hygiene.memory_content_types` |
164174

165175
## Additional Governance Docs
166176

167177
Authoritative policy documents in `docs/agent-engineering/`:
168178

169179
| Document | Content |
170-
|----------|---------|
180+
| --- | --- |
171181
| `CLARIFICATION-POLICY.md` | When to call `vscode/askQuestions` vs NEEDS_INPUT |
172182
| `TOOL-ROUTING.md` | External tool routing rules (web/fetch, githubRepo, MCP) |
173183
| `SCORING-SPEC.md` | Quantitative scoring formula |

docs/tutorial-ru/10-governance.md

Lines changed: 24 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
## Что такое governance в ControlFlow
88

99
Governance — это **5 JSON-файлов** в `governance/`, фиксирующих:
10+
1011
1. Какие тулзы каждый агент имеет право использовать.
1112
2. Какие runtime-параметры (retry, тиры, ревью-пайплайн) применяются.
1213
3. Какие модели маршрутизируются на какие роли.
@@ -17,7 +18,7 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
1718
## Карта governance-файлов
1819

1920
| Файл | Назначение | Меняется при |
20-
|------|-----------|--------------|
21+
| --- | --- | --- |
2122
| `agent-grants.json` | Канонические права тулз агента | Изменении тулз агента |
2223
| `tool-grants.json` | Single source для frontmatter `tools:` | Том же |
2324
| `runtime-policy.json` | Operational параметры Orchestrator-а | Изменении retry/тира/ревью |
@@ -34,6 +35,7 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
3435
**Eval-проверка:** drift-checker сверяет `tools:` frontmatter каждого агента с `tool-grants.json`. Расхождение = fail.
3536

3637
**Пример (упрощённо):**
38+
3739
```json
3840
// agent-grants.json (концептуально)
3941
{
@@ -66,7 +68,7 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
6668
Определяет, какие ревьюеры активны на каждом тире:
6769

6870
| Tier | Активные ревьюеры |
69-
|------|-------------------|
71+
| --- | --- |
7072
| TRIVIAL | none |
7173
| SMALL | `["PlanAuditor"]` |
7274
| MEDIUM | `["PlanAuditor", "AssumptionVerifier"]` |
@@ -79,7 +81,7 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
7981
Бюджет PLAN_REVIEW-итераций:
8082

8183
| Tier | Max iterations |
82-
|------|---------------|
84+
| --- | --- |
8385
| TRIVIAL | — (skip) |
8486
| SMALL | 2 |
8587
| MEDIUM | 5 |
@@ -113,6 +115,7 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
113115
### plan_review_gate_trigger_conditions
114116

115117
Когда активировать PLAN_REVIEW:
118+
116119
- `min_phases` — минимальное количество фаз для триггера.
117120
- `confidence_threshold` — порог confidence (например, 0.9).
118121
- Деструктивные операции в скоупе.
@@ -133,7 +136,7 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
133136
Пороговые значения и стандарты качества памяти:
134137

135138
| Ключ | Значение | Назначение |
136-
|------|---------|-----------|
139+
| --- | --- | --- |
137140
| `notes_md_max_lines` | 20 | Максимум строк в NOTES.md (проверяется CI, Pass 7) |
138141
| `archive_completed_plans_threshold_days` | 14 | Дней до архивации закрытого плана |
139142
| `archive_eligible_statuses` | `["DONE","SUPERSEDED","DEFERRED"]` | Статусы планов, подлежащих авто-архивации |
@@ -145,34 +148,44 @@ Governance — это **5 JSON-файлов** в `governance/`, фиксирую
145148
## model-routing.json — логические роли моделей
146149

147150
Каждый агент в frontmatter имеет:
151+
148152
- `model: GPT-5.4 (copilot)` — литеральная строка модели.
149153
- `model_role: research-capable` — логическая роль.
150154

151155
`governance/model-routing.json` определяет роли:
156+
152157
```json
153158
{
154159
"research-capable": {
155-
"default": "GPT-5.4 (copilot)",
160+
"primary": "GPT-5.4 (copilot)",
161+
"fallbacks": ["GPT-5.5 (copilot)", "Claude Opus 4.7 (copilot)"],
156162
"by_tier": {
157-
"TRIVIAL": "GPT-5.4 mini (copilot)"
163+
"TRIVIAL": {
164+
"primary": "GPT-5.4 mini (copilot)",
165+
"fallbacks": ["GPT-5.4 (copilot)", "Claude Sonnet 4.6 (copilot)"],
166+
"cost_tier": "low",
167+
"latency_tier": "fast"
168+
}
158169
}
159170
}
160171
}
161172
```
162173

163174
**Зачем:** decoupling агентских файлов от хардкоженных моделей. Можно менять модель в одном месте, не трогая 13 агентов.
164175

165-
**Сейчас:** runtime читает `model:` напрямую; `model_role` существует для логического индексирования, eval-выравнивания и будущей миграции.
176+
**Сейчас:** runtime читает `model:` напрямую при внешнем вызове (от пользователя); при внутреннем вызове Orchestrator/Planner используют `governance/model-routing.json` для вычисления нужной модели и передают её явно через параметр `model` внешних вызовов (например, `agent/runSubagent`). Поля `model_role` существуют для логического индексирования, eval-выравнивания и будущей миграции.
166177

167178
**by_tier convention:**
168-
- Полный override на тир → `{"TRIVIAL": "..."}`.
169-
- Inherit → `{"TRIVIAL": "inherit_from: default"}`.
179+
180+
- Полный override на тир → объектное значение (например, `{"TRIVIAL": { "primary": "GPT-5.5 (copilot)", "fallbacks": [...] }}`).
181+
- Inherit → объект с явным указанием `{"TRIVIAL": { "inherit_from": "default" }}`.
170182

171183
См. [docs/agent-engineering/MODEL-ROUTING.md](../agent-engineering/MODEL-ROUTING.md).
172184

173185
## rename-allowlist.json
174186

175187
Drift-чекер замечает, когда файлы появляются/исчезают. Без allowlist каждый намеренный rename ломал бы eval. Allowlist:
188+
176189
```json
177190
{
178191
"renames": [
@@ -211,7 +224,7 @@ flowchart TD
211224
## Где жить какому знанию
212225

213226
| Знание | Место |
214-
|--------|-------|
227+
| --- | --- |
215228
| Что агенту разрешено делать | `agent-grants.json` / `tool-grants.json` |
216229
| Сколько раз retry | `runtime-policy.json` → retry_budgets |
217230
| Какие ревьюеры на каком тире | `runtime-policy.json` → review_pipeline_by_tier |
@@ -223,6 +236,7 @@ flowchart TD
223236
## Дополнительные governance-доки
224237

225238
В `docs/agent-engineering/` (см. [главу 03](03-agent-roster.md) — список):
239+
226240
- `PART-SPEC.md` — структура агентского файла.
227241
- `RELIABILITY-GATES.md` — verification gates (build/tests/lint).
228242
- `CLARIFICATION-POLICY.md` — 5 классов уточнений.

0 commit comments

Comments
 (0)