Skip to content

Commit e40e7b6

Browse files
os-zhuangclaude
andcommitted
chore(spec): govern ai (Agent/Tool/Skill) + allowlist mode in the liveness gate
Third category for the spec-liveness gate. Two gate capabilities make noisy categories tractable: - allowlist mode (`"mode":"allowlist"` + `"governed":[...]`) — only the named authorable schemas are checked; the rest of the category is out of scope. The json-schema categories don't map to authorable types (ai is mostly embedder/ knowledge/model DTOs), so blanket per-category classification doesn't scale. A governed name that no longer resolves is reported (catches renames). - auto-classify ADR-0010 framework provenance/lock fields (_lock*, _provenance, _packageId/Version, _lockDocsUrl, protection) as `live`, so every type's ledger doesn't repeat them. ai ledger seeded from docs/audits/2026-06-{agent,tool,skill}schema-*.md: - 63 props: 46 live / 5 experimental / 12 dead, 0 unclassified. - Agent: core + model/skills/tools/active/planning live; access/permissions/ visibility/tenantId/knowledge DEAD (access-control is a no-op — chat route hardcodes ['ai:chat','ai:agents'], ADR-0049 #1884); lifecycle/memory/guardrails/ structuredOutput experimental (aspirational autonomy). - Tool: write-only metadata — name/description/parameters/objectName live (via the parallel AIToolDefinition); category/requiresConfirmation/active/builtIn/ permissions dead; outputSchema experimental. - Skill: name/label/description/instructions/tools/triggerConditions/active live; triggerPhrases dead (no matcher), permissions dead (+ requiredPermissions drift). Stacked on #1924 (identity). GOVERNED = security, identity, ai. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 82c7438 commit e40e7b6

4 files changed

Lines changed: 301 additions & 10 deletions

File tree

.changeset/ai-liveness.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
---
3+
4+
chore(spec): govern the `ai` category (Agent/Tool/Skill) in the spec-liveness gate, and add two gate capabilities: an **allowlist mode** (`"mode":"allowlist"`) so categories dominated by protocol/engine DTOs only classify their authorable subset, and **auto-classification of ADR-0010 framework provenance/lock fields** (`_lock*`/`_provenance`/`_packageId`/`protection`). Seeded from the agent/tool/skill liveness audits. Repo-internal tooling; no package version impact.

packages/spec/liveness/README.md

Lines changed: 44 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,30 @@ so the ratchet does not flag additions to a wholesale-dead/internal subtree —
2828
additions to per-property schemas (the mixed ones like `ObjectPermission`,
2929
`PermissionSet`). Use `_schema` only for subtrees that are genuinely all-one-status.
3030

31+
## Two governance modes
32+
33+
A category's ledger picks how the gate scopes it:
34+
35+
- **default***every* authorable object schema in the category must be classified
36+
(`"_schema": "internal"` exempts non-authorable ones). Right for clean,
37+
fully-authorable categories: `security`, `identity`.
38+
- **allowlist** (`"mode": "allowlist"` + `"governed": ["Agent","Tool","Skill"]`) —
39+
only the named schemas are checked; the rest of the category is out of scope.
40+
Right for categories dominated by protocol/engine/runtime DTOs where the
41+
authorable types are a small subset: `ai` (Agent/Tool/Skill among embedder/
42+
knowledge/model DTOs). A `governed` name that no longer resolves to a schema is
43+
reported (catches renames that would silently drop coverage).
44+
45+
## Framework fields (auto-classified)
46+
47+
The ADR-0010 provenance/lock overlay fields — `_lock`, `_lockReason`, `_lockSource`,
48+
`_lockDocsUrl`, `_provenance`, `_packageId`, `_packageVersion`, `protection` — appear
49+
on every authorable type and are system-stamped, not type-specific surface. The gate
50+
auto-classifies them `live`, so ledgers don't repeat them.
51+
3152
## Files
3253

33-
- `<category>.json` — the ledger for a governed category (currently: `security`).
54+
- `<category>.json` — the ledger for a governed category (`security`, `identity`, `ai`).
3455
- `../scripts/liveness/check-liveness.mjs` — the gate. Reads the generated
3556
`packages/spec/json-schema/<category>/*.json`, resolves each authorable
3657
property's status, and exits non-zero on any UNCLASSIFIED property.
@@ -49,18 +70,32 @@ CI: `.github/workflows/spec-liveness-check.yml` runs the gate on PRs touching
4970
## Rolling out the next category
5071

5172
Governed categories are listed in `GOVERNED` at the top of `check-liveness.mjs`,
52-
rolled out **highest-risk-first**. To add one (e.g. `automation`, `data`):
73+
rolled out **highest-risk-first**. To add one (e.g. `automation`, `ui`, `data`):
5374

54-
1. `--dump <category>` to inventory its authorable properties.
75+
1. `--dump <category>` to inventory its authorable properties. **The json-schema
76+
categories do NOT map to "authorable types"** — most (`data`, `automation`, `ui`,
77+
`kernel`) are dominated by ObjectQL/engine/protocol DTOs, and some authorable
78+
types live elsewhere (Agent/Tool/Skill in `ai`, Dataset in `ui`). Decide the
79+
handful of authorable schemas and use **allowlist mode** unless the whole
80+
category is authorable.
5581
2. Seed `<category>.json` from that category's liveness audit (file:line evidence)
5682
and targeted greps for anything the audit didn't cover. **Classify only with
5783
evidence**`live` needs a cited consumer; `dead` needs a confirmed absence.
5884
3. Add the category to `GOVERNED` and confirm the gate is green.
5985

60-
## Current state — `security`
86+
## Current state
87+
88+
| Category | Mode | Properties | Notes |
89+
|---|---|---|---|
90+
| `security` | default | 93 (26 live / 1 exp / 66 dead)* | ~71% parsed-but-unenforced; enforce-or-remove worklist |
91+
| `identity` | default | 4 (3 live / 1 dead) | `Role` authorable; rest internal (SCIM/auth runtime) |
92+
| `ai` | allowlist | 63 (46 live / 5 exp / 12 dead) | Agent/Tool/Skill; `Tool` is write-only, agent access-control dead |
93+
94+
\* security numbers shift to 26 / 35 / 32 once the PolicySchema experimental
95+
markers (ADR-0049 #1882) land.
6196

62-
93 authorable properties: **66 dead, 26 live, 1 experimental.** ~71% of the
63-
authorable security surface is parsed-but-unenforced. The `dead` entries are the
64-
worklist for the security enforce-or-remove ADR — most urgently the destructive
65-
`ObjectPermission.allow{Transfer,Restore,Purge}` (ungated) and the entirely-dead
66-
`Policy` tree (password/session/`forceMfa`/network/audit).
97+
The `dead` entries are the cross-category enforce-or-remove worklist (ADR-0049).
98+
Highest-signal: the destructive `ObjectPermission.allow{Transfer,Restore,Purge}`
99+
(ungated), the entirely-dead `Policy` tree, and **agent `access`/`permissions`/
100+
`visibility`** — "who can chat with this agent" is a no-op (the chat route hardcodes
101+
`['ai:chat','ai:agents']`).

packages/spec/liveness/ai.json

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
{
2+
"category": "ai",
3+
"mode": "allowlist",
4+
"governed": [
5+
"Agent",
6+
"Tool",
7+
"Skill"
8+
],
9+
"_note": "The `ai` category is dominated by embedder/knowledge/model/runtime DTOs; the authorable metadata types are Agent, Tool, Skill (allowlist). Seeded from docs/audits/2026-06-{agent,tool,skill}schema-property-liveness.md (file:line evidence). Framework provenance/lock fields (_lock*, _provenance, _packageId, protection) are auto-classified by the gate (ADR-0010).",
10+
"schemas": {
11+
"Agent": {
12+
"_note": "agent-runtime.ts is the runtime consumer; AgentPreview is display-only (not a runtime reader).",
13+
"props": {
14+
"name": {
15+
"status": "live",
16+
"evidence": "packages/services/service-ai/src/agent-runtime.ts",
17+
"note": "core identity."
18+
},
19+
"label": {
20+
"status": "live",
21+
"note": "display/core."
22+
},
23+
"avatar": {
24+
"status": "live",
25+
"note": "display."
26+
},
27+
"role": {
28+
"status": "live",
29+
"evidence": "agent-runtime.ts",
30+
"note": "persona → system prompt; also derives chatbot picker description."
31+
},
32+
"instructions": {
33+
"status": "live",
34+
"evidence": "agent-runtime.ts",
35+
"note": "system prompt."
36+
},
37+
"model": {
38+
"status": "live",
39+
"evidence": "packages/services/service-ai/src/agent-runtime.ts:264",
40+
"note": "PARTIAL — model/temperature/maxTokens applied; model.provider + model.topP are DEAD (provider comes from the configured adapter)."
41+
},
42+
"skills": {
43+
"status": "live",
44+
"evidence": "agent-runtime.ts",
45+
"note": "Agent→Skill→Tool; contributes tools + prompt."
46+
},
47+
"tools": {
48+
"status": "live",
49+
"evidence": "agent-runtime.ts",
50+
"note": "legacy direct-tool fallback."
51+
},
52+
"active": {
53+
"status": "live",
54+
"evidence": "agent-runtime.ts",
55+
"note": "gates listing + 403 on chat."
56+
},
57+
"planning": {
58+
"status": "live",
59+
"evidence": "packages/services/service-ai/src/agent-runtime.ts",
60+
"note": "PARTIAL — only planning.maxIterations is live (3 call sites); planning.strategy/allowReplan are DEAD."
61+
},
62+
"access": {
63+
"status": "dead",
64+
"evidence": "packages/services/service-ai/src/routes/agent-routes.ts:109",
65+
"note": "'who can chat' is a no-op — the chat route hardcodes ['ai:chat','ai:agents'] regardless. Latent access-control gap (ADR-0049 #1884: enforce or remove)."
66+
},
67+
"permissions": {
68+
"status": "dead",
69+
"evidence": "agent-routes.ts:109 (hardcoded perms)",
70+
"note": "display-only; not enforced. Security gap."
71+
},
72+
"visibility": {
73+
"status": "dead",
74+
"evidence": "no runtime reader",
75+
"note": "defaults 'organization' but gates nothing."
76+
},
77+
"tenantId": {
78+
"status": "dead",
79+
"evidence": "no runtime reader",
80+
"note": "not read at runtime."
81+
},
82+
"knowledge": {
83+
"status": "dead",
84+
"evidence": "no runtime reader; AgentPreview.tsx:213 reads knowledge.sources but spec defines {topics,indexes}",
85+
"note": "shape drift; RAG is wired via service-knowledge, not agent.knowledge."
86+
},
87+
"lifecycle": {
88+
"status": "experimental",
89+
"evidence": "no runtime reader (StateMachine)",
90+
"note": "aspirational autonomy — prune or keep as roadmap (ADR-0049 enforce-or-remove)."
91+
},
92+
"memory": {
93+
"status": "experimental",
94+
"evidence": "no runtime reader",
95+
"note": "aspirational autonomy (shortTerm/longTerm/reflectionInterval)."
96+
},
97+
"guardrails": {
98+
"status": "experimental",
99+
"evidence": "no runtime reader",
100+
"note": "aspirational — real limits enforced by an unrelated quota service, not these."
101+
},
102+
"structuredOutput": {
103+
"status": "experimental",
104+
"evidence": "no runtime reader",
105+
"note": "aspirational."
106+
}
107+
}
108+
},
109+
"Tool": {
110+
"_note": "Tool metadata is WRITE-ONLY — projected one-way from the code-built ToolRegistry into metadata for Studio display (plugin.ts:813). No code reads `tool` metadata back; the runtime contract is a separate AIToolDefinition. The LIVE props are live via that same-named AIToolDefinition surface, not via metadata read-back.",
111+
"props": {
112+
"name": {
113+
"status": "live",
114+
"evidence": "packages/services/service-ai/src/adapters/vercel-adapter.ts:46",
115+
"note": "selection + LLM function key."
116+
},
117+
"label": {
118+
"status": "live",
119+
"note": "display."
120+
},
121+
"description": {
122+
"status": "live",
123+
"evidence": "packages/services/service-ai/src/adapters/vercel-adapter.ts:46",
124+
"note": "sent to the LLM."
125+
},
126+
"parameters": {
127+
"status": "live",
128+
"evidence": "packages/services/service-ai/src/adapters/vercel-adapter.ts:46",
129+
"note": "LLM function schema."
130+
},
131+
"objectName": {
132+
"status": "live",
133+
"evidence": "packages/services/service-ai/src/tools/action-tools.ts:535",
134+
"note": "action-tool dispatch."
135+
},
136+
"outputSchema": {
137+
"status": "experimental",
138+
"evidence": "tools/action-tools.ts:437 (keys folded into description only)",
139+
"note": "docstring claims output validation + chaining, but no validation exists."
140+
},
141+
"category": {
142+
"status": "dead",
143+
"evidence": "listing/preview tag only; enum drift vs AIToolDefinition free-string",
144+
"note": "not sent to the model."
145+
},
146+
"requiresConfirmation": {
147+
"status": "dead",
148+
"evidence": "tools/action-tools.ts:239 (HITL re-derives from action.ai.requiresConfirmation)",
149+
"note": "never read off the def — redundant mirror."
150+
},
151+
"permissions": {
152+
"status": "dead",
153+
"evidence": "tool.form.ts only; not on AIToolDefinition, no consumer"
154+
},
155+
"active": {
156+
"status": "dead",
157+
"evidence": "never set by any registration path"
158+
},
159+
"builtIn": {
160+
"status": "dead",
161+
"evidence": "never set by any registration path"
162+
}
163+
}
164+
},
165+
"Skill": {
166+
"_note": "skill-registry.ts + agent-runtime.ts are the runtime consumers.",
167+
"props": {
168+
"name": {
169+
"status": "live",
170+
"evidence": "packages/services/service-ai/src/skill-registry.ts"
171+
},
172+
"label": {
173+
"status": "live",
174+
"evidence": "packages/services/service-ai/src/skill-registry.ts:247",
175+
"note": "injected into the agent system prompt."
176+
},
177+
"description": {
178+
"status": "live",
179+
"evidence": "packages/services/service-ai/src/skill-registry.ts:247",
180+
"note": "injected into prompt."
181+
},
182+
"instructions": {
183+
"status": "live",
184+
"evidence": "packages/services/service-ai/src/skill-registry.ts:247",
185+
"note": "injected into prompt."
186+
},
187+
"tools": {
188+
"status": "live",
189+
"evidence": "packages/services/service-ai/src/skill-registry.ts:206",
190+
"note": "tool-contribution path incl. action_* wildcard expand."
191+
},
192+
"triggerConditions": {
193+
"status": "live",
194+
"evidence": "packages/services/service-ai/src/skill-registry.ts:153",
195+
"note": "THE activation gate — AND of {field,operator,value}."
196+
},
197+
"active": {
198+
"status": "live",
199+
"evidence": "packages/services/service-ai/src/skill-registry.ts:93",
200+
"note": "inactive skills dropped."
201+
},
202+
"triggerPhrases": {
203+
"status": "dead",
204+
"evidence": "no intent/phrase matcher; only populates the slash-command palette summary (toSummary)",
205+
"note": "schema frames them as activating the skill, but nothing matches user text — intent routing unimplemented."
206+
},
207+
"permissions": {
208+
"status": "dead",
209+
"evidence": "no code restricts a skill by skill.permissions; naming drift (preview calls it requiredPermissions)",
210+
"note": "enforce-or-remove + fix naming (ADR-0049)."
211+
}
212+
}
213+
}
214+
}
215+
}

packages/spec/scripts/liveness/check-liveness.mjs

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ const ledgerRoot = join(specRoot, 'liveness');
3838

3939
// Categories whose authorable schemas must be fully classified. Extend
4040
// highest-risk-first as each category's ledger is seeded from its audit.
41-
const GOVERNED = ['security', 'identity'];
41+
const GOVERNED = ['security', 'identity', 'ai'];
4242

4343
const args = process.argv.slice(2);
4444
const asJson = args.includes('--json');
@@ -50,6 +50,16 @@ const MARKER_RE = {
5050
planned: /\[planned|not yet implemented|coming soon/i,
5151
};
5252

53+
// Framework-managed provenance/lock overlay fields (ADR-0010 + metadata
54+
// provenance stamping). These appear on EVERY authorable metadata type, are
55+
// system-stamped (not type-specific authorable surface), and are consumed by the
56+
// overlay/lock loader. Auto-classified `live` so each category's ledger doesn't
57+
// have to repeat them.
58+
const FRAMEWORK_FIELDS = new Set([
59+
'_lock', '_lockReason', '_lockSource', '_lockDocsUrl',
60+
'_provenance', '_packageId', '_packageVersion', 'protection',
61+
]);
62+
5363
function loadSchemas(category) {
5464
const dir = join(schemaRoot, category);
5565
if (!existsSync(dir)) return [];
@@ -98,13 +108,30 @@ const report = { categories: {}, totals: { classified: 0, unclassified: 0, bySta
98108

99109
for (const category of GOVERNED) {
100110
const ledger = loadLedger(category) || { schemas: {} };
111+
// Two governance modes:
112+
// - default: every authorable object schema in the category must be classified
113+
// (`internal` exempts non-authorable ones). Right for clean, fully-authorable
114+
// categories (security, identity).
115+
// - allowlist (`"mode":"allowlist"` + `"governed":[...]`): only the named schemas
116+
// are checked; the rest of the category is out of scope. Right for categories
117+
// dominated by protocol/engine DTOs where the authorable types are a small
118+
// subset (ai: Agent/Tool/Skill among embedder/knowledge/model DTOs).
119+
const allowlist = ledger.mode === 'allowlist' ? new Set(ledger.governed || []) : null;
101120
const cat = { classified: 0, unclassified: 0, byStatus: {} };
121+
const seen = new Set();
102122
for (const { name, schema } of loadSchemas(category)) {
123+
if (allowlist && !allowlist.has(name)) continue; // allowlist: skip un-governed schemas
124+
seen.add(name);
103125
const props = topProps(schema);
104126
if (props.length === 0) continue; // enums / scalars carry no authorable props
105127
const entry = ledger.schemas[name] || {};
106128
if (entry._schema === 'internal') continue; // exempt: not authorable
107129
for (const { key, description } of props) {
130+
if (FRAMEWORK_FIELDS.has(key)) { // ADR-0010 provenance/lock — auto-live
131+
cat.classified++; cat.byStatus.live = (cat.byStatus.live || 0) + 1;
132+
report.totals.byStatus.live = (report.totals.byStatus.live || 0) + 1;
133+
continue;
134+
}
108135
const led = entry.props && entry.props[key];
109136
const status = (led && led.status) || entry._schema || markerStatus(description);
110137
if (!status) {
@@ -122,6 +149,16 @@ for (const category of GOVERNED) {
122149
}
123150
}
124151
}
152+
// allowlist sanity: a governed schema name that no longer exists is a typo or a
153+
// rename that silently drops coverage — surface it as unclassified.
154+
if (allowlist) {
155+
for (const want of allowlist) {
156+
if (!seen.has(want)) {
157+
cat.unclassified++;
158+
report.unclassified.push(`${category}/${want} (governed schema not found — renamed or removed?)`);
159+
}
160+
}
161+
}
125162
report.categories[category] = cat;
126163
report.totals.classified += cat.classified;
127164
report.totals.unclassified += cat.unclassified;

0 commit comments

Comments
 (0)