Skip to content

Commit 1b8a8f8

Browse files
committed
fix(skill): provenance-gated explain_skill_activation() and invalid-JSON hardening
Changes: - skill_loader: explain_skill_activation() now reads provenance.json from skill bundle to determine governance_status (candidate_installed vs direct_write vs unknown). Includes write_targets map. - skill_candidates: record installed_via=candidate in provenance. - chat_agent: ModelDecisionEngine now raises RuntimeError for workspace tasks when JSON parsing fails after max retries, instead of silently emitting invalid_model_decision_json. Test updated. - docs: new analysis doc (dynamic-skill-generation audit), e2e test roadmap (3 sprints), updated spec/risks with 3 new failure modes, acceptance doc counter bumped. - test: regression test for RSS candidate bundle governance report. Constraint: provenance must stay beside SKILL.md, not in a separate tree Tested: pytest -q (42 passed, 1 warning), ruff check passed, ruff format --check passed, git diff --check passed Confidence: high
1 parent 9931d90 commit 1b8a8f8

10 files changed

Lines changed: 1080 additions & 16 deletions

docs/acceptance.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ acceptance flow writes the user TUI state file. In sandboxed environments, run
2121
them with permission to bind localhost ports and write the TeaAgent state
2222
directory.
2323

24-
**Current acceptance test count: `440 passed`**
24+
**Current acceptance test count: `441 passed`**
2525

2626
## Acceptance Flows
2727

docs/analysis/dynamic-skill-generation-and-long-result-audit-2026-06-05.md

Lines changed: 402 additions & 0 deletions
Large diffs are not rendered by default.

docs/modules/skills/risks.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,38 @@
3030
**File**: `skill_rag.py`
3131
**Risk**: If skill descriptions are similar, semantic routing may invoke the wrong skill.
3232
**Failure mode**: Unexpected behavior; hard to debug since skill name is not shown by default.
33+
34+
### SKL-R-007: Direct active-skill writes bypass candidate governance
35+
**File**: `skill_writer.py`, `workspace_tools/_files.py`, `skill_loader.py`
36+
**Risk**: An agent can write a `SKILL.md` directly into an active discovery
37+
directory such as `.opencode/skill/` or `.config/agent/skills/`. The loader will
38+
discover it, but no candidate artifacts, offline eval, review, or install
39+
provenance prove that it is safe or useful.
40+
**Failure mode**: The UI reports a skill as available even though it is an
41+
unreviewed direct write. Users may mistake compatibility discovery for a
42+
governed TeaAgent skill lifecycle.
43+
**Mitigation**: Treat missing candidate provenance as `direct_write` in
44+
explainability output; block or quarantine direct writes to active skill
45+
directories by default; allow reviewed installs through `skill candidate
46+
install`.
47+
48+
### SKL-R-008: Skill loaded does not prove skill used
49+
**File**: `chat_agent.py`, `skill_loader.py`
50+
**Risk**: Agent Skills can be injected into the prompt but ignored by the model,
51+
especially when the model fails to read supporting resources or emits invalid
52+
tool-decision JSON.
53+
**Failure mode**: A run claims a skill-based task succeeded, but the skill only
54+
appeared in context and did not drive the output.
55+
**Mitigation**: Add explicit `skill_activated`, `skill_resource_read`, and
56+
`skill_output_verified` audit events; add user-forced activation; compare
57+
with-skill and without-skill behavior in candidate evals.
58+
59+
### SKL-R-009: Long skill or web results lose required evidence
60+
**File**: `workspace_tools/_files.py`, `chat_agent.py`, future web/RSS tools
61+
**Risk**: Long RSS/WebSearch/skill results can be truncated or summarized before
62+
the model sees the evidence required for a faithful answer.
63+
**Failure mode**: The model writes a plausible summary from a partial preview,
64+
or creates a placeholder helper script and claims completion.
65+
**Mitigation**: Standardize a long-result envelope with preview, truncation
66+
metadata, full artifact path, content hash, cursor, and compaction-preserved
67+
source IDs. Acceptance tests must verify final output against fixture sources.

docs/modules/skills/spec.md

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@
22

33
## Purpose
44

5-
Discovers, routes, and executes skill tool modules — reusable, composable agent behaviors packaged as Python or WASM files. Skills are isolated from the main agent and run in sandboxed environments.
5+
Discovers, routes, evaluates, installs, and in some cases executes reusable
6+
agent capabilities. TeaAgent currently has two related skill surfaces:
7+
8+
1. **Agent Skills prompt packages**: directories containing `SKILL.md` and
9+
optional references/scripts/assets. These are discovered by `skill_loader.py`
10+
and injected into agent context.
11+
2. **Executable skill tools**: Python/WASM/Docker callable modules handled by
12+
`skill_executor.py` and `skill_router.py`.
13+
14+
The daily-driver skill path should prefer the governed Agent Skills candidate
15+
workflow before a skill becomes active in a project.
616

717
## Behavior Contract
818

@@ -19,9 +29,30 @@ Discovers, routes, and executes skill tool modules — reusable, composable agen
1929
5. **Result wrapping** — always returns `SkillExecutionResult(success, sandbox_type, output, error)`.
2030

2131
### Skill Loading (`skill_loader.py`)
22-
1. **Discovery** — scans `~/.teaagent/skills/` and project `.teaagent/skills/` for skill directories.
23-
2. **Manifest parsing** — reads `skill.yaml` or `manifest.json` for metadata.
24-
3. **RAG indexing** — indexes skill descriptions for semantic routing via `skill_rag.py`.
32+
1. **Discovery** — scans project and user skill directories in priority order (first match wins per skill name):
33+
- Project scope: `.config/agent/skills/`, `.claude/skills/`, `.opencode/skill/`, `.opencode/skills/`
34+
- User scope: `~/.config/agent/skills/`, `~/.claude/skills/`, `~/.config/opencode/skills/`
35+
2. **Review gate** — loads only `SKILL.md` files that pass `review_skill()`.
36+
3. **Candidate artifact gate** — installed candidate bundles with policy,
37+
provenance, cost, or contract artifacts must pass artifact validation.
38+
4. **Prompt mode** — eager mode loads full skill text; index-only mode exposes
39+
metadata without injecting full instructions.
40+
5. **Explainability**`explain_skill_activation()` reports loaded skills,
41+
shadowed paths, searched directories, token estimates, governance status, and
42+
expected project write targets.
43+
44+
### Skill Candidate Governance (`skill_candidates.py`)
45+
1. **Proposal** — creates a quarantined candidate under `.teaagent/skill-candidates/`.
46+
2. **Artifact bundle** — requires `SKILL.md`, `REFERENCE.md`,
47+
`tool_call_contract.json`, `cost_profile.json`, `interaction_policy.json`,
48+
and `provenance.json`.
49+
3. **Offline eval** — validates artifacts, size, review findings, provenance,
50+
reference content, and candidate-specific `eval_dataset.json` checks.
51+
4. **Review** — a candidate must pass review before installation.
52+
5. **Install** — project installs write to `.config/agent/skills/<name>`;
53+
personal installs require explicit personal-install attestation.
54+
6. **Provenance** — installed candidates record install scope and candidate
55+
origin so CLI/TUI can distinguish reviewed skills from direct writes.
2556

2657
### Skill Router (`skill_router.py`)
2758
1. **Semantic matching**`SkillRouter.route(query)` returns ranked skills by description similarity.
@@ -32,3 +63,8 @@ Discovers, routes, and executes skill tool modules — reusable, composable agen
3263
- Skill execution never modifies the main agent's workspace directly (isolation guarantee).
3364
- `SkillExecutionResult.success=False` always has a non-empty `error` string.
3465
- Native execution timeout is enforced (subprocess or thread limit).
66+
- A reviewed candidate install must carry provenance next to `SKILL.md`.
67+
- A loaded active skill without candidate provenance should be treated as an
68+
unmanaged/direct-write skill for explainability and review purposes.
69+
- "Skill loaded" does not imply "skill used"; runtime activation and output
70+
verification require separate audit evidence.

0 commit comments

Comments
 (0)