Merge pull request #58 from JRS1986/feature/governance-and-eval

Priorities 3, 4 & 7: MCP/skill governance, agent permissions, readiness benchmark

Author: JRS1986

CHANGELOG.md

@@ -9,6 +9,36 @@ aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Added
 
+- **Machine-readable agent permissions (`coding-scaffold permissions write`).** Writes
+  `.coding-scaffold/agent-permissions.json` with the canonical permission shape — filesystem
+  read/write/deny patterns, shell-command allowlist + approval-required list, network defaults,
+  and MCP defaults. Idempotent; pass `--force` to regenerate. The `shell.allowed` list is
+  lightly project-aware (Python projects get `pytest`/`ruff`, Node projects get `npm test`,
+  etc.). See [Security / Machine-Readable Permissions](docs/wiki/Security.md#machine-readable-permissions-artifact).
+- **MCP governance (`coding-scaffold mcp policy init` / `scan` / `snapshot` / `diff`).**
+  Reviewable team policy at `.coding-scaffold/mcp-policy.json`. The scanner inspects known
+  MCP-config locations (`opencode.json`, `.claude/settings.json`) and flags remote servers,
+  unpinned npm packages, risky launchers (curl-pipe-shell, sudo, bash -c), broad filesystem
+  access (root / home), unapproved servers, denied servers, and review-required capabilities.
+  `snapshot` + `diff` together let you commit a known-good state and detect drift in CI.
+  `mcp diff` exits non-zero when anything changed. No commands are executed; no network
+  calls. See [Security / MCP Governance](docs/wiki/Security.md#mcp-governance).
+- **Reviewable skill packs (`coding-scaffold skills new` / `lint` / `approve` / `export`).**
+  Each skill lives at `.coding-scaffold/skills/<name>/` with `SKILL.md`, `manifest.json`,
+  optional `scripts/` and `tests/`, and a `CHECKSUM` file frozen at approval time.
+  `skills lint` flags broad "always use this" language, hidden-instruction phrases,
+  undeclared capabilities (network / shell / credential), missing required sections, invalid
+  manifest fields, placeholder owners, and drift since the recorded checksum.
+  `skills export` bundles a skill into a `tar.gz` for sharing. See
+  [Security / Skill Pack Governance](docs/wiki/Security.md#skill-pack-governance).
+- **Readiness benchmark (`coding-scaffold eval init` / `run` / `report`).** Deterministic
+  checks that score how prepared a repo is for safe agentic coding: detectable build/test/lint
+  signals, agent instructions present, policy pack present, non-empty deny list, PR template
+  present, MCP policy present when MCP is detected, session-trace location available, context
+  lint clean, and context budget under the limit. No model-intelligence benchmark — purely
+  observable artifacts. `eval run` writes `.coding-scaffold/eval-report.json` and exits
+  non-zero when any check fails, so it can gate CI. See
+  [Team Rollout / Readiness Benchmark](docs/wiki/Team-Rollout.md#readiness-benchmark).
 - **Agent-context linter (`coding-scaffold context lint`).** Deterministic, heuristic-only
   checker for `AGENTS.md`, `CLAUDE.md`, `llms.txt`, and the `.coding-scaffold/` guidance
   docs. Flags vague rules without verifiers, dangerous shell recommendations

docs/wiki/Security.md

@@ -118,6 +118,80 @@ Auditors should expect:
 - Sensitive runtime values (API keys, Azure endpoints/deployments) are not present in committed
   config; they live in `.env.local` and are resolved at agent-start time.
 
+## Machine-Readable Permissions Artifact
+
+`coding-scaffold permissions write` generates `.coding-scaffold/agent-permissions.json` — a
+canonical, machine-readable declaration of what the agent may touch:
+
+```json
+{
+  "filesystem": {
+    "read": ["src/**", "tests/**", "docs/**", "..."],
+    "write": ["src/**", "tests/**"],
+    "deny": [".env", ".env.*", "secrets/**", "build/**", "dist/**", "..."]
+  },
+  "shell": {
+    "allowed": ["pytest", "ruff", "..."],
+    "requires_approval": ["rm", "git push", "curl", "wget", "docker", "kubectl"]
+  },
+  "network": "disabled_by_default",
+  "mcp": {
+    "remote_servers": "requires_approval",
+    "unapproved_servers": "deny"
+  }
+}
+```
+
+The file is reviewable guidance, not enforcement. Coding tools (Claude Code, Codex, OpenCode,
+…) read it as authoritative configuration; actual enforcement is the tool's responsibility.
+The file is idempotent — re-running `permissions write` skips it unless `--force` is passed.
+
+## MCP Governance
+
+The scaffold ships a lightweight review helper for MCP servers (not a full security scanner):
+
+```bash
+coding-scaffold mcp policy init    # write .coding-scaffold/mcp-policy.json
+coding-scaffold mcp scan           # report findings against the policy
+coding-scaffold mcp snapshot       # checkpoint the current server set
+coding-scaffold mcp diff           # report changes since the last snapshot
+```
+
+`mcp scan` flags: remote servers, unpinned npm packages, risky launchers (curl-pipe-shell,
+sudo, bash -c), broad filesystem access (root or home), unapproved servers (when the policy
+has a non-empty approved list), denied servers (as errors), and review-required capabilities.
+
+`mcp snapshot` + `mcp diff` together let you commit a known-good state and detect drift
+in CI. `mcp diff` exits non-zero when anything changed since the snapshot, so it can gate
+merges that quietly added a server.
+
+## Skill Pack Governance
+
+Reviewable skills live under `.coding-scaffold/skills/<name>/`:
+
+```
+SKILL.md       human-readable contract
+manifest.json  machine-readable metadata (owner, version, risk_level, capabilities)
+scripts/       optional helpers
+tests/         optional verification scripts
+README.md      usage and examples
+CHECKSUM       sha256(SKILL.md || manifest.json) frozen at approval
+```
+
+`coding-scaffold skills lint` checks every skill for:
+
+- broad "always use this" language in SKILL.md
+- hidden-instruction phrases (`do not tell the user`, `ignore the system prompt`, …)
+- references to network / shell / credential capabilities that aren't declared in the
+  `Capabilities required` section
+- missing `When to use` or `Verification` sections
+- missing required manifest fields (name, version, owner, risk_level, description)
+- invalid `risk_level` (must be one of low / medium / high / critical)
+- placeholder owners (`<your-handle>`)
+- drift since the recorded CHECKSUM (re-run `skills approve <name>` after legitimate edits)
+
+`skills export <name>` produces a sharable `tar.gz` for inter-team reuse.
+
 ## What this scaffold does not promise
 
 - It does not sandbox agent execution.

docs/wiki/Team-Rollout.md

@@ -195,6 +195,37 @@ coding-scaffold team doctor --target .
 When the manifest changes, every developer runs `team sync` and reviews the diff before merging
 imports into their working knowledge.
 
+## Readiness Benchmark
+
+Once the scaffold is in place, run the readiness benchmark to check whether the repo is
+prepared for safe agentic coding:
+
+```bash
+coding-scaffold eval init    # optional: write .coding-scaffold/eval-config.json
+coding-scaffold eval run     # execute all enabled checks
+coding-scaffold eval report --cached    # re-print the most recent report
+```
+
+The benchmark runs deterministic checks only — no shell execution, no LLM calls. It looks for:
+
+- a detectable build signal (`pyproject.toml`, `package.json`, `Cargo.toml`, …)
+- a test command mentioned in the agent-context files
+- a lint configuration file (or a `[tool.ruff]` / `[tool.black]` / `[tool.mypy]` section)
+- agent instructions (`AGENTS.md`, `CLAUDE.md`, or `llms.txt`)
+- a policy pack under `.coding-scaffold/policy/`
+- a non-empty deny list in `agent-permissions.json`
+- a PR template under `.github/PULL_REQUEST_TEMPLATE/`
+- an MCP policy file *if* MCP config is detected
+- a session-trace location
+- a clean `context lint` result
+- a context budget within the configured token limit
+
+`eval run` writes the report to `.coding-scaffold/eval-report.json` and exits non-zero when
+any check fails, so the benchmark can gate CI before AI-assisted PRs are accepted.
+
+The benchmark scores readiness; it does *not* measure model intelligence. The point is to
+confirm the repo gives the agent enough to work with — not to rank the agent.
+
 ## Pilot Metrics
 
 These are measurement templates, not features. Track them manually during the pilot. If a number