feat(governance): YAML-native governance layer + strict validation + duplicate REQ cleanup

## Changes

### Data Fixes
- Remove 23 duplicate REQs (REQ-221..243 were exact duplicates of REQ-130..135
  and REQ-161..179 added in a previous sprint). REQs: 280 → 257, all now covered.
- Sync workitems.json: 107 → 257 entries mirroring all 257 REQs

### YAML-native governance (Stage 2+3+4)
- New module: src/specsmith/governance_yaml.py
  - load_yaml_requirements / load_yaml_tests: read from docs/requirements/*.yml
  - save_yaml_requirements / save_yaml_tests: write back to grouped files
  - generate_requirements_md / generate_tests_md: YAML → MD generation
  - is_yaml_mode: checks .specsmith/governance-mode flag
  - strict_validate: 8 schema checks (dup IDs, orphan tests, missing fields, etc.)
- New: docs/requirements/*.yml and docs/tests/*.yml (7 files each, grouped by domain)
- New: .specsmith/governance-mode = yaml (YAML-first mode enabled)
- New: scripts/migrate_governance_to_yaml.py (idempotent migration script)

### Sync flipped to YAML-first (Stage 4)
- src/specsmith/sync.py: when governance-mode=yaml, reads from YAML files,
  updates JSON cache, and regenerates docs/REQUIREMENTS.md + docs/TESTS.md as
  derived artifacts. Markdown-primary mode still works for legacy projects.

### New CLI commands (Stage 2+3)
- specsmith validate --strict [--json]: schema enforcement (duplicate IDs,
  orphan tests, untested REQs, missing required fields, title duplicates,
  sync drift). Exits 1 on errors; warnings do not block.
- specsmith generate docs [--check] [--json]: YAML → MD regeneration.
  The YAML-first equivalent of specsmith sync for human-readable artifacts.

### CI gate (Stage 2)
- .github/workflows/ci.yml: new validate-strict job runs on every push/PR.
  Catches governance schema violations before they reach main.

### API surface
- tests/fixtures/api_surface.json: regenerated to include new commands.

Co-Authored-By: Oz <oz-agent@warp.dev>

Author: tbitcs

.github/workflows/ci.yml

@@ -87,6 +87,23 @@ jobs:
           SPECSMITH_PYPI_CHECKED: "1"
         run: python -m specsmith sync --check --project-dir .
 
+  validate-strict:
+    # YAML governance schema guard: duplicate IDs, orphan tests, missing fields.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: python -m pip install --upgrade pip
+      - run: pip install -e ".[dev]"
+      - name: Strict governance schema validation
+        env:
+          SPECSMITH_NO_AUTO_UPDATE: "1"
+          SPECSMITH_PYPI_CHECKED: "1"
+        run: python -m specsmith validate --strict --json --project-dir .
+
   api-surface:
     # REQ-140 guard: regenerates the public CLI surface and fails the build
     # if the live output drifts from the committed fixture. Catches accidental

.specsmith/governance-mode

@@ -0,0 +1 @@
+yaml

.specsmith/requirements.json

@@ -1532,167 +1532,6 @@
     "source": "BTWS-2027 AI Governance Report [REG-015]",
     "status": "defined"
   },
-  {
-    "id": "REQ-221",
-    "title": "Instinct Persistence System",
-    "description": "specsmith MUST implement an instinct persistence system in src/specsmith/instinct.py storing patterns extracted from successful sessions.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-001]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-222",
-    "title": "Instinct Record Schema",
-    "description": "Each instinct record MUST contain: id, trigger_pattern, content, confidence, project_scope, created, last_used, use_count.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-002]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-223",
-    "title": "Session End Instinct Extraction",
-    "description": "The SESSION_END hook MUST extract candidate instincts from session history for user review before the session closes.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-003]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-224",
-    "title": "Learn Command",
-    "description": "The /learn command MUST promote a user-approved pattern to an instinct with an initial confidence score and persist it to the instinct store.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-004]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-225",
-    "title": "Instinct Confidence Updates",
-    "description": "Instinct confidence MUST be updated based on application success or rejection — increasing on accepted application and decreasing on rejection.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-005]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-226",
-    "title": "Instinct Import Export",
-    "description": "Instincts MUST be importable and exportable as .md files for cross-project and cross-team sharing.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-006]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-227",
-    "title": "Instinct Status Command",
-    "description": "/instinct-status MUST display all active instincts sorted by confidence descending, with use_count and last_used fields.",
-    "source": "PLANNED-REQUIREMENTS.md [LRN-007]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-228",
-    "title": "Eval Harness Module",
-    "description": "specsmith MUST implement an eval harness in src/specsmith/eval/ supporting eval-driven development workflows.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-001]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-229",
-    "title": "Eval Data Model",
-    "description": "The eval model MUST define: Task, Trial, Grader, Transcript, Outcome as core types.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-002]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-230",
-    "title": "Eval Task Storage",
-    "description": "Tasks MUST be stored as Markdown files at .specsmith/evals/{feature}.md with YAML frontmatter.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-003]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-231",
-    "title": "Eval Grader Types",
-    "description": "The eval harness MUST support CodeGrader, ModelGrader, and HumanFlag grader types for different validation strategies.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-004]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-232",
-    "title": "Eval Pass at K Metrics",
-    "description": "The eval harness MUST compute pass@k and pass^k metrics for measuring agent capability across multiple trials.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-005]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-233",
-    "title": "Git-Based Eval Grading",
-    "description": "Default grading MUST be git-based outcome grading (checking actual changes in VCS) rather than execution-path assertion.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-006]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-234",
-    "title": "Eval Run Command",
-    "description": "/eval run --trials k MUST run k independent trials and report pass@k results with per-trial transcripts.",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-007]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-235",
-    "title": "Capability vs Regression Evals",
-    "description": "The eval harness MUST distinguish capability evals (new functionality) from regression evals (existing functionality preservation).",
-    "source": "PLANNED-REQUIREMENTS.md [EDD-008]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-236",
-    "title": "Agent Memory Module",
-    "description": "specsmith MUST implement cross-session agent memory in src/specsmith/memory.py persisting patterns, facts, and history across sessions.",
-    "source": "PLANNED-REQUIREMENTS.md [MEM-001]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-237",
-    "title": "Agent Memory Schema",
-    "description": "Agent memory MUST be structured JSON containing: accumulated patterns, preferred approaches, known project facts, and failure history.",
-    "source": "PLANNED-REQUIREMENTS.md [MEM-002]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-238",
-    "title": "Session Start Memory Injection",
-    "description": "The SESSION_START hook MUST inject relevant memories into the system prompt, respecting the configured token budget to avoid context overrun.",
-    "source": "PLANNED-REQUIREMENTS.md [MEM-003]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-239",
-    "title": "Typed Execution Layer",
-    "description": "All tool handlers MUST use a typed ProjectOperations class for file, git/VCS, and search operations. Direct raw shell string assembly in tool handlers is prohibited.",
-    "source": "PLANNED-REQUIREMENTS.md [OPS-001]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-240",
-    "title": "ProjectOperations File Interface",
-    "description": "ProjectOperations MUST expose file operations (read_file, write_file, list_dir, glob, search) implemented via Python pathlib/stdlib with no subprocess calls.",
-    "source": "PLANNED-REQUIREMENTS.md [OPS-002]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-241",
-    "title": "ProjectOperations VCS Interface",
-    "description": "ProjectOperations MUST expose git/VCS operations (status, log, diff, add, commit, push, create_branch, create_pr) returning structured typed result objects.",
-    "source": "PLANNED-REQUIREMENTS.md [OPS-003]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-242",
-    "title": "ProjectOperations Result Schema",
-    "description": "All ProjectOperations methods MUST return a typed result containing at minimum: exit_code, stdout, stderr, and elapsed_ms.",
-    "source": "PLANNED-REQUIREMENTS.md [OPS-004]",
-    "status": "defined"
-  },
-  {
-    "id": "REQ-243",
-    "title": "Cross-Platform ProjectOperations",
-    "description": "ProjectOperations MUST be cross-platform (Windows, Linux, macOS) without platform-specific code branches at call sites.",
-    "source": "PLANNED-REQUIREMENTS.md [OPS-006]",
-    "status": "defined"
-  },
   {
     "id": "REQ-244",
     "title": "GPU-Aware Context Window Sizing",

.specsmith/testcases.json

@@ -2628,28 +2628,6 @@
     "expected_behavior": {},
     "confidence": 1.0
   },
-  {
-    "id": "TEST-282",
-    "title": "HF Leaderboard Sync Persists Bucket Scores to JSON",
-    "description": "`sync_from_huggingface_blocking(force_static=True, scores_path=tmp_path/\"scores.json\")` creates the file at the given path, whose JSON root contains a `\"bucket_scores\"` dict. Each entry has `reasoning_score`, `conversational_score`, `longform_score`, and `model_name` keys.",
-    "requirement_id": "REQ-263",
-    "type": "unit",
-    "verification_method": "pytest",
-    "input": {},
-    "expected_behavior": {},
-    "confidence": 1.0
-  },
-  {
-    "id": "TEST-283",
-    "title": "HF Token Included in Request Headers When Set",
-    "description": "When `SPECSMITH_HF_TOKEN` is set to a non-empty string, `test_hf_connection()` returns `{\"token_set\": true}` and the rate_limit_tier includes \"authenticated\". The `_fetch_page` request (captured via mock) includes `Authorization: Bearer <token>` in its headers.",
-    "requirement_id": "REQ-265",
-    "type": "unit",
-    "verification_method": "pytest",
-    "input": {},
-    "expected_behavior": {},
-    "confidence": 1.0
-  },
   {
     "id": "TEST-263",
     "title": "HF Leaderboard Static Fallback Loads Without Network",
@@ -2858,5 +2836,27 @@
     "input": {},
     "expected_behavior": {},
     "confidence": 1.0
+  },
+  {
+    "id": "TEST-282",
+    "title": "HF Leaderboard Sync Persists Bucket Scores to JSON",
+    "description": "`sync_from_huggingface_blocking(force_static=True, scores_path=tmp_path/\"scores.json\")` creates the file at the given path, whose JSON root contains a `\"bucket_scores\"` dict. Each entry has `reasoning_score`, `conversational_score`, `longform_score`, and `model_name` keys.",
+    "requirement_id": "REQ-263",
+    "type": "unit",
+    "verification_method": "pytest",
+    "input": {},
+    "expected_behavior": {},
+    "confidence": 1.0
+  },
+  {
+    "id": "TEST-283",
+    "title": "HF Token Included in Request Headers When Set",
+    "description": "When `SPECSMITH_HF_TOKEN` is set to a non-empty string, `test_hf_connection()` returns `{\"token_set\": true}` and the rate_limit_tier includes \"authenticated\". The `_fetch_page` request (captured via mock) includes `Authorization: Bearer <token>` in its headers.",
+    "requirement_id": "REQ-265",
+    "type": "unit",
+    "verification_method": "pytest",
+    "input": {},
+    "expected_behavior": {},
+    "confidence": 1.0
   }
 ]