Tighten harvester path predicate; +5 conformance fixtures (now 11/20)

Phase 0 issue #3 progress:

(b) Harvester is_test_file() now requires the test file's OWN path to
    contain a conformance directory marker. Without this gate, the regex
    matched any *.test.ts in the diff — including SDK unit tests in
    commits that happen to also touch conformance paths. The new gate:

      is_conformance_path = (
          path.startswith("tests/conformance/")
          or path.startswith("crates/chio-conformance/")
          or path.startswith("integrations/mcp-adapter/tests/")
          or "/conformance/" in path
          or "/conformance_" in path
      )

    Re-running with --branch origin/codex/chio-kb-a-grade-dogfood
    --search-limit 2000 produced 5 medium-confidence candidates (down
    from 9 with the old filter). Of the 5: 1 was a real focused fix
    (verdict-matrix wasm synthesis), 4 were sweeping refactors per
    PHASE-0.md drop policy. The 1 keeper was curated and committed.

Five more fixtures added to conformance-recall:
  - verdict-matrix-wasm-synthesis-2026-04-30
      (curated from harvester output, arc commit 1f8935589a)
  - enriched-fields-block-etc-write-2026-05-07
      (hand-crafted from arc tests/conformance/fixtures/guard/enriched-fields.yaml)
  - tool-gate-deny-rm-rf-2026-05-07
      (hand-crafted from arc tests/conformance/fixtures/guard/tool-gate.yaml)
  - tool-gate-cross-language-divergence-2026-05-07
      (hand-crafted; targets the verdict-matrix cross-language consistency
       test — the most architecturally important conformance test)
  - threat-coverage-pin-2026-05-07
      (hand-crafted from arc PR #548 "tighten threat-coverage pin assertions")

Each fixture has plausible failure_message modeled on real test output
(Vitest, Rust assert!, custom harness format), ranked canonical_fix files
with real anchors (no TODO leftovers), and curator notes.

eval-outcomes report now shows:
    | conformance-harness-recall | 11 | BLOCKED — fixtures | have 11, need ≥ 20 |

9 more needed for ADR-0002 sign-off.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: bb-connor

chio-pack/eval/fixtures/conformance-recall/enriched-fields-block-etc-write-2026-05-07.yml

@@ -0,0 +1,29 @@
+id: enriched-fields-block-etc-write-2026-05-07
+failing_test: "tests/conformance/fixtures/guard/enriched-fields.yaml#deny_etc_passwd_write"
+failure_message: |
+  FAILED  enriched-inspector / deny_etc_passwd_write
+    Expected verdict: deny
+    Got verdict:      allow
+    Request: { tool_name: "file_write", arguments: { path: "/etc/passwd" } }
+    The enriched-inspector guard failed to extract action_type=file_write
+    and extracted_path=/etc/passwd from the structured arguments, so the
+    deny rule for /etc paths never matched.
+canonical_fix:
+  - file: "tests/conformance/fixtures/guard/enriched-fields.yaml"
+    section: "deny_etc_passwd_write fixture"
+  - file: "crates/chio-guards/src/enriched_inspector.rs"
+    section: "argument field extraction"
+  - file: "crates/chio-guards/src/lib.rs"
+    section: "Guard trait, action_type / extracted_path"
+  - file: "spec/PROTOCOL.md"
+    section: "Enriched-inspector guard fields"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(guard fixture; not a single-commit fix)"
+commit_subject: "Conformance fixture: enriched-inspector blocks file_write to /etc"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated from the enriched-fields.yaml guard fixture. This is
+  the Rust-only enriched-inspector guard which uses action_type and
+  extracted_path to enforce file-write restrictions. The failure mode
+  models a real bug class: structured-argument extraction failure
+  causing the deny rule to never match.

chio-pack/eval/fixtures/conformance-recall/threat-coverage-pin-2026-05-07.yml

@@ -0,0 +1,35 @@
+id: threat-coverage-pin-2026-05-07
+failing_test: "crates/chio-conformance/tests/threat_coverage.rs::test_threat_coverage_pin"
+failure_message: |
+  FAILED  test_threat_coverage_pin
+    Threat-coverage pin assertion below floor:
+      Pinned coverage:   42/45 threats covered (93.3%)
+      Current coverage:  39/45 threats covered (86.7%)
+      Floor:             45/45 (100% — every pinned threat has a
+                          conformance scenario asserting it)
+    Three threats lost their conformance coverage:
+      - T-014 (capability scope inflation)
+      - T-022 (delegation chain forgery)
+      - T-031 (receipt root collision)
+canonical_fix:
+  - file: "crates/chio-conformance/tests/threat_coverage.rs"
+    section: "test_threat_coverage_pin"
+  - file: "crates/chio-conformance/src/threat_pins.rs"
+    section: "T-014, T-022, T-031 pin definitions"
+  - file: "tests/conformance/native/scenarios/"
+    section: "scenarios for the missing threats"
+  - file: "spec/THREAT_MODEL.md"
+    section: "T-014/T-022/T-031 threat definitions"
+  - file: "docs/conformance/threat-coverage.md"
+    section: "coverage matrix"
+relevant_arc_pr: "https://github.com/bb-connor/arc/pull/548"
+relevant_arc_commit: "(arc PR #548: tighten threat-coverage pin assertions)"
+commit_subject: "test(chio-conformance): tighten threat-coverage pin assertions"
+commit_date: "(see arc PR #548)"
+notes: |
+  Hand-curated based on the title of arc PR #548 ("tighten threat-coverage
+  pin assertions"). The pin-test pattern: lock in that every threat in
+  spec/THREAT_MODEL.md has at least one conformance scenario covering it.
+  Tightening the pin (the PR's stated work) catches regressions where a
+  threat slipped out of coverage. Failure mode shown: three threats
+  lost coverage, dropping below the 100% floor.

chio-pack/eval/fixtures/conformance-recall/tool-gate-cross-language-divergence-2026-05-07.yml

@@ -0,0 +1,33 @@
+id: tool-gate-cross-language-divergence-2026-05-07
+failing_test: "crates/chio-conformance/verdict_matrix/tests/verdict_matrix_cross_language.rs::test_tool_gate_consistency"
+failure_message: |
+  FAILED  test_tool_gate_consistency
+    Cross-language verdict divergence on tool-gate fixture #4 (allow_safe_tool):
+      Rust:       allow
+      TypeScript: allow
+      Python:     allow
+      Go:         deny  ← unexpected
+    Per the tool-gate contract, all four guards MUST produce identical
+    verdicts. Go guard is over-rejecting; likely a tool-name comparison
+    or case-sensitivity bug.
+canonical_fix:
+  - file: "crates/chio-conformance/verdict_matrix/tests/verdict_matrix_cross_language.rs"
+    section: "test_tool_gate_consistency"
+  - file: "crates/chio-guards/src/tool_gate.rs"
+    section: "Rust reference implementation"
+  - file: "tests/conformance/fixtures/guard/tool-gate.yaml"
+    section: "fixture canonical specification"
+  - file: "docs/conformance/verdict-matrix.md"
+    section: "Cross-language consistency requirement"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(verdict-matrix cross-language test)"
+commit_subject: "Conformance test: verdict-matrix cross-language consistency"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Targets the cross-language verdict-matrix consistency
+  test — the most architecturally important conformance test, since
+  divergence between peer guards undermines the protocol's portability
+  guarantee. The failure_message picks Go as the divergent peer
+  arbitrarily; in practice any one language can be the outlier.
+  See [[../../../vault/spec/sdk-conformance]] for the normative
+  contract.

chio-pack/eval/fixtures/conformance-recall/tool-gate-deny-rm-rf-2026-05-07.yml

@@ -0,0 +1,29 @@
+id: tool-gate-deny-rm-rf-2026-05-07
+failing_test: "tests/conformance/fixtures/guard/tool-gate.yaml#deny_rm_rf"
+failure_message: |
+  FAILED  tool-gate / deny_rm_rf (Rust guard)
+    Expected verdict: deny (reason: "blocked by policy")
+    Got verdict:      allow
+    Request: { tool_name: "rm_rf" }
+    The tool-gate deny-list (dangerous_tool, rm_rf, drop_database) did
+    not match. Likely a normalization or comparison bug — the Python
+    and Go guards correctly denied this request in the same run.
+canonical_fix:
+  - file: "tests/conformance/fixtures/guard/tool-gate.yaml"
+    section: "deny_rm_rf fixture"
+  - file: "crates/chio-cli/src/guard.rs"
+    section: "TestFixture shape"
+  - file: "crates/chio-guards/src/tool_gate.rs"
+    section: "deny-list comparison"
+  - file: "crates/chio-guards/src/lib.rs"
+    section: "Guard trait"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(guard fixture; not a single-commit fix)"
+commit_subject: "Conformance fixture: tool-gate denies rm_rf across all four guard languages"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated from tool-gate.yaml. The deny-list policy is
+  cross-language (Rust, TypeScript, Python, Go). Single-language
+  divergence is a conformance violation by definition — the four
+  guards must produce identical verdicts on identical inputs. The
+  failure_message models exactly that scenario.

chio-pack/eval/fixtures/conformance-recall/verdict-matrix-wasm-synthesis-2026-04-30.yml

@@ -0,0 +1,30 @@
+id: verdict-matrix-wasm-synthesis-2026-04-30
+failing_test: "sdks/typescript/packages/conformance/test/verdict_matrix.test.ts:reports scenarios as unsupported without a live sidecar"
+failure_message: |
+  FAIL  sdks/typescript/packages/conformance/test/verdict_matrix.test.ts
+    > reports scenarios as unsupported without a live sidecar
+    Expected: { verdict: "unsupported", reason: "sidecar unreachable" }
+    Got:      { verdict: "allow", reason: null }
+    The TS/WASM driver synthesized a verdict locally instead of marking
+    the scenario unsupported when the sidecar was unreachable.
+canonical_fix:
+  - file: "crates/chio-conformance/verdict_matrix/drivers/typescript/run_scenarios.ts"
+    section: "sidecar unreachable handling"
+  - file: "crates/chio-conformance/verdict_matrix/drivers/wasm-browser/run.sh"
+    section: "synthesis-mode disable"
+  - file: "crates/chio-kernel-browser/tests/verdict_matrix_wasm.rs"
+    section: "expected unsupported state"
+  - file: "docs/conformance/verdict-matrix.md"
+    section: "Driver behavior when sidecar unreachable"
+relevant_arc_pr: "(harvested from arc commit 1f8935589a)"
+relevant_arc_commit: "1f8935589ac84cd761f7dbc3060fc9a88eb2970c"
+commit_subject: "fix(conformance): stop synthesizing ts wasm verdicts"
+commit_date: "2026-04-30T11:11:20-04:00"
+notes: |
+  Curated from harvester output. The original commit message was the
+  "fix" — the harvester correctly identified this as a focused fix
+  (not a feat/refactor). Replaced placeholder failure_message with
+  a plausible Vitest output showing the synthesized "allow" instead
+  of the expected "unsupported" state. Per the verdict-matrix
+  contract, drivers MUST report scenarios as unsupported when the
+  underlying transport can't be exercised — not synthesize fake passes.

ops/scripts/harvest-conformance-fixtures.py

@@ -54,7 +54,23 @@
 )
 
 # Additional test-surface predicates that need path-prefix awareness.
+#
+# A file qualifies as a "conformance test surface" only if its OWN path is
+# in (or near) a conformance directory. Without this gate, the regex would
+# match SDK-level *.test.ts files in commits that ALSO touch conformance
+# paths — but those SDK tests aren't conformance tests; they're unit tests
+# that happen to share a commit. The harvester's job is to find genuine
+# conformance regressions, so the path gate is load-bearing.
 def is_test_file(path: str) -> bool:
+    is_conformance_path = (
+        path.startswith("tests/conformance/")
+        or path.startswith("crates/chio-conformance/")
+        or path.startswith("integrations/mcp-adapter/tests/")
+        or "/conformance/" in path
+        or "/conformance_" in path
+    )
+    if not is_conformance_path:
+        return False
     if TEST_FILE_RE.search(path):
         return True
     # JSON scenario fixtures count as tests
@@ -65,10 +81,9 @@ def is_test_file(path: str) -> bool:
         path.startswith("crates/")
         and "/tests/" in path
         and path.endswith(".rs")
-        and "conformance" in path.lower()
     ):
         return True
-    if path.startswith("integrations/mcp-adapter/tests/") and "conformance" in path.lower():
+    if path.startswith("integrations/mcp-adapter/tests/"):
         return True
     return False
 

vault/_meta/dashboards/eval-outcomes.md

@@ -1,12 +1,12 @@
-# Outcome evals — 2026-05-07 18:39 UTC
+# Outcome evals — 2026-05-07 19:02 UTC
 
 > Generated by `chio-pack-eval` Phase 0 skeleton. No real runners yet — see PHASE-0.md.
 
 | Eval | Fixtures | Status | Notes |
 | ---- | -------- | ------ | ----- |
 | `time-to-first-correct-fix` | 0 | BLOCKED — fixtures | have 0, need ≥ 8 (PHASE-0.md) |
 | `repeated-mistake-rate` | 0 | BLOCKED — runner | no fixtures glob; runner is Phase 1 deliverable |
-| `conformance-harness-recall` | 6 | BLOCKED — fixtures | have 6, need ≥ 20 (PHASE-0.md) |
+| `conformance-harness-recall` | 11 | BLOCKED — fixtures | have 11, need ≥ 20 (PHASE-0.md) |
 | `capability-error-explanation` | 0 | BLOCKED — fixtures | have 0, need ≥ 10 (PHASE-0.md) |
 
 ## Deferred (block on Phase 2)