Phase 0 issue #3: 9 more conformance fixtures (20/20 — gate cleared)

Hand-curated 9 fixtures from arc's untapped scenario hierarchy:

  auth/
    auth-unauthorized-challenge       (401 + WWW-Authenticate emission)
    auth-token-exchange-initialize    (RFC 8693 narrowing exchange)

  mcp_core/
    initialize                        (protocol version negotiation)
    tools-call-simple-text            (CallToolResult content shape)

  chio-extensions/
    deny-receipt-emitted              (verdict-receipt obligation)

  tasks/
    tasks-call-get-result             (premature Done status race)

  nested_callbacks/
    nested-elicitation-form-create    (callback round-trip routing)

  notifications/
    resources-subscribe-updated       (long-lived transport state)

  arc PRs/
    peer-lock-resolution-pr-147       (runtime peer-lock resolution sub-fix)

Each has a plausible failure_message modeled on the relevant
driver's assertion-failure output (HTTP 401 with header expectations,
Status::Done timing, JSON content-type negotiation, etc), ranked
canonical_fix files with real anchors, and curator notes explaining
why the fixture is good fixture material.

Coverage now spans the MCP surface broadly: auth, lifecycle, tools,
tasks (experimental), notifications, nested callbacks, chio-extension
verdict-receipts. 12 of 20 fixtures came from real arc scenarios; 8
came from real arc PR fixes.

eval-outcomes report status flips:
  | conformance-harness-recall | 20 | BLOCKED — runner | fixtures present; runner is Phase 1 deliverable |

**All four Phase 0 outcome evals are now runner-blocked**, not
fixture-blocked. Issue #3's fixture deliverable is complete.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: bb-connor

chio-pack/eval/fixtures/conformance-recall/auth-token-exchange-initialize-2026-05-07.yml

@@ -0,0 +1,30 @@
+id: auth-token-exchange-initialize-2026-05-07
+failing_test: "tests/conformance/scenarios/auth/auth-token-exchange-initialize.json#auth-token-exchange-initialize"
+failure_message: |
+  FAILED  auth-token-exchange-initialize (RFC 8693)
+    Expected: token exchange request → response with new access token
+              bound to the requested resource scope
+    Got:      response token has scope "*" instead of the requested
+              narrow scope `chio:tool:database_read`
+    The token endpoint over-granted scope. Per RFC 8693, the issued
+    token MUST NOT have wider scope than was requested AND wider than
+    the original token's scope. Both bounds were violated.
+canonical_fix:
+  - file: "tests/conformance/scenarios/auth/auth-token-exchange-initialize.json"
+    section: "expected response scope"
+  - file: "crates/chio-mcp-adapter/src/transport.rs"
+    section: "token exchange handling"
+  - file: "crates/chio-core-types/src/capability.rs"
+    section: "scope narrowing"
+  - file: "spec/PROTOCOL.md"
+    section: "RFC 8693 token exchange"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: token exchange narrows scope correctly"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. RFC 8693 token exchange is a security-critical MCP
+  surface — over-granting scope here is a privilege-escalation bug.
+  Pairs with capability-scope-related vault content. The agent must
+  recognize that "scope: *" in a response is NEVER correct for a
+  narrowing exchange.

chio-pack/eval/fixtures/conformance-recall/auth-unauthorized-challenge-2026-05-07.yml

@@ -0,0 +1,27 @@
+id: auth-unauthorized-challenge-2026-05-07
+failing_test: "tests/conformance/scenarios/auth/auth-unauthorized-challenge.json#auth-unauthorized-challenge"
+failure_message: |
+  FAILED  auth-unauthorized-challenge (specVersion 2025-11-25, streamable-http)
+    Expected: HTTP 401 with WWW-Authenticate: Bearer realm="...", resource_metadata="..."
+    Got:      HTTP 200 (unauthenticated initialize succeeded)
+    The server returned a successful initialize response to an
+    unauthenticated request — should have challenged with a
+    protected-resource metadata reference.
+canonical_fix:
+  - file: "tests/conformance/scenarios/auth/auth-unauthorized-challenge.json"
+    section: "expected pass criteria"
+  - file: "crates/chio-mcp-edge/src/runtime.rs"
+    section: "auth challenge handling"
+  - file: "crates/chio-mcp-adapter/src/transport.rs"
+    section: "WWW-Authenticate emission"
+  - file: "spec/PROTOCOL.md"
+    section: "Authorization challenge"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: unauthenticated initialize gets WWW-Authenticate challenge"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Auth challenge is the entry point to OAuth-style
+  protected-resource discovery. A server that fails closed (401 with
+  metadata reference) is the spec'd behavior; failing open (200) is the
+  fault mode the scenario catches.

chio-pack/eval/fixtures/conformance-recall/chio-deny-receipt-emitted-2026-05-07.yml

@@ -0,0 +1,29 @@
+id: chio-deny-receipt-emitted-2026-05-07
+failing_test: "tests/conformance/scenarios/chio-extensions/deny-receipt-emitted.json#deny-receipt-emitted"
+failure_message: |
+  FAILED  chio-extensions/deny-receipt-emitted (specVersion 2026-03-19)
+    Expected: denied tool call emits `receipt.guard-verdict.<id>` with
+              verdict = "deny" and signed by the kernel
+    Got:      no receipt emitted (deny was silent)
+    Per spec.guard.pipeline, every guard verdict (regardless of outcome)
+    MUST emit receipt.guard-verdict — silent denials are forbidden.
+canonical_fix:
+  - file: "tests/conformance/scenarios/chio-extensions/deny-receipt-emitted.json"
+    section: "expected receipt emission"
+  - file: "crates/chio-guards/src/pipeline.rs"
+    section: "verdict receipt emission"
+  - file: "crates/chio-receipts/src/lib.rs"
+    section: "guard-verdict receipt construction"
+  - file: "spec/PROTOCOL.md"
+    section: "Deny receipts (chio-extension)"
+  - file: "docs/standards/CHIO_RECEIPTS_PROFILE.md"
+    section: "Verdict receipt obligation"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: denied actions emit signed deny receipts"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Tests the receipt-emission obligation that spec.guard.pipeline
+  treats as a normative MUST. Silent deny = covert channel. Pairs with
+  [[../../vault/spec/guard-pipeline]] (the normative spec) and
+  [[../../vault/spec/receipt-commitment]] (the receipt-format spec).

chio-pack/eval/fixtures/conformance-recall/mcp-core-initialize-2026-05-07.yml

@@ -0,0 +1,31 @@
+id: mcp-core-initialize-2026-05-07
+failing_test: "tests/conformance/scenarios/mcp_core/initialize.json#initialize"
+failure_message: |
+  FAILED  mcp-core/initialize (specVersion 2025-11-25)
+    Expected: server emits InitializeResult with protocolVersion negotiated
+              to "2025-11-25"
+    Got:      InitializeResult.protocolVersion = "2024-11-05"
+    Server's negotiated protocol version is below the floor pinned by
+    the conformance suite. The peer must support 2025-11-25 to pass
+    initialize-tier conformance.
+canonical_fix:
+  - file: "tests/conformance/scenarios/mcp_core/initialize.json"
+    section: "expected protocolVersion negotiation"
+  - file: "crates/chio-mcp-adapter/src/lib.rs"
+    section: "initialize handler"
+  - file: "crates/chio-mcp-edge/src/runtime.rs"
+    section: "version negotiation"
+  - file: "spec/PROTOCOL.md"
+    section: "Initialize lifecycle"
+  - file: "docs/conformance/verdict-matrix.md"
+    section: "MCP-core initialize verdict cell"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: client/server initialize lifecycle"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Initialize is the most basic MCP scenario — every peer
+  must pass it. Failure at this gate cascades: every other mcp-core
+  scenario fails because the session can't be established. Tests the
+  agent's ability to recognize that "fix initialize first" is the
+  blocking issue when many tests are red.

chio-pack/eval/fixtures/conformance-recall/mcp-core-tools-call-simple-text-2026-05-07.yml

@@ -0,0 +1,29 @@
+id: mcp-core-tools-call-simple-text-2026-05-07
+failing_test: "tests/conformance/scenarios/mcp_core/tools-call-simple-text.json#tools-call-simple-text"
+failure_message: |
+  FAILED  mcp-core/tools-call-simple-text (specVersion 2025-11-25)
+    Expected: tool call returns CallToolResult with content[0].type = "text"
+              and content[0].text = "<canonical response>"
+    Got:      CallToolResult.content[0].type = "image"
+    The server returned a non-text content type for a tool whose
+    fixture defines text output. Likely a content-type negotiation bug
+    OR the server is returning the wrong content for this tool.
+canonical_fix:
+  - file: "tests/conformance/scenarios/mcp_core/tools-call-simple-text.json"
+    section: "expected content[0] shape"
+  - file: "crates/chio-mcp-adapter/src/lib.rs"
+    section: "CallToolResult marshaling"
+  - file: "tests/conformance/peers/python/server.py"
+    section: "test tool implementation"
+  - file: "tests/conformance/peers/js/server.mjs"
+    section: "test tool implementation"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: tool returns simple text result"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Simple-text tool calls are the conformance baseline for
+  the tools/* MCP surface. Failure here often points at a peer
+  implementation bug (server returning wrong content) more than an
+  adapter bug — tests the agent's ability to navigate the peer dirs
+  vs the adapter dirs.

chio-pack/eval/fixtures/conformance-recall/nested-elicitation-form-create-2026-05-07.yml

@@ -0,0 +1,29 @@
+id: nested-elicitation-form-create-2026-05-07
+failing_test: "tests/conformance/scenarios/nested_callbacks/nested-elicitation-form-create.json#nested-elicitation-form-create"
+failure_message: |
+  FAILED  nested_callbacks/nested-elicitation-form-create
+    Expected: server-initiated elicitation/createForm propagates
+              through to client; client response routes back via the
+              same callback channel
+    Got:      timeout waiting for client response (10s budget exceeded)
+    The nested elicitation channel never delivered the client response.
+    Likely a callback ID routing bug or a transport-layer ordering
+    issue under the streamable-http nested case.
+canonical_fix:
+  - file: "tests/conformance/scenarios/nested_callbacks/nested-elicitation-form-create.json"
+    section: "expected callback round-trip"
+  - file: "crates/chio-mcp-adapter/src/lib.rs"
+    section: "nested callback routing"
+  - file: "crates/chio-mcp-edge/src/runtime.rs"
+    section: "elicitation channel"
+  - file: "spec/PROTOCOL.md"
+    section: "Nested callbacks / elicitation"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: nested elicitation form create round-trip"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Nested callbacks are an under-tested MCP surface where
+  bugs hide behind happy paths. Form-create elicitation is one of the
+  more complex cases because it involves a server-initiated flow that
+  resolves via a client-initiated response on the same logical channel.

chio-pack/eval/fixtures/conformance-recall/notifications-resources-subscribe-2026-05-07.yml

@@ -0,0 +1,31 @@
+id: notifications-resources-subscribe-2026-05-07
+failing_test: "tests/conformance/scenarios/notifications/resources-subscribe-updated-notification.json#resources-subscribe-updated-notification"
+failure_message: |
+  FAILED  notifications/resources-subscribe-updated-notification
+    Expected: after resources/subscribe(uri="..."), an updated
+              notification arrives within 1s when the resource changes
+    Got:      no notification received (5s timeout)
+    Either the subscription didn't register, OR the notification
+    didn't fire on resource change. Per the MCP notifications contract,
+    the subscription's transport channel must remain open for
+    server-initiated pushes.
+canonical_fix:
+  - file: "tests/conformance/scenarios/notifications/resources-subscribe-updated-notification.json"
+    section: "expected notification timing"
+  - file: "crates/chio-mcp-adapter/src/transport.rs"
+    section: "notification streaming"
+  - file: "crates/chio-mcp-edge/src/runtime.rs"
+    section: "subscription registry"
+  - file: "spec/PROTOCOL.md"
+    section: "Resource subscriptions and notifications"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: resources/subscribe receives change notifications"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Notifications are the only MCP surface that mandates
+  long-lived transport state. Failure mode: the subscription is
+  registered but the transport gets recycled (e.g., HTTP request
+  completion) and the subscription silently drops. Tests retrieval's
+  ability to surface BOTH the runtime registration AND the transport-
+  layer keep-alive contract.

chio-pack/eval/fixtures/conformance-recall/peer-lock-resolution-pr-147.yml

@@ -0,0 +1,32 @@
+id: peer-lock-resolution-pr-147
+failing_test: "crates/chio-cli/tests/conformance_cli.rs::test_peer_lock_resolves_at_runtime"
+failure_message: |
+  FAILED  test_peer_lock_resolves_at_runtime
+    Expected: chio conformance run --peer python loads
+              tests/conformance/peers/python/peers.lock.toml at runtime
+    Got:      "Error: peer 'python' not registered (compile-time
+             registry built without peers.lock.toml)"
+    The conformance CLI was building its peer registry at COMPILE time
+    from the workspace layout, not resolving peers.lock.toml at runtime.
+    A user with a custom peer set can't extend the registry without
+    rebuilding the binary.
+canonical_fix:
+  - file: "crates/chio-cli/tests/conformance_cli.rs"
+    section: "test_peer_lock_resolves_at_runtime"
+  - file: "crates/chio-cli/src/conformance/mod.rs"
+    section: "peer registry construction"
+  - file: "tests/conformance/peers/python/peers.lock.toml"
+    section: "lock file format"
+  - file: "docs/conformance/verdict-matrix.md"
+    section: "Peer registration"
+relevant_arc_pr: "https://github.com/bb-connor/arc/pull/147"
+relevant_arc_commit: "(arc PR #147: fix(conformance): correct crate package layout, runtime peer-lock resolution, and CLI argument validation)"
+commit_subject: "fix(conformance): correct crate package layout, runtime peer-lock resolution, and CLI argument validation"
+commit_date: "(see arc PR #147)"
+notes: |
+  Curated from arc PR #147 metadata. The PR's title is multi-part — this
+  fixture targets the **runtime peer-lock resolution** sub-fix
+  specifically (crate-layout and CLI-arg sub-fixes are out of scope as
+  conformance-recall fixtures since they're packaging concerns, not
+  protocol behavior). Tests retrieval's ability to surface both the
+  CLI test, the registry construction site, and the lock file format.

chio-pack/eval/fixtures/conformance-recall/tasks-call-get-result-2026-05-07.yml

@@ -0,0 +1,31 @@
+id: tasks-call-get-result-2026-05-07
+failing_test: "tests/conformance/scenarios/tasks/tasks-call-get-result.json#tasks-call-get-result"
+failure_message: |
+  FAILED  tasks/tasks-call-get-result (specVersion 2025-11-25, mcp-experimental)
+    Expected: tasks/call returns task_id; tasks/get returns Status::Running
+              while task is in flight; tasks/result returns final value
+              when status transitions to Status::Done
+    Got:      tasks/get returned Status::Done before task actually finished
+    Server returning premature Done status — likely a race between the
+    task scheduler and the status query. The conformance harness uses
+    deterministic fixtures so this isn't a flake.
+canonical_fix:
+  - file: "tests/conformance/scenarios/tasks/tasks-call-get-result.json"
+    section: "expected tasks/get progress"
+  - file: "crates/chio-mcp-adapter/src/lib.rs"
+    section: "tasks/* surface"
+  - file: "crates/chio-mcp-edge/src/runtime.rs"
+    section: "task scheduler"
+  - file: "spec/PROTOCOL.md"
+    section: "Tasks (experimental)"
+relevant_arc_pr: "(scenario authored on origin/codex/chio-kb-a-grade-dogfood)"
+relevant_arc_commit: "(scenario fixture)"
+commit_subject: "Conformance scenario: tasks/* lifecycle (call → get → result)"
+commit_date: "2026-05-07T00:00:00-04:00"
+notes: |
+  Hand-curated. Tests the experimental tasks/* MCP surface. Premature
+  Done is one of the harder failure modes to debug because the
+  expected and actual responses are both well-formed JSON — only the
+  ordering and timing distinguish correct from incorrect. Tests
+  retrieval's ability to surface the scheduler code, not just the
+  surface-level RPC handler.

vault/_meta/dashboards/eval-outcomes.md

@@ -1,12 +1,12 @@
-# Outcome evals — 2026-05-07 19:07 UTC
+# Outcome evals — 2026-05-07 21:28 UTC
 
 > Generated by `chio-pack-eval` Phase 0 skeleton. No real runners yet — see PHASE-0.md.
 
 | Eval | Fixtures | Status | Notes |
 | ---- | -------- | ------ | ----- |
 | `time-to-first-correct-fix` | 8 | BLOCKED — runner | fixtures present; runner is Phase 1 deliverable |
 | `repeated-mistake-rate` | 0 | BLOCKED — runner | no fixtures glob; runner is Phase 1 deliverable |
-| `conformance-harness-recall` | 11 | BLOCKED — fixtures | have 11, need ≥ 20 (PHASE-0.md) |
+| `conformance-harness-recall` | 20 | BLOCKED — runner | fixtures present; runner is Phase 1 deliverable |
 | `capability-error-explanation` | 10 | BLOCKED — runner | fixtures present; runner is Phase 1 deliverable |
 
 ## Deferred (block on Phase 2)