Skip to content

Commit f6dcbaf

Browse files
bb-connorclaude
bb-connor
and
claude
committed
Phase 0 issue #4: 10 capability-error-explanation scenarios + RATERS.md
10 scenarios covering distinct Chio capability-error failure modes: - revoked-cap-still-presented (revocation list rejection) - capability-insufficient-scope (scope mismatch, valid cap) - capability-past-expiration (exp claim past with drift window) - delegation-chain-broken-link (mid-chain signature corruption) - capability-wrong-audience (aud mismatch — not a retry case) - capability-unrecognized-scope-identifier (vocabulary versioning) - cross-issuer-revocation-attempt (NOT-SUPPORTED in v1) - dpop-proof-htu-mismatch (RFC 9449 binding failure) - capability-mangled-jwt-signature (base64 vs base64url corruption) - concurrent-revocation-race (RVL propagation staleness) Each fixture exercises the four canonical augmentations (raw control, kb_brief_feature, +signed_receipt, +episode_link) so raters score the augmentation matrix not the scenario. Scenarios deliberately span: - rejection vs retry actionability - kernel-side vs transport-side root cause - documented spec gaps (cross-issuer revocation Open Question) - protocol-edge cases (RFC 9449 DPoP, base64url encoding) RATERS.md: documented the **solo-maintainer fallback** that PHASE-0.md didn't anticipate. Two paths surfaced: 1. Recruit two humans (preferred — preserves cross-rater variance). 2. LLM-judge fallback (rater-B = Sonnet 4.6, rater-C = Haiku 4.5 with rubric variant). Costs documented: rubric drift undetectable if both LLMs share blind spots; subjective dimensions drift toward LLM stylistic preferences. rater-A (human) catches via flagged- scenario re-rate. ADR-0004-rater-pool proposed in RATERS.md as the unblocking ADR before ADR-0002 baseline sign-off. eval-outcomes report status: | time-to-first-correct-fix | 8 | BLOCKED — runner | | repeated-mistake-rate | 0 | BLOCKED — runner | | conformance-harness-recall | 11 | BLOCKED — fixtures | (need 9 more) | capability-error-explanation | 10 | BLOCKED — runner | Two of four outcome evals fixture-complete. The Phase 0 work that remains: 9 more conformance-recall fixtures, the repeated-mistake-rate session-log harness, and rater identification per the new ADR-0004. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent c798537 commit f6dcbaf

12 files changed

Lines changed: 498 additions & 5 deletions

‎chio-pack/eval/RATERS.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,16 +10,26 @@ The `capability-error-explanation` eval requires exactly three raters per scenar
1010

1111
| Rater | Pseudonym | Role | Joined | Calibration last passed |
1212
| ----- | --------- | ---- | ------ | ----------------------- |
13-
| @connor | rater-A | Chio author | 2026-05-07 | TBD |
14-
| TBD | rater-B | TBD | TBD | TBD |
15-
| TBD | rater-C | TBD | TBD | TBD |
13+
| @connor | rater-A | Chio author / maintainer | 2026-05-07 | TBD (Run-0 pending) |
14+
| TBD | rater-B | Pending: Chio-familiar engineer (target: 1 internal Backbay engineer) | TBD | — |
15+
| TBD | rater-C | Pending: external rater OR LLM-judge fallback (see "Solo-maintainer fallback" below) | TBD | — |
1616

1717
A fourth (alternate) is named for tiebreak when the disagreement flag fires (`max - min > 1` on any dimension).
1818

1919
| Alternate | Pseudonym | Joined | Available |
2020
| --------- | --------- | ------ | --------- |
2121
| TBD | rater-D | TBD | flagged-rerate only |
2222

23+
## Solo-maintainer fallback (Phase 0 reality)
24+
25+
The original PHASE-0.md spec assumed three named human raters at Phase 0 baseline time. As of v0.0.1-vault-complete the maintainer pool is **single-human** (@connor). Two paths to unblock:
26+
27+
1. **Recruit two humans.** Identify two Chio-familiar engineers (one internal, one external if open-source community grows) who can commit to ~30 minutes per eval run, ~4 runs in the first six months. Preferred path. RATERS.md table fills in; Run-0 calibration proceeds normally.
28+
29+
2. **LLM-judge fallback.** Document an explicit Phase-0-only configuration where rater-B and rater-C are LLM judges using **distinct prompt strategies** (e.g., rater-B = Claude Sonnet 4.6 with the rubric verbatim; rater-C = Claude Haiku 4.5 with a rubric variant emphasizing accuracy). Disagreement-flag rate becomes the sanity check rather than human-vs-human variance. **Costs:** rubric drift goes undetected if both LLMs share blind spots; subjective dimensions (clarity especially) drift toward LLM stylistic preferences. **Mitigation:** rater-A (human) catches LLM-stylistic bias by re-rating any flagged scenario.
30+
31+
The current state is "**Path 1 preferred, Path 2 documented.**" An ADR (proposed `ADR-0004-rater-pool`) should record which path the project commits to before ADR-0002 sign-off.
32+
2333
## Pseudonymization rule
2434

2535
Stored output uses the pseudonym (`rater-A`, `rater-B`, `rater-C`, `rater-D`) — **never the handle**. The mapping above is the only place the bridge is recorded; it lives only in this committed file (no DB, no env var, no log line).

‎chio-pack/eval/fixtures/cap-error-explanation/capability-insufficient-scope.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
id: capability-insufficient-scope
2+
source: "Drafted from spec.capability.revocation + spec.guard.pipeline"
3+
4+
scenario: |
5+
Agent presents a valid capability for a tool action, but the requested
6+
action falls outside the capability's scope. Kernel returns:
7+
8+
CapabilityScope: requested action `database_write` not in granted
9+
scopes [`database_read`, `cache_read`]
10+
11+
context_query: "CapabilityScope requested action not in granted scopes"
12+
13+
augmentations_under_test:
14+
- name: raw
15+
body: "CapabilityScope: requested action `database_write` not in granted scopes [`database_read`, `cache_read`]"
16+
17+
- name: kb_brief_feature
18+
body_source:
19+
tool: kb_brief_feature
20+
args:
21+
feature: "capability scope enforcement"
22+
focus_paths:
23+
- "crates/chio-core-types/src/capability.rs"
24+
- "crates/chio-kernel-core/src/capability_verify.rs"
25+
26+
- name: kb_brief_feature + signed_receipt
27+
body_source:
28+
tool: kb_brief_feature
29+
args:
30+
feature: "capability scope enforcement"
31+
focus_paths:
32+
- "crates/chio-core-types/src/capability.rs"
33+
include_signed_receipt: true
34+
35+
- name: kb_brief_feature + episode_link
36+
body_source:
37+
tool: kb_brief_feature
38+
args:
39+
feature: "capability scope enforcement"
40+
focus_paths:
41+
- "crates/chio-core-types/src/capability.rs"
42+
include_episode_link: "episode.chio-architecture-summary"
43+
44+
rubric_overrides: {}
45+
46+
notes: |
47+
Common scope-mismatch — the agent has a capability but used it for
48+
the wrong action. Distinguishes from revocation (cap is still valid).
49+
Actionable next step: request a wider-scope cap from the issuer
50+
(with a justification), or refactor the action to fit existing scopes.

‎chio-pack/eval/fixtures/cap-error-explanation/capability-mangled-jwt-signature.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
id: capability-mangled-jwt-signature
2+
source: "Drafted from spec.protocol.wire-format + arc tests/conformance/native/scenarios/capability-validation.json"
3+
4+
scenario: |
5+
Agent presents a capability JWT whose signature segment is
6+
base64url-corrupted. Kernel returns:
7+
8+
CapabilitySignature: signature verification failed — base64url
9+
decode error at byte 247 (expected `-`/`_`/`A-Za-z0-9`, found `+`)
10+
11+
context_query: "CapabilitySignature signature verification failed base64url"
12+
13+
augmentations_under_test:
14+
- name: raw
15+
body: "CapabilitySignature: signature verification failed — base64url decode error at byte 247 (expected `-`/`_`/`A-Za-z0-9`, found `+`)"
16+
17+
- name: kb_brief_feature
18+
body_source:
19+
tool: kb_brief_feature
20+
args:
21+
feature: "capability JWT signature decoding"
22+
focus_paths:
23+
- "crates/chio-core-types/src/capability.rs"
24+
- "crates/chio-kernel-core/src/capability_verify.rs"
25+
26+
- name: kb_brief_feature + signed_receipt
27+
body_source:
28+
tool: kb_brief_feature
29+
args:
30+
feature: "capability JWT signature decoding"
31+
include_signed_receipt: true
32+
33+
- name: kb_brief_feature + episode_link
34+
body_source:
35+
tool: kb_brief_feature
36+
args:
37+
feature: "capability JWT signature decoding"
38+
include_episode_link: "episode.chio-architecture-summary"
39+
40+
rubric_overrides: {}
41+
42+
notes: |
43+
Common transport-corruption case: a `+` somewhere in the signature
44+
means standard base64 was used somewhere instead of base64url. The
45+
fix is at the transport layer (re-fetch the cap, fix the encoder),
46+
not at the kernel. Raters should reward augmentations that surface
47+
the wire-format encoding contract (base64url, not base64).

‎chio-pack/eval/fixtures/cap-error-explanation/capability-past-expiration.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
id: capability-past-expiration
2+
source: "Drafted from spec.capability.revocation"
3+
4+
scenario: |
5+
Agent presents a capability whose `exp` claim was 30 seconds ago.
6+
Kernel returns:
7+
8+
CapabilityExpiry: capability expired at 2026-05-07T18:24:30Z
9+
(current time: 2026-05-07T18:25:00Z, drift_window: 5s)
10+
11+
context_query: "CapabilityExpiry capability expired drift window"
12+
13+
augmentations_under_test:
14+
- name: raw
15+
body: "CapabilityExpiry: capability expired at 2026-05-07T18:24:30Z (current time: 2026-05-07T18:25:00Z, drift_window: 5s)"
16+
17+
- name: kb_brief_feature
18+
body_source:
19+
tool: kb_brief_feature
20+
args:
21+
feature: "capability expiration drift window"
22+
focus_paths:
23+
- "crates/chio-core-types/src/capability.rs"
24+
- "crates/chio-kernel-core/src/capability_verify.rs"
25+
26+
- name: kb_brief_feature + signed_receipt
27+
body_source:
28+
tool: kb_brief_feature
29+
args:
30+
feature: "capability expiration drift window"
31+
include_signed_receipt: true
32+
33+
- name: kb_brief_feature + episode_link
34+
body_source:
35+
tool: kb_brief_feature
36+
args:
37+
feature: "capability expiration drift window"
38+
include_episode_link: "episode.capability-revocation-architecture"
39+
40+
rubric_overrides: {}
41+
42+
notes: |
43+
Expiry-vs-revocation distinction. Both reject, but the right next step
44+
differs: expired → mint a fresh capability (cheap, no policy change);
45+
revoked → understand why it was revoked (could be compromise or policy
46+
shift). Raters should score actionability higher when augmentations
47+
surface the mint-renewal path.

‎chio-pack/eval/fixtures/cap-error-explanation/capability-unrecognized-scope-identifier.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
id: capability-unrecognized-scope-identifier
2+
source: "Drafted from spec.protocol.wire-format"
3+
4+
scenario: |
5+
Agent presents a capability granting scope
6+
`chio:tool:database_write/v3:json`. Kernel returns:
7+
8+
UnknownScope: scope identifier `chio:tool:database_write/v3:json`
9+
not in registered scope vocabulary (last sync: 2026-05-07T16:00:00Z)
10+
11+
context_query: "UnknownScope scope identifier registered vocabulary"
12+
13+
augmentations_under_test:
14+
- name: raw
15+
body: "UnknownScope: scope identifier `chio:tool:database_write/v3:json` not in registered scope vocabulary (last sync: 2026-05-07T16:00:00Z)"
16+
17+
- name: kb_brief_feature
18+
body_source:
19+
tool: kb_brief_feature
20+
args:
21+
feature: "scope vocabulary registration"
22+
focus_paths:
23+
- "crates/chio-core-types/src/capability.rs"
24+
- "spec/schemas/chio-wire/v1/"
25+
26+
- name: kb_brief_feature + signed_receipt
27+
body_source:
28+
tool: kb_brief_feature
29+
args:
30+
feature: "scope vocabulary registration"
31+
include_signed_receipt: true
32+
33+
- name: kb_brief_feature + episode_link
34+
body_source:
35+
tool: kb_brief_feature
36+
args:
37+
feature: "scope vocabulary registration"
38+
include_episode_link: "episode.kb-upgrade-decision"
39+
40+
rubric_overrides: {}
41+
42+
notes: |
43+
The unknown-scope case is a vocabulary-versioning issue: the
44+
capability cites a `/v3:json` scope that the kernel doesn't recognize.
45+
Could be a fresh-from-issuer cap that the kernel hasn't synced yet,
46+
OR a typo, OR an `x-`-style vendor extension that wasn't prefixed.
47+
The fix path varies — tests retrieval's ability to disambiguate.

‎chio-pack/eval/fixtures/cap-error-explanation/capability-wrong-audience.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
id: capability-wrong-audience
2+
source: "Drafted from spec.protocol.wire-format + spec.sdk.conformance"
3+
4+
scenario: |
5+
Agent presents a capability whose `aud` claim is `did:chio:tool:a`
6+
but the request is going to `did:chio:tool:b`. Kernel returns:
7+
8+
AudienceMismatch: capability `aud` did:chio:tool:a does not match
9+
target did:chio:tool:b
10+
11+
context_query: "AudienceMismatch capability aud does not match target"
12+
13+
augmentations_under_test:
14+
- name: raw
15+
body: "AudienceMismatch: capability `aud` did:chio:tool:a does not match target did:chio:tool:b"
16+
17+
- name: kb_brief_feature
18+
body_source:
19+
tool: kb_brief_feature
20+
args:
21+
feature: "capability audience binding"
22+
focus_paths:
23+
- "crates/chio-core-types/src/capability.rs"
24+
- "spec/PROTOCOL.md"
25+
26+
- name: kb_brief_feature + signed_receipt
27+
body_source:
28+
tool: kb_brief_feature
29+
args:
30+
feature: "capability audience binding"
31+
include_signed_receipt: true
32+
33+
- name: kb_brief_feature + episode_link
34+
body_source:
35+
tool: kb_brief_feature
36+
args:
37+
feature: "capability audience binding"
38+
include_episode_link: "episode.chio-architecture-summary"
39+
40+
rubric_overrides: {}
41+
42+
notes: |
43+
`aud` mismatch is a confusing failure mode for newcomers — the
44+
capability is "valid" but bound to a different target. Raters should
45+
reward augmentations that explain the binding contract (clarity 5)
46+
and point at the spec where audience-binding is normative
47+
(accuracy 5). The fix is not to retry — it's to request a capability
48+
bound to the correct target.

‎chio-pack/eval/fixtures/cap-error-explanation/concurrent-revocation-race.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
id: concurrent-revocation-race
2+
source: "Drafted from spec.capability.revocation + crates/chio-revocation-oracle/tests/property_oracle.rs"
3+
4+
scenario: |
5+
Agent presents a capability that was revoked 200ms ago. The kernel's
6+
RVL had not yet propagated to this verifier replica. Kernel returns:
7+
8+
RevocationStaleness: capability revoked at 2026-05-07T18:25:00.000Z,
9+
but RVL replica observed it as live at 2026-05-07T18:25:00.200Z
10+
(replica RVL version: 47; current authoritative version: 48)
11+
12+
context_query: "RevocationStaleness RVL replica version propagation"
13+
14+
augmentations_under_test:
15+
- name: raw
16+
body: "RevocationStaleness: capability revoked at 2026-05-07T18:25:00.000Z, but RVL replica observed it as live at 2026-05-07T18:25:00.200Z (replica RVL version: 47; current authoritative version: 48)"
17+
18+
- name: kb_brief_feature
19+
body_source:
20+
tool: kb_brief_feature
21+
args:
22+
feature: "RVL propagation"
23+
focus_paths:
24+
- "crates/chio-kernel/src/revocation_store.rs"
25+
- "crates/chio-revocation-oracle/tests/property_oracle.rs"
26+
27+
- name: kb_brief_feature + signed_receipt
28+
body_source:
29+
tool: kb_brief_feature
30+
args:
31+
feature: "RVL propagation"
32+
include_signed_receipt: true
33+
34+
- name: kb_brief_feature + episode_link
35+
body_source:
36+
tool: kb_brief_feature
37+
args:
38+
feature: "RVL propagation"
39+
include_episode_link: "episode.capability-revocation-architecture"
40+
41+
rubric_overrides: {}
42+
43+
notes: |
44+
Edge-case race condition during RVL propagation. The capability IS
45+
revoked (authoritative truth: version 48) but the verifying replica
46+
is one version behind. Per spec.capability.revocation, the kernel
47+
should fail-closed under partition. Tests retrieval's ability to
48+
surface the property test (`property_oracle.rs`) which exercises
49+
this exact race. The fix isn't application-level — it's RVL
50+
propagation tightening or fail-closed-on-stale.

‎chio-pack/eval/fixtures/cap-error-explanation/cross-issuer-revocation-attempt.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
id: cross-issuer-revocation-attempt
2+
source: "Drafted from spec.capability.revocation Open Questions"
3+
4+
scenario: |
5+
Operator (issuer C) attempts to revoke a capability minted by issuer A
6+
and delegated to grantee B. Kernel returns:
7+
8+
CrossIssuerRevocation: revocation rejected — capability was minted
9+
by did:chio:issuer:A, but the revocation request is signed by
10+
did:chio:issuer:C (no trust-graph entry authorizes C-revokes-A)
11+
12+
context_query: "CrossIssuerRevocation revocation rejected trust-graph"
13+
14+
augmentations_under_test:
15+
- name: raw
16+
body: "CrossIssuerRevocation: revocation rejected — capability was minted by did:chio:issuer:A, but the revocation request is signed by did:chio:issuer:C (no trust-graph entry authorizes C-revokes-A)"
17+
18+
- name: kb_brief_feature
19+
body_source:
20+
tool: kb_brief_feature
21+
args:
22+
feature: "cross-issuer revocation"
23+
focus_paths:
24+
- "crates/chio-kernel/src/kernel/delegation.rs"
25+
- "crates/chio-kernel/src/revocation_store.rs"
26+
27+
- name: kb_brief_feature + signed_receipt
28+
body_source:
29+
tool: kb_brief_feature
30+
args:
31+
feature: "cross-issuer revocation"
32+
include_signed_receipt: true
33+
34+
- name: kb_brief_feature + episode_link
35+
body_source:
36+
tool: kb_brief_feature
37+
args:
38+
feature: "cross-issuer revocation"
39+
include_episode_link: "episode.capability-revocation-architecture"
40+
41+
rubric_overrides: {}
42+
43+
notes: |
44+
Cross-issuer revocation is documented as NOT SUPPORTED in v1 (per
45+
spec.capability.revocation Open Questions). The operator's error is
46+
procedural, not a system bug. Raters should reward augmentations that
47+
surface the "Open question" status clearly — actionability is "ask
48+
issuer A to revoke; or, if A is unreachable, file an ADR for the
49+
trust-graph extension."

0 commit comments

Comments
 (0)