Skip to content

feat(m2): HMAC-SHA256 seal upgrade for eval-result.json#6

Open
suzuke wants to merge 2 commits into
feat/m2-reporter-comparefrom
feat/m2-hmac-seal
Open

feat(m2): HMAC-SHA256 seal upgrade for eval-result.json#6
suzuke wants to merge 2 commits into
feat/m2-reporter-comparefrom
feat/m2-hmac-seal

Conversation

@suzuke
Copy link
Copy Markdown
Owner

@suzuke suzuke commented Apr 25, 2026

Summary

Stacked on #5 (M2 PR 11 reporter compare). Adds opt-in HMAC-SHA256 sealing for eval-result.json, closing the gap from "corruption-detect" to "tamper-evident under Docker isolation". Default stays content-sha256 so existing projects are byte-identical to M1.

Threat model closed

In Docker mode (per spec §INV-2 — env_allowlist=[], non-root user), the agent has no access to the host's CRUCIBLE_SEAL_KEY. After this PR, an agent that can write to eval-result.json still cannot forge a valid HMAC because it doesn't have the key. The host-side verify_seal detects tampering on read.

What's new

Seal format

  • M1: content-sha256:<sha256-hex> (corruption check only)
  • M2: hmac-sha256:<key-id>:<hmac-hex> (tamper-evident)

The first colon-split field is the algorithm name. Schema is NOT bumped — the prefix differentiates formats.

src/crucible/sealing.py (new module, ~225 LOC)

  • compute_seal(canonical_bytes, *, config) — dispatches on config.algorithm
  • verify_seal(canonical_bytes, seal, *, config, allow_legacy_content=False) — strict-by-default policy enforcement
  • validate_seal_config(config) — startup precheck so HMAC misconfig fails fast
  • _load_key(config) — resolves key from key_file (priority) or env var (CRUCIBLE_SEAL_KEY default), with hex validation + ≥32-byte minimum + whitespace stripping
  • SealKeyError — clear error class for config invariant violations

SealConfig (new dataclass)

seal:
  algorithm: hmac-sha256        # default: content-sha256
  key_id: "default"             # opaque label, embedded in seal string
  key_env_var: CRUCIBLE_SEAL_KEY
  key_file: null                # optional; takes precedence over env

Strict-by-default policy enforcement

Under config.algorithm == "hmac-sha256", verify_seal rejects content-sha256: seals by default to close the algorithm-downgrade bypass (see reviewer round 2 below). Migration tools may opt into legacy compat via allow_legacy_content=True.

Reviewer trail

Round Verdict Key finding
1 (design) NEEDS_TWEAK Q3 narrowed: single active key only in v1; multi-key rotation a follow-up. Strict missing-key error confirmed. Hex-encoded ≥32-byte keys.
2 (impl) REJECTED Algorithm-downgrade bypass: under HMAC config, an attacker without the key could tamper with payload and write content-sha256:<sha256(tampered)>verify_seal accepted it because dispatch went on the seal prefix, not config policy. I had even written a test asserting this as expected behavior.
3 (fix) VERIFIED Strict-by-default verify_seal closes the bypass. The bad assertion inverted to assert correct rejection. Legacy compat scoped to explicit allow_legacy_content=True for migration tools.

Stats

  • 2 commits (initial impl + downgrade-bypass fix), 6 files changed (+737 / −21 LOC)
  • 2,448 passed / 4 skipped, 0 regressions on top of PR 11 baseline (2,415)
  • 33 sealing tests in test_sealing.py covering reviewer's full constraint matrix

Test plan

Round-trip + tamper detection

  • content-sha256 round-trip + byte-identical to M1
  • hmac-sha256 round-trip with env key / file key
  • Tamper detection: byte flip / wrong key changes HMAC

Strict-by-default policy enforcement (reviewer F1)

  • Recomputed content-sha256 under HMAC policy → False (bypass closed)
  • Explicit allow_legacy_content=True accepts M1 seals (migration path works)
  • Legacy flag does NOT excuse tampering (still requires correct hash)
  • Content policy still accepts content seals (M1 path preserved)
  • Content policy accepts hmac seals if key matches (asymmetric upgrade direction)

Key resolution

  • key_file > key_env_var precedence
  • Whitespace / multi-line hex tolerated
  • Non-hex / short / unreadable / missing key → clear SealKeyError

verify_seal non-throwing on garbage

  • Missing colon / unknown algorithm / non-hex / odd-length / truncated / wrong key_id / blank key_id → False (no raise)

Startup validation

  • validate_seal_config no-ops for content-sha256
  • Raises immediately for hmac-sha256 without key
  • Passes when key set

Defensive

  • compute_seal raises clear ValueError on unknown algorithm
  • Config-time validation rejects unknown algorithm + key_id containing :

Known limitations (M2 follow-ups)

  • Multi-key rotation: this PR ships single-active-key. Multi-key directory (.crucible/keys/<key-id>.hex) for verifying old artefacts under rotated keys is a deliberate follow-up — different config UX, separate scope.
  • crucible verify CLI: verify_seal is unit-tested but not exposed as a user command yet. That needs CLI UX, failure reporting, ledger traversal — separate PR.
  • Auto-key-generation on crucible init: not done. Users opt into HMAC manually by setting algo + providing key. Could be a wizard improvement later.

Migration path

  • Existing projects: no action needed. Default content-sha256 keeps M1 byte-identical seals.
  • Opt into HMAC: set seal.algorithm: hmac-sha256 in .crucible/config.yaml and provide CRUCIBLE_SEAL_KEY (hex-encoded ≥32 bytes) in env or seal.key_file.
  • Read old M1 seals after switching: future migration tool will use verify_seal(..., allow_legacy_content=True).

🤖 Generated with Claude Code

suzuke and others added 2 commits April 26, 2026 00:06
Adds opt-in tamper-evident sealing of `eval-result.json`. M1's
`content-sha256:<hex>` only detected accidental corruption; M2 introduces
`hmac-sha256:<key-id>:<hex>` which detects tampering by anyone without
the host's CRUCIBLE_SEAL_KEY (per spec §INV-2: in Docker isolation, the
agent has no access to host env vars).

Seal string format:
  - content-sha256:<sha256-hex>          (M1, default — backward compat)
  - hmac-sha256:<key-id>:<hmac-hex>      (M2 opt-in)

The first colon-split field is the algorithm name; verify_seal accepts
either format regardless of current config, so projects that switch
algorithms can still verify older artefacts.

Reviewer round 1 verdict: NEEDS_TWEAK — narrow Q3 scope. Single active
key only in this PR; multi-key directory rotation is a follow-up.
Other constraints folded in:
- Default `content-sha256` preserves M1 byte-identical output
- Strict SealKeyError if HMAC requested without key (no silent downgrade)
- validate_seal_config() called at orchestrator startup so misconfig
  fails before the first eval runs
- hmac.compare_digest() for both algorithms (timing-safe)
- verify_seal returns False (no raise) on malformed seal text
- Hex-encoded keys only, ≥32 decoded bytes, whitespace stripped
- key_file > key_env_var precedence (locked by tests)
- key_id must not contain ':' (would corrupt seal format)
- Never log key bytes / full seal — only key_id + algorithm

New module `src/crucible/sealing.py` with `compute_seal`, `verify_seal`,
`validate_seal_config`, plus internal `_load_key` and `_is_hex` helpers.
SealConfig dataclass added to config.py with full validation in
`_build_seal()`.

Tests: 30 new unit tests in `test_sealing.py` covering reviewer's full
constraint list + cross-algorithm verification (M1 seals readable under
M2 config) + tamper-and-recompute attack defence (recomputed
content-sha256 does NOT verify when policy is hmac-sha256).

Existing tests updated: 4 fake_orchestrator config blocks in
test_orchestrator_ledger.py now include the seal SimpleNamespace.

Full suite: 2445 passed / 4 skipped, 0 regressions vs PR 11 (2415).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(F1)

Reviewer round 2 REJECTED PR 12 with a real algorithm-downgrade bypass:
under HMAC policy, an attacker without the host key could tamper with
`eval-result.json` and write a fresh `content-sha256:<sha256(tampered)>`
seal — the seal IS a valid sha256 hash, and verify_seal happily
accepted it because dispatch went on the seal string's algorithm prefix
rather than the project's policy.

I had even WRITTEN a test asserting this bypass as expected behavior
(`test_recomputed_content_sha256_does_not_verify_as_hmac` had the
assertion flipped). Reviewer caught both.

Fix:
- `verify_seal(... , allow_legacy_content: bool = False)` new kwarg.
- Under `config.algorithm == "hmac-sha256"`, content-sha256 seals are
  rejected unless `allow_legacy_content=True` is explicitly passed by a
  migration / legacy-read tool. Default False — normal verification
  enforces policy strictly, closing the bypass.
- Hmac seals under content-sha256 policy still verify (hmac is strictly
  stronger; no asymmetric attack vector).

Tests:
- Inverted the bad assertion: `test_recomputed_content_sha256_does_not_
  verify_as_hmac` now correctly asserts `is False` for the bypass case.
- Replaced the legacy-compat test with three explicit ones:
  - `test_verify_rejects_content_sha256_under_hmac_policy_by_default`
  - `test_verify_accepts_content_sha256_with_explicit_legacy_flag`
  - `test_legacy_flag_does_not_excuse_tampering` (flag is policy escape,
    not integrity bypass)
- Added two explicit policy tests:
  - `test_content_policy_still_accepts_content_seals` (M1 path)
  - `test_content_policy_accepts_hmac_seal_if_key_matches` (upgrade dir
    is fine — hmac is strictly stronger)

Full suite: 2448 passed / 4 skipped, 0 regressions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant