feat(m2): HMAC-SHA256 seal upgrade for eval-result.json#6
Open
suzuke wants to merge 2 commits into
Open
Conversation
Adds opt-in tamper-evident sealing of `eval-result.json`. M1's `content-sha256:<hex>` only detected accidental corruption; M2 introduces `hmac-sha256:<key-id>:<hex>` which detects tampering by anyone without the host's CRUCIBLE_SEAL_KEY (per spec §INV-2: in Docker isolation, the agent has no access to host env vars). Seal string format: - content-sha256:<sha256-hex> (M1, default — backward compat) - hmac-sha256:<key-id>:<hmac-hex> (M2 opt-in) The first colon-split field is the algorithm name; verify_seal accepts either format regardless of current config, so projects that switch algorithms can still verify older artefacts. Reviewer round 1 verdict: NEEDS_TWEAK — narrow Q3 scope. Single active key only in this PR; multi-key directory rotation is a follow-up. Other constraints folded in: - Default `content-sha256` preserves M1 byte-identical output - Strict SealKeyError if HMAC requested without key (no silent downgrade) - validate_seal_config() called at orchestrator startup so misconfig fails before the first eval runs - hmac.compare_digest() for both algorithms (timing-safe) - verify_seal returns False (no raise) on malformed seal text - Hex-encoded keys only, ≥32 decoded bytes, whitespace stripped - key_file > key_env_var precedence (locked by tests) - key_id must not contain ':' (would corrupt seal format) - Never log key bytes / full seal — only key_id + algorithm New module `src/crucible/sealing.py` with `compute_seal`, `verify_seal`, `validate_seal_config`, plus internal `_load_key` and `_is_hex` helpers. SealConfig dataclass added to config.py with full validation in `_build_seal()`. Tests: 30 new unit tests in `test_sealing.py` covering reviewer's full constraint list + cross-algorithm verification (M1 seals readable under M2 config) + tamper-and-recompute attack defence (recomputed content-sha256 does NOT verify when policy is hmac-sha256). Existing tests updated: 4 fake_orchestrator config blocks in test_orchestrator_ledger.py now include the seal SimpleNamespace. Full suite: 2445 passed / 4 skipped, 0 regressions vs PR 11 (2415). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(F1)
Reviewer round 2 REJECTED PR 12 with a real algorithm-downgrade bypass:
under HMAC policy, an attacker without the host key could tamper with
`eval-result.json` and write a fresh `content-sha256:<sha256(tampered)>`
seal — the seal IS a valid sha256 hash, and verify_seal happily
accepted it because dispatch went on the seal string's algorithm prefix
rather than the project's policy.
I had even WRITTEN a test asserting this bypass as expected behavior
(`test_recomputed_content_sha256_does_not_verify_as_hmac` had the
assertion flipped). Reviewer caught both.
Fix:
- `verify_seal(... , allow_legacy_content: bool = False)` new kwarg.
- Under `config.algorithm == "hmac-sha256"`, content-sha256 seals are
rejected unless `allow_legacy_content=True` is explicitly passed by a
migration / legacy-read tool. Default False — normal verification
enforces policy strictly, closing the bypass.
- Hmac seals under content-sha256 policy still verify (hmac is strictly
stronger; no asymmetric attack vector).
Tests:
- Inverted the bad assertion: `test_recomputed_content_sha256_does_not_
verify_as_hmac` now correctly asserts `is False` for the bypass case.
- Replaced the legacy-compat test with three explicit ones:
- `test_verify_rejects_content_sha256_under_hmac_policy_by_default`
- `test_verify_accepts_content_sha256_with_explicit_legacy_flag`
- `test_legacy_flag_does_not_excuse_tampering` (flag is policy escape,
not integrity bypass)
- Added two explicit policy tests:
- `test_content_policy_still_accepts_content_seals` (M1 path)
- `test_content_policy_accepts_hmac_seal_if_key_matches` (upgrade dir
is fine — hmac is strictly stronger)
Full suite: 2448 passed / 4 skipped, 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #5 (M2 PR 11 reporter compare). Adds opt-in HMAC-SHA256 sealing for
eval-result.json, closing the gap from "corruption-detect" to "tamper-evident under Docker isolation". Default stayscontent-sha256so existing projects are byte-identical to M1.Threat model closed
In Docker mode (per spec §INV-2 —
env_allowlist=[], non-root user), the agent has no access to the host'sCRUCIBLE_SEAL_KEY. After this PR, an agent that can write toeval-result.jsonstill cannot forge a valid HMAC because it doesn't have the key. The host-sideverify_sealdetects tampering on read.What's new
Seal format
content-sha256:<sha256-hex>(corruption check only)hmac-sha256:<key-id>:<hmac-hex>(tamper-evident)The first colon-split field is the algorithm name. Schema is NOT bumped — the prefix differentiates formats.
src/crucible/sealing.py(new module, ~225 LOC)compute_seal(canonical_bytes, *, config)— dispatches onconfig.algorithmverify_seal(canonical_bytes, seal, *, config, allow_legacy_content=False)— strict-by-default policy enforcementvalidate_seal_config(config)— startup precheck so HMAC misconfig fails fast_load_key(config)— resolves key fromkey_file(priority) or env var (CRUCIBLE_SEAL_KEYdefault), with hex validation + ≥32-byte minimum + whitespace strippingSealKeyError— clear error class for config invariant violationsSealConfig(new dataclass)Strict-by-default policy enforcement
Under
config.algorithm == "hmac-sha256",verify_sealrejectscontent-sha256:seals by default to close the algorithm-downgrade bypass (see reviewer round 2 below). Migration tools may opt into legacy compat viaallow_legacy_content=True.Reviewer trail
content-sha256:<sha256(tampered)>—verify_sealaccepted it because dispatch went on the seal prefix, not config policy. I had even written a test asserting this as expected behavior.verify_sealcloses the bypass. The bad assertion inverted to assert correct rejection. Legacy compat scoped to explicitallow_legacy_content=Truefor migration tools.Stats
test_sealing.pycovering reviewer's full constraint matrixTest plan
Round-trip + tamper detection
content-sha256round-trip + byte-identical to M1hmac-sha256round-trip with env key / file keyStrict-by-default policy enforcement (reviewer F1)
content-sha256under HMAC policy → False (bypass closed)allow_legacy_content=Trueaccepts M1 seals (migration path works)Key resolution
key_file>key_env_varprecedenceSealKeyErrorverify_sealnon-throwing on garbageStartup validation
validate_seal_configno-ops for content-sha256Defensive
compute_sealraises clearValueErroron unknown algorithm:Known limitations (M2 follow-ups)
.crucible/keys/<key-id>.hex) for verifying old artefacts under rotated keys is a deliberate follow-up — different config UX, separate scope.crucible verifyCLI:verify_sealis unit-tested but not exposed as a user command yet. That needs CLI UX, failure reporting, ledger traversal — separate PR.crucible init: not done. Users opt into HMAC manually by setting algo + providing key. Could be a wizard improvement later.Migration path
content-sha256keeps M1 byte-identical seals.seal.algorithm: hmac-sha256in.crucible/config.yamland provideCRUCIBLE_SEAL_KEY(hex-encoded ≥32 bytes) in env orseal.key_file.verify_seal(..., allow_legacy_content=True).🤖 Generated with Claude Code