From fdefed775d169966c41d190beed4def676480cce Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sat, 18 Jul 2026 04:18:46 -0500 Subject: [PATCH 01/25] spec(proposal): draft Phase 5 approval semantics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/PHASE-5-PROPOSAL-approval-semantics.md | 337 +++++++++++++++++++ 1 file changed, 337 insertions(+) create mode 100644 specs/PHASE-5-PROPOSAL-approval-semantics.md diff --git a/specs/PHASE-5-PROPOSAL-approval-semantics.md b/specs/PHASE-5-PROPOSAL-approval-semantics.md new file mode 100644 index 0000000..362ca9c --- /dev/null +++ b/specs/PHASE-5-PROPOSAL-approval-semantics.md @@ -0,0 +1,337 @@ +# PHASE-5 PROPOSAL — Approval semantics over the Phase-2 Ward + +**Status:** PROPOSAL (draft — Phase 5 has not opened; phase gates per `AGENTS.md` still bind) +**Date:** 2026-07-18 +**Bead:** `threads-uqx` +**Authors:** Sage lane (drafted by coordinator session agent) + +--- + +## 0. Scope and sources + +This is a proposal document only. It does not start Phase 5, amend frozen +Phase 0, or change the daemon. Val and Nova decide whether Phase 5 opens. + +Sources read and cited: + +- `specs/PHASE-0-DESIGN.md`: frozen v0.2; Thread (authority relationship: + surface -> writer), Weave (enforced pattern of threads), Strand (fiber inside + a thread), Channel (axis of load), predicate-authoritative / descriptor- + derived rule, Gate-4 fail-closed posture, and WARD-C1..C7 lineage. +- `../familiar-contract/rfcs/RFC-0001-familiar-contract.md`: §4.2 protected + invariants, §4.3 editable surface, §5.3 approval tiers, §5.4 gates, §5.5 + probes. RFC amendments for closure precondition and provenance predicate are + pending on issues #3/#4 and are treated as upstream-in-flight. +- `../coven-grimoire/articles/drafts/2026-06-24-ward-layer-spec-brief.md` §9: + canonical home of WARD-C1..C7. +- `specs/PHASE-4-CAVE-SURFACES.md`: approval flow §3.7, fail-closed rendering + rules §4, and non-goals §6. +- `../coven/crates/coven-cli/src/ward.rs`, `threads_gate.rs`, `api.rs`: current + daemon reality — path tiers, Gate 1-4 comments, protected-surface weave + construction, and `decide_threads_proposal` as the only tier-0 commit point. +- `~/.coven/workspaces/familiars/nova/ward.toml.v01.bak`: read-only example of + the retired dialect. This draft paraphrases its invariant shape and does not + reproduce Val's exact invariant strings. + +--- + +## 1. What v0.1 expressed that Phase 2 cannot + +Phase 2 reduced Ward configuration to path-tier mapping: tier 0 protected, +tier 1 reviewed, tier 2 logged, tier 3 free. In the daemon, `WardConfig` stores +surface globs and `protected_surface`; `Ward::apply` refuses blocked proposals +as a unit, holds Tier 1 and Gate-1-authorized Tier 0 for Gate 3, and writes only +Tier 2/3. `apply_after_threads_approval` still reruns Gates 1 and 2, and +`api.rs::decide_threads_proposal` revalidates a pending proposal, calls the +threads gate, appends audit rows, advances baselines, removes the pending file, +and is the only current tier-0 approval commit point. + +The retired v0.1 dialect carried three semantic groups Phase 2 cannot express. + +### 1.1 Approval tiers as named governance lanes + +v0.1 distinguished four proposal lanes: automatic regression-gated promotion +with a human veto window; familiar-coherence review with a shorter veto window; +human pre-approval; and human approval with rationale plus audit. Each lane +bound named semantic blocks to a gate. + +Phase 2 can say "logged" or "reviewed." It cannot say "this semantic region is +auto-promoted after regression but vetoable for 48 hours," nor distinguish +human approval from human approval with rationale. RFC-0001 §5.3 still names +these approval tiers and forbids auto-promotion of protected-surface proposals. + +### 1.2 Editable harness blocks as semantic regions + +v0.1 named editable blocks such as execution and recovery prompt regions, tool +defaults, skill configuration, subagent templates, output formats, heartbeat +behavior, tool grants, skill activations, memory conventions, session +introduction, and protected-surface adjacency. These are not file paths. One +block may cut across files; one file may contain several blocks. + +Phase 2 only classifies materialized paths. It cannot answer which semantic +region a diff changed unless that region aligns with a glob. That loses the +RFC-0001 §4.3 editable-surface shape. + +### 1.3 Protected invariants as cross-file identity predicates + +v0.1 carried textual identity assertions over familiar metadata and purpose. +The real backup shows the shape: compact predicate-like statements about name, +person binding, pronouns, purpose, and Coven membership. This document does not +copy them. + +Phase 2 protects files and globs. It does not protect an identity fact that can +be invalidated through an allowed path, prompt block, derived context, or future +portability envelope. RFC-0001 §4.2 says path-only protection is insufficient: +identity-probe inconsistency must reject a proposal regardless of target path. + +--- + +## 2. Carry forward vs deliberately dead + +### 2.1 Approval tiers: carry forward, old encoding dead + +Carry forward the governance distinction; do not revive the v0.1 table shape. +The useful semantic is four approval paths: deterministic low-risk gate, +familiar-coherence gate, human approval, and human approval with rationale. +That matches RFC-0001 §5.3 and explains why Phase 4 already models +`proposal_approved`, `proposal_rejected`, and `proposal_vetoed` audit events. + +What stays dead is stringly coupling among `blocks = [...]`, `gate = "..."`, +and ad hoc veto fields. If Phase 5 opens, policy should be typed and +daemon-owned. The Cave may render it; it must not become a policy engine. + +Recommended name: `ApprovalPath`, not `Tier`. "Tier" is already the path-trust +axis in Phase 2. Reusing it would collapse where a write lands with which +ceremony promotes it. + +### 2.2 Harness blocks: carry forward as regions, not authority + +Carry forward the region abstraction; kill any claim that labels are +authoritative by themselves. A block label is a descriptor unless a +parser/extractor maps bytes to that region deterministically and the daemon can +replay that mapping at apply time. Phase 0 §2.2 binds the rule: predicate +authoritative, descriptor derived. + +Recommended name: `SurfaceRegion` above `SurfaceId`. A `SurfaceRegion` is a +typed semantic region with an extractor predicate over materialized content. +Its descriptor may say "output formats" for the Cave, but the daemon gates on +extractor evidence and final materialized diff. + +### 2.3 Protected invariants: carry forward as predicates, strings dead + +Carry forward strongly. RFC-0001 §4.2 still requires semantic invariants, and +Phase 0 already gives them a home: `PatternPredicate::coherent` is the +authoritative gate; `describe()` yields a derived descriptor. Identity +invariants should become predicate implementations in the weave, not strings +interpreted by clients. + +The v0.1 invariant syntax should stay dead. It was legible but underspecified: +parser semantics, evidence collection, normalization, provenance, and failure +handling were not defined. Pending RFC issues #3/#4 should land, or be waived +by Val/Nova, before Phase 5 freezes invariant interfaces. + +--- + +## 3. Mapping sketches inside the frozen metaphor + +### 3.1 Approval paths: gate policy over existing channels + +Do not model approval paths as new `Channel` variants by default. Channel (axis +of load) says why a thread is stressed: deliberate, forced, serialization, or +mutation. Approval path says which promotion ceremony is required. A mutation +that needs human rationale is still `Channel::Mutation`. + +Recommended default: + +```rust +pub enum ApprovalPath { + AutoRegression { veto: Option }, + FamiliarCoherence { veto: Option }, + HumanApproval, + HumanApprovalWithRationale, +} + +pub struct ProposalClassification { + pub channel: Channel, + pub affected_regions: Vec, + pub path_tier_floor: Tier, + pub approval_path: ApprovalPath, +} +``` + +Path tier remains a floor. A Tier 0 target still requires principal authority +and the threads gate. A Tier 2 path touching a high-risk semantic region may +still require human review. Highest ceremony wins for the proposal as a unit, +matching current all-or-nothing Ward behavior. + +Only add a new channel if Val/Nova decide approval-policy edits themselves need +a distinct load axis. That should not be the default. + +### 3.2 Invariants: PatternPredicate implementations + +Identity invariants should compile into `PatternPredicate` implementations. +The predicate takes relevant threads, strands, and verified proposal evidence; +its result is authoritative. Its descriptor is Cave-readable, labeled derived, +and never enforced on. + +Concrete referent mapping: + +- Thread (authority relationship: surface -> writer): protected identity + surfaces keep principal-writer threads. +- Strand (fiber inside a thread): invariant evidence may use hashes, manifest + entries, audit anchors, serialization markers, and only later a new strand if + evidence forces one. +- Weave (enforced pattern of threads): identity predicates are part of the weave + pattern, not side-table rows. +- Channel (axis of load): invariants must hold under mutation and serialization + at minimum; forced compaction connects through WARD-C1..C6. + +Recommended default: predicate-first; no new strand kind until implementation +evidence requires it. + +### 3.3 Harness blocks: SurfaceRegion above SurfaceId + +A `SurfaceRegion` should be daemon-replayable over a materialized diff: + +```rust +pub trait SurfaceRegionPredicate { + fn materialize(&self, proposal: &MaterializedDiff) -> RegionEvidence; + fn describe(&self) -> SurfaceRegionDescriptor; +} +``` + +The descriptor is for humans. The predicate output feeds classification and +Gate 4 replay. It cannot depend on Cave state, agent self-report, or stale +metadata. + +Thread options: + +- one thread per block: inspectable, but risks threads detached from + source-authoritative surfaces; +- one thread per source surface, with region evidence on the mutation request: + closer to Phase 2 and safer by default. + +Recommended default: keep threads source-bound; attach region evidence to +proposal classification. Promote a region to thread status only when it has a +stable source-authoritative projection. + +--- + +## 4. Veto windows vs synchronous gates + +A synchronous gate answers before write: permit, stage, or reject. Phase 2's +protected-target flow stages on fray, waits for principal approval, revalidates, +and applies. Nothing writes before authority clears. + +A veto window has two possible meanings: + +1. delayed apply: gates pass, the proposal becomes pending-visible, and the + daemon applies only after the window expires with no veto; +2. provisional apply: gates pass, the daemon applies immediately, and a later + veto triggers rollback. + +Recommended default: delayed apply. Provisional apply should be forbidden until +Val/Nova explicitly accept rollback semantics. Gate-4 fail-closed and Phase 4's +"no optimistic UI, no queued decisions" posture both prefer visible pending +state over speculative mutation. + +Delayed apply needs audit moments: + +- `proposal_submitted`: classification, diff hash, affected paths/regions, + predicate evidence, deadline; +- `proposal_window_opened`: gate evidence and veto deadline recorded; +- final `proposal_approved`, `proposal_vetoed`, `proposal_expired`, or + `proposal_rejected`: daemon revalidation result immediately before apply or + refusal. + +WARD-C6 matters by analogy: a time-separated, lossy or delayed boundary needs a +ledger. A proposal whose evidence cannot be replayed at deadline fails closed. +Future Cave surfaces should extend `ProposalView` only after daemon evidence +exists; until then unknown or stale veto state renders blocked under Phase 4 §4. + +--- + +## 5. Explicit non-goals + +- No general policy engine; README anti-goals and Phase 4 §6 bind. +- No client-side approval authority; Cave forwards, daemon decides. +- No revival of v0.1 TOML as normative format. +- No Phase-5 start and no changes to frozen Phase 0. +- No weakening of Gate 4; every path ends in live daemon re-materialization. +- No `.af` compatibility work; Phase 3 already decided `.weave` canonical plus + lossy one-way `.af` exporter. +- No broad runtime capability system; tool grants may classify proposals, but + runtime chamber authority remains separate. + +--- + +## 6. Open questions for Val/Nova + +1. Should approval semantics be `ApprovalPath`, separate from `Channel`? + Recommended default: yes; channels remain load axes. +2. Are veto windows delayed-apply or provisional-apply? + Recommended default: delayed apply only. +3. Do harness regions classify proposal ceremony only, or become threads? + Recommended default: classify first; promote only with stable source + projection. +4. Are invariants checked by deterministic extraction, identity probes, or both? + Recommended default: deterministic where possible; probes are Gate-3 + evidence, not sole auto authority. +5. Must RFC issues #3/#4 land before freeze? + Recommended default: yes for freeze, no for exploratory design. +6. Should Phase 5 extend Cave `ProposalView`, or wait for daemon policy? + Recommended default: daemon contract first, Cave extension second. +7. Preserve labels `auto`, `familiar_review`, `human_review`, `human_required`? + Recommended default: preserve display names, encode typed variants. +8. Add `proposal_window_opened` audit event? + Recommended default: yes if delayed apply ships. + +--- + +## 7. Rough phase shape, if Phase 5 opens + +Skeleton only; do not create these beads until Val/Nova open Phase 5. + +- `threads-uqx` — [EPIC] approval semantics over the Phase-2 Ward. +- `threads-uqx.1` — Sage/Nova gate: decide `ApprovalPath` vs `Channel` and veto + semantics. +- `threads-uqx.2` — RFC dependency check: issues #3/#4; freeze blocked unless + landed or waived. +- `threads-uqx.3` — Core type sketch: `ApprovalPath`, `VetoWindow`, + `ProposalClassification`, region evidence, audit event shape. +- `threads-uqx.4` — Predicate design: identity invariant + `PatternPredicate`; descriptor-derived proof. +- `threads-uqx.5` — Surface-region design: extractor predicates over + materialized diffs and Gate-4 replay. +- `threads-uqx.6` — Daemon integration design: classification, delayed-apply + scheduler, revalidation at deadline, append-only audit in `coven.sqlite3`. +- `threads-uqx.7` — Cave contract amendment after daemon read models exist; + add blocked fixtures for veto states. +- `threads-uqx.8` — Cody implementation lane: core crate and daemon tests. +- `threads-uqx.9` — Nova sign-off: RFC round-trip, Gate-4 fail-closed proof, + descriptor-not-authority review. +- `threads-uqx.10` — Val freeze: Phase-5 design frozen or rejected. + +Suggested gates: proposal review; Nova design gate; Val authority gate; RFC +amendment gate; implementation gate proving no apply bypass; Cave fail-closed +rendering gate; final Val freeze after Nova sign-off. + +--- + +## 8. Recommendation + +Open Phase 5 only if Val/Nova want to recover semantic approval behavior, not +just document why it was lost. + +Carry forward: approval paths as typed promotion ceremony over existing +channels; harness blocks as daemon-replayable surface-region classifiers; +protected invariants as predicate-authoritative identity checks. + +Leave dead: v0.1 stringly gate/block coupling; invariant strings as enforcement +objects; client-side or descriptor-based authority; provisional apply during +veto windows unless explicitly accepted. + +The conservative design is not to make Ward more general. It is to restore the +minimum semantic distinctions RFC-0001 still names while preserving the Phase-0 +spine: source-authoritative surfaces, predicate-authoritative patterns, +daemon-owned audit, and fail-closed Gate 4. From 6829e5d4fe57f8270df3cce2e68ca9d3bb66a772 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sat, 18 Jul 2026 05:27:18 -0500 Subject: [PATCH 02/25] spec(phase-4): align route-1 Returns with WeaveListView shape (threads-k9s) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cave implementation wraps route-1 data as { weaves, degraded } rather than overloading a WeaveSummary[] with degraded entries — the wrapped shape avoids any fabricated health fields on degraded familiars. Same-day alignment of the 2026-07-18 §2.7/R12 amendment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/PHASE-4-CAVE-SURFACES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/PHASE-4-CAVE-SURFACES.md b/specs/PHASE-4-CAVE-SURFACES.md index e4432f3..5c47191 100644 --- a/specs/PHASE-4-CAVE-SURFACES.md +++ b/specs/PHASE-4-CAVE-SURFACES.md @@ -258,7 +258,7 @@ never apply edits, never touch `pending/`, never write sqlite. | # | Route | Method | Returns | |---|---|---|---| -| 1 | `/api/weaves` | GET | `WeaveSummary[]` — optional `?familiar=` filter; degraded familiars surface per §2.7 + §4.R12 | +| 1 | `/api/weaves` | GET | `WeaveListView` — `{ "weaves": WeaveSummary[], "degraded": DegradedFamiliarView[] }` (§2.7); optional `?familiar=` filter; degraded familiars render per §4.R12 | | 2 | `/api/weaves/[id]` | GET | `WeaveDetail` | | 3 | `/api/threads/[id]` | GET | `ThreadView` | | 4 | `/api/threads/[id]/strands` | GET | `StrandView[]` (with `fray` diff blocks where blamed) | From b085fc81a1f623bddf8b6b56bf35418c405bb290 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sat, 18 Jul 2026 15:37:50 -0500 Subject: [PATCH 03/25] spec(phase-5): record Val+Nova decisions, open Phase 5 (threads-uqx) All 8 open questions resolved on Sage's recommended defaults: - ApprovalPath separate from Channel (load axis preserved) - Delayed apply only for veto windows (fail-closed posture) - Classify-first for harness regions - Deterministic + probe for invariants (not LLM-judge-only) - RFC #3/#4 blocker resolved (familiar-contract@2a5bb98) - Daemon contract before Cave ProposalView extension - Display labels preserved, typed variants internal - proposal_window_opened audit event lands with delayed apply Rename PHASE-5-PROPOSAL -> PHASE-5-APPROVAL-SEMANTICS to reflect open status. Co-authored-by: Nova --- ...ntics.md => PHASE-5-APPROVAL-SEMANTICS.md} | 85 ++++++++++++------- 1 file changed, 55 insertions(+), 30 deletions(-) rename specs/{PHASE-5-PROPOSAL-approval-semantics.md => PHASE-5-APPROVAL-SEMANTICS.md} (82%) diff --git a/specs/PHASE-5-PROPOSAL-approval-semantics.md b/specs/PHASE-5-APPROVAL-SEMANTICS.md similarity index 82% rename from specs/PHASE-5-PROPOSAL-approval-semantics.md rename to specs/PHASE-5-APPROVAL-SEMANTICS.md index 362ca9c..fb1b8d6 100644 --- a/specs/PHASE-5-PROPOSAL-approval-semantics.md +++ b/specs/PHASE-5-APPROVAL-SEMANTICS.md @@ -1,9 +1,11 @@ # PHASE-5 PROPOSAL — Approval semantics over the Phase-2 Ward -**Status:** PROPOSAL (draft — Phase 5 has not opened; phase gates per `AGENTS.md` still bind) +**Status:** PROPOSAL (decisions recorded — awaiting PR merge to open Phase 5) **Date:** 2026-07-18 +**Decision date:** 2026-07-18 **Bead:** `threads-uqx` **Authors:** Sage lane (drafted by coordinator session agent) +**Decided by:** Val Alexander + Nova (main session) --- @@ -265,38 +267,61 @@ exists; until then unknown or stale veto state renders blocked under Phase 4 §4 --- -## 6. Open questions for Val/Nova - -1. Should approval semantics be `ApprovalPath`, separate from `Channel`? - Recommended default: yes; channels remain load axes. -2. Are veto windows delayed-apply or provisional-apply? - Recommended default: delayed apply only. -3. Do harness regions classify proposal ceremony only, or become threads? - Recommended default: classify first; promote only with stable source - projection. -4. Are invariants checked by deterministic extraction, identity probes, or both? - Recommended default: deterministic where possible; probes are Gate-3 - evidence, not sole auto authority. -5. Must RFC issues #3/#4 land before freeze? - Recommended default: yes for freeze, no for exploratory design. -6. Should Phase 5 extend Cave `ProposalView`, or wait for daemon policy? - Recommended default: daemon contract first, Cave extension second. -7. Preserve labels `auto`, `familiar_review`, `human_review`, `human_required`? - Recommended default: preserve display names, encode typed variants. -8. Add `proposal_window_opened` audit event? - Recommended default: yes if delayed apply ships. +## 6. Decisions (Val + Nova, 2026-07-18) + +All eight questions resolved on Sage's recommended defaults. RFC dependency +blocker (issues #3/#4) resolved: both landed in OpenCoven/familiar-contract@2a5bb98 +and issues closed before this decision record was written. + +1. **`ApprovalPath` separate from `Channel`? → YES.** + Channel remains the load axis (why a thread is stressed). ApprovalPath is + the promotion ceremony. Conflating them would make Phase 0 load-axis + semantics ambiguous. `ApprovalPath` proceeds as a distinct type. + +2. **Veto windows: delayed-apply or provisional-apply? → DELAYED APPLY ONLY.** + Matches Gate-4 fail-closed posture and Phase 4 "no optimistic UI, no queued + decisions." Provisional apply + rollback is a distinct threat model; not + opening that door in Phase 5. + +3. **Harness regions: classify ceremony or become threads? → CLASSIFY FIRST.** + Region → thread promotion only when a stable source-authoritative projection + exists. Default: threads source-bound, region evidence on proposal + classification. No premature thread proliferation. + +4. **Invariants: deterministic extraction, identity probes, or both? + → DETERMINISTIC WHERE POSSIBLE; probes as Gate-3 evidence, not sole authority.** + Consistent with RFC-0001 rule that LLM-judge-only is forbidden for auto-tier. + Same logic applies to invariant checking. + +5. **Must RFC issues #3/#4 land before freeze? → MOOT; both landed.** + OpenCoven/coven-threads#3 and #4 closed 2026-07-18. Phase 5 freeze has no + RFC amendment blocker. + +6. **Cave `ProposalView` extension: daemon contract first or simultaneous? + → DAEMON CONTRACT FIRST.** + Cave extension only after daemon read models exist. Consistent with Phase 4 + pattern. + +7. **Preserve display labels `auto`, `familiar_review`, `human_review`, + `human_required`? → YES — display names preserved, typed variants internal.** + Daemon owns the typed `ApprovalPath` enum; Cave renders the human-readable + labels. No mental-model breakage. + +8. **Add `proposal_window_opened` audit event? → YES, when delayed apply ships.** + Minimal ledger entry for auditable veto windows. Connects to issue #5 + (`ward_audit` / `ApplyAudit`); that lane (Cody) can consume both in the same + schema pass. --- -## 7. Rough phase shape, if Phase 5 opens +## 7. Phase shape — Phase 5 is open -Skeleton only; do not create these beads until Val/Nova open Phase 5. +Phase 5 opened 2026-07-18 (Val + Nova decision). Beads are live. - `threads-uqx` — [EPIC] approval semantics over the Phase-2 Ward. -- `threads-uqx.1` — Sage/Nova gate: decide `ApprovalPath` vs `Channel` and veto - semantics. -- `threads-uqx.2` — RFC dependency check: issues #3/#4; freeze blocked unless - landed or waived. +- `threads-uqx.1` — ✅ RESOLVED: `ApprovalPath` separate from `Channel`; delayed + apply; classify-first for regions; deterministic+probe for invariants. +- `threads-uqx.2` — ✅ RESOLVED: RFC #3/#4 landed (familiar-contract@2a5bb98). - `threads-uqx.3` — Core type sketch: `ApprovalPath`, `VetoWindow`, `ProposalClassification`, region evidence, audit event shape. - `threads-uqx.4` — Predicate design: identity invariant @@ -318,10 +343,10 @@ rendering gate; final Val freeze after Nova sign-off. --- -## 8. Recommendation +## 8. Recommendation — executed -Open Phase 5 only if Val/Nova want to recover semantic approval behavior, not -just document why it was lost. +Phase 5 opened. Val and Nova confirmed: we want to recover semantic approval +behavior, not just document why it was lost. Carry forward: approval paths as typed promotion ceremony over existing channels; harness blocks as daemon-replayable surface-region classifiers; From 3fe72caf5b6524d5fca816adb2447045f84da486 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sat, 18 Jul 2026 15:41:13 -0500 Subject: [PATCH 04/25] spec(phase-5): fold Sage + Echo review into decision record (threads-uqx) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add substantive annotations from Sage and Echo review (2026-07-18): Decision #1: rationale callback — ApprovalPath is authoritative, Channel is derived; if enforcement rule shifts, revisit all eight decisions. Decision #2: audit implication — close event needs reason field (applied|vetoed|expired|superseded); delayed-apply without it is a post-hoc audit black hole. Decision #3: forward-only promotion — retroactive region projection corrupts authority trail with apparently-authored writes from before promotion. Decision #4: fail-closed on extraction ambiguity — not silent LLM fallback; ambiguity is an explicit ignored/blocked state. Decision #5: RFC provenance pointer added. Decision #6: Cave ProposalView labeled [DESIGNED, NOT SHIPPED] from the start. Decision #7: label-variant round-trip is daemon wire contract (load-time reject on drift); Cave has zero policy freedom over label strings. Decision #8: proposal_window_opened paired with close event; window is a first-class audit interval. threads-uqx.3 design constraints (Sage): evidence_replay_hash on ProposalClassification; VetoWindow.min_visible: Duration; label mapping as daemon wire contract. Co-authored-by: Sage Co-authored-by: Echo Co-authored-by: Nova --- specs/PHASE-5-APPROVAL-SEMANTICS.md | 34 +++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/specs/PHASE-5-APPROVAL-SEMANTICS.md b/specs/PHASE-5-APPROVAL-SEMANTICS.md index fb1b8d6..5f5fdb0 100644 --- a/specs/PHASE-5-APPROVAL-SEMANTICS.md +++ b/specs/PHASE-5-APPROVAL-SEMANTICS.md @@ -277,40 +277,62 @@ and issues closed before this decision record was written. Channel remains the load axis (why a thread is stressed). ApprovalPath is the promotion ceremony. Conflating them would make Phase 0 load-axis semantics ambiguous. `ApprovalPath` proceeds as a distinct type. + *Follows the source-authoritative-enforcement rule: ApprovalPath is + authoritative; Channel is a derived load descriptor. Never gate on Channel. + If this rule shifts, revisit all eight decisions.* (Echo, Sage 2026-07-18) 2. **Veto windows: delayed-apply or provisional-apply? → DELAYED APPLY ONLY.** Matches Gate-4 fail-closed posture and Phase 4 "no optimistic UI, no queued decisions." Provisional apply + rollback is a distinct threat model; not opening that door in Phase 5. + *Audit implication: the veto window close event needs an explicit reason + field — `applied | vetoed | expired | superseded`. Without it, + delayed-apply is a post-hoc audit black hole. Confirm `ApplyAudit` (issue + #5) captures the close event, not just the apply event. (Echo 2026-07-18)* 3. **Harness regions: classify ceremony or become threads? → CLASSIFY FIRST.** Region → thread promotion only when a stable source-authoritative projection exists. Default: threads source-bound, region evidence on proposal classification. No premature thread proliferation. + *Forward-only promotion: if a region reclassifies mid-session, promotion + applies forward only. Retroactive projection would corrupt the authority + trail with apparently-authored writes from before the promotion decision. + (Echo 2026-07-18)* 4. **Invariants: deterministic extraction, identity probes, or both? → DETERMINISTIC WHERE POSSIBLE; probes as Gate-3 evidence, not sole authority.** Consistent with RFC-0001 rule that LLM-judge-only is forbidden for auto-tier. Same logic applies to invariant checking. + *When deterministic extraction fails or is ambiguous, default is + fail-closed (no promotion) — not silent fallback to LLM judgment. + Ambiguity is an explicit ignored/blocked state. (Echo 2026-07-18)* 5. **Must RFC issues #3/#4 land before freeze? → MOOT; both landed.** OpenCoven/coven-threads#3 and #4 closed 2026-07-18. Phase 5 freeze has no - RFC amendment blocker. + RFC amendment blocker. See familiar-contract@2a5bb98 for the resolved shape; + issues #3/#4 preserved in git history for provenance. 6. **Cave `ProposalView` extension: daemon contract first or simultaneous? → DAEMON CONTRACT FIRST.** Cave extension only after daemon read models exist. Consistent with Phase 4 - pattern. + pattern. Cave ProposalView for Phase 5 starts as `[DESIGNED, NOT SHIPPED]` + until release evidence exists. (Echo 2026-07-18) 7. **Preserve display labels `auto`, `familiar_review`, `human_review`, `human_required`? → YES — display names preserved, typed variants internal.** Daemon owns the typed `ApprovalPath` enum; Cave renders the human-readable labels. No mental-model breakage. + *Label-variant round-trip must be a daemon wire contract, not Cave + convention. Daemon emits `{variant, label, veto_deadline}`; Cave has zero + policy freedom over label strings. Daemon should reject at load if a variant + has no corresponding display label or vice versa. (Sage 2026-07-18)* 8. **Add `proposal_window_opened` audit event? → YES, when delayed apply ships.** Minimal ledger entry for auditable veto windows. Connects to issue #5 (`ward_audit` / `ApplyAudit`); that lane (Cody) can consume both in the same schema pass. + *Also requires a corresponding close event with reason field (see decision + #2 audit note). The window is a first-class audit interval, not a gap.* --- @@ -324,6 +346,14 @@ Phase 5 opened 2026-07-18 (Val + Nova decision). Beads are live. - `threads-uqx.2` — ✅ RESOLVED: RFC #3/#4 landed (familiar-contract@2a5bb98). - `threads-uqx.3` — Core type sketch: `ApprovalPath`, `VetoWindow`, `ProposalClassification`, region evidence, audit event shape. + *Design constraints from Sage/Echo (2026-07-18) to incorporate:* + *(a) `evidence_replay_hash` on `ProposalClassification` — delayed-apply* + *revalidation at deadline must prove it replays the same evidence that* + *opened the window (WARD-C7 generalizes: evidence must survive the time gap).* + *(b) `VetoWindow.min_visible: Duration` — veto window is only fail-closed if* + *pending state was actually visible long enough for a human to act on it* + *(same shape as two-compaction contract).* + *(c) Label mapping as daemon wire contract with load-time reject on drift.* - `threads-uqx.4` — Predicate design: identity invariant `PatternPredicate`; descriptor-derived proof. - `threads-uqx.5` — Surface-region design: extractor predicates over From 19436aa206867b74dcc4c5228fe21b846ef7186d Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sat, 18 Jul 2026 16:24:35 -0500 Subject: [PATCH 05/25] feat(audit): add ApplyAudit event type, detail column, migration SQL (v0.1.4) Closes coven-threads#5. Unblocks coven#414. Changes: - AuditEventType::ApplyAudit variant + apply_audit CHECK tag - WardAuditRecord.detail: nullable JSON column for event-specific payload - WardAuditRecord::for_apply() constructor - apply_prev_sha256_hex() / apply_bytes_written() decode helpers - WARD_AUDIT_MIGRATION_V014_SQL: transaction rebuild for v0.1.3 stores - APPLY_AUDIT_DETAIL_KEY_PREV / _BYTES stable constants - Exhaustiveness test extended; 3 new tests (shape, roundtrip, migration) - Workspace version bumped 0.1.3 -> 0.1.4 - CHANGELOG.md created Design: prev_sha256+bytes_written ride in detail JSON (Option A); migration exports SQL constant, no rusqlite dep added (Option B-lite). Bead: threads-3ym --- .beads/interactions.jsonl | 13 ++ CHANGELOG.md | 27 +++ Cargo.toml | 2 +- crates/coven-threads-core/src/audit.rs | 291 ++++++++++++++++++++++++- 4 files changed, 331 insertions(+), 2 deletions(-) create mode 100644 CHANGELOG.md diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index c00e752..359a09c 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -23,3 +23,16 @@ {"id":"int-826011e5ee8ee8d205e7e838693c8fa1","kind":"field_change","created_at":"2026-07-17T14:52:45.121813Z","actor":"Val Alexander","issue_id":"threads-986.17.9","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Phase 4 FREEZE executed 2026-07-17 (Val approval recorded on this bead 2026-07-15; gate:val-phase4-freeze satisfied). Epic AC verified: (1) all four surfaces live+functional behind fixtures adapter (daemon flip filed as threads-v3g) — /weaves rail, thread pane, strand inspector, /proposals approval, merged in coven-cave PR #3223; (2) metaphor-referent binding preserved — Nova sign-off .17.8 closed 2026-07-17 with all three coherence checks PASS, no findings; source-pin suites hold referents in CI; (3) Charm pass complete (.17.7); (4) proposal flow E2E over real staged fixtures (.17.6); contract spec landed on main via PR #1 (a55d99af1). Children .17.1-.17.8 + .17.10 all closed with evidence. goals.md updated."}} {"id":"int-a0bacc8f5edaee7a580c1a218a43157a","kind":"field_change","created_at":"2026-07-17T14:52:46.121787Z","actor":"Val Alexander","issue_id":"threads-986.17","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"EPIC SUMMARY — Phase 4 Cave UX complete. Built: PHASE-4-CAVE-SURFACES.md contract (PR #1, a55d99af1); read plumbing with dual adapters (fixtures-first default, daemon behind same interface); weave rail with tension rollup (snapped>frayed>unknown>stale>holds); thread pane Holds/Frayed/Snapped with strands + channel bindings; strand inspector with tri-state diff + R7 lineage; proposal approval flow (staged DegradeToProposal, approve/reject daemon-forwarders, fail-closed everywhere) — all merged via coven-cave PR #3223 (645a1894 head, 18 checks green, test:app 740/740). Gates: Charm voice pass .17.7; Nova coherence sign-off .17.8 (PASS, no findings); Val UX-accept .17.10; Val freeze approval .17.9. Rendering rules R1-R11 enforced fail-closed; PatternDescriptor never rendered as enforcement; Gate-4 unknown states render Blocked. Follow-up: threads-v3g (daemon endpoints + adapter flip). Screenshots 01-07 at ~/.openclaw/workspace/sage/goals/phase4-artifacts/."}} {"id":"int-8181a70aeede0f1b0c58cd5fa4521a68","kind":"field_change","created_at":"2026-07-17T14:53:21.464406Z","actor":"Val Alexander","issue_id":"threads-986","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"PROGRAM EPIC COMPLETE — all four AC met. (1) Phases 0-4 delivered: v0.2 design freeze (.4); Phase 1 crate freeze (.18, coven-threads-core through v0.1.3); Phase 2 daemon integration freeze (.20, coven PR #382 merged f745117 — gate operational on main, + WARD-C6 compaction hook PR #393); Phase 3 portability freeze (.21, + jq4 surfaces map/.af exporter PR #2); Phase 4 Cave UX freeze (.17, coven-cave PR #3223). (2) RFC-0001 §5 conformance mirror 14/14 (.11). (3) Daemon imports validator: tier-0 edits route through validate_fail_closed, pinned v0.1.3. (4) Portability ships with C7 survives-serialization enforced (c7_roundtrip suite; C1-C7 jointly canonical in grimoire §9 per .12). 30 child beads closed with evidence. Remaining non-child follow-up: threads-v3g (daemon endpoints + adapter flip)."}} +{"id":"int-66ac26cbefee070c08321748420bad15","kind":"field_change","created_at":"2026-07-17T19:21:41.010774Z","actor":"Val Alexander","issue_id":"threads-eql","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"MERGED coven PR #409 (a9724b09), all checks green. ward_manifest baselines advance on every committed protected write (store_baseline upsert + advance_surface_baseline post-apply); refusals keep pending files and append proposal-scoped audit rows. Independently live-verified on the PR branch with a hermetic daemon over the real socket: baseline hash advanced after approve; the previously-broken second cycle (drift->stage->restore-to-advanced-baseline->approve) applied 200; refused approve kept file + audited and retry stayed viable. Clarified: tier-0 POST /edits is HeldForCoherence by design — approve is the sole tier-0 commit point."}} +{"id":"int-8c4cb8b33a94f385798ab5be6d435d8d","kind":"field_change","created_at":"2026-07-17T19:21:42.881808Z","actor":"Val Alexander","issue_id":"threads-v3g","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"COMPLETE, both sides merged. Daemon: coven PR #408 (85566ba) GET /api/v1/threads/weaves + POST /api/v1/threads/proposals/{id}/{approve|reject} per spec §3/§3.7, + P1 follow-up PR #409 (a9724b09, threads-eql) making multi-cycle approves work. Cave: coven-cave PR #3362 (3db8547c8) — daemon adapter is the DEFAULT (spec §1 fixtures-first condition lapsed; COVEN_THREADS_ADAPTER=fixtures opts out), daemon semantic refusals surface verbatim (proposal-refused/not-found/proposal-corrupt, 409/404) instead of transport-shaped 503. Live-verified end to end through the real socket with a hermetic daemon: reads normalize, staging, drift refusal, approve apply, corrupt guard, unknown-id fail-closed. Validation: coven gates green; cave test:app 758/758 + typecheck on merged head. Stowaway Desktop commit rescued to rescue/val-ui-tweaks-2026-07-17 (needs its own PR)."}} +{"id":"int-be6d3f843b0a440d8eb1d1c81d475d32","kind":"field_change","created_at":"2026-07-18T07:02:18.244791Z","actor":"Val Alexander","issue_id":"threads-vmf","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"MERGED coven PR #418 (b98504cd, all checks green incl. windows after cfg fix 2001da04). Root cause: handler Err propagated out of threads_weaves_response on the first WardConfig::load failure and daemon::handle_http_stream ?-returned before writing any HTTP bytes (empty reply, curl exit 52). Fix: per-familiar isolation (log to stderr + daemon-recovery.log, skip, continue — never abort fleet read) and handle_http_stream converts api handler errors to JSON 500 envelopes for every endpoint. Independently live-verified on the PR branch: fleet of healthy sage + Val's REAL v0.1-dialect nova ward.toml returns 200 with sage listed. Tests: threads_weaves_skips_malformed_ward_without_aborting_fleet_read, handle_http_stream_converts_api_handler_error_to_json_500. Surfacing of malformed-config familiars in the UI deferred to threads-v60 migration lane."}} +{"id":"int-b1b336460d93e610124263159faeada3","kind":"field_change","created_at":"2026-07-18T07:02:19.14866Z","actor":"Val Alexander","issue_id":"threads-wro","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"MERGED coven-cave PR #3407 (d1edf3e6, 19/19 checks). preferences-store + theme-store suites now set a hermetic temp COVEN_HOME so cave-home reconciliation stops walking the real ~/.coven legacy symlinks against temp canonical overrides. Verified locally on a migrated machine: both suites + backdrop/migration siblings green; remaining test:api failure is the separately-tracked pre-existing tts flake (cave-k7i3 pattern)."}} +{"id":"int-b0352cdd9a72fadc88de97b11bb2f4b9","kind":"field_change","created_at":"2026-07-18T08:11:35.780927Z","actor":"Val Alexander","issue_id":"threads-v60","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"COMPLETE. Command: coven ward migrate merged in coven PR #419 (74b09cd1, all checks green incl. windows after TOML-escape fixture fix 023dc5bb); shipping in v0.1.5 (signed tag pushed, release pipeline running). Fleet migration APPLIED on Val's machine 2026-07-18 with Val's decisions (SSH signing-key fingerprint via runtime arg; protected->tier 0 + protected_surface; editable->tier 2 Logged; backup-only legacy disposition): all nine familiars migrated with byte-exact ward.toml.v01.bak backups; thoth additionally needed a familiars.toml workspace= line (added, backup kept). Live verification post-kickstart: GET /api/v1/threads/weaves HTTP 200 with nine weaves, 4-8 threads each, all tensions Holds, all coherence Coherent — the authority-boundary protection layer is now live over real familiar memory. v0.1 approval_tiers/harness_blocks/invariants semantics intentionally not carried (no Phase-2 equivalent; preserved in backups)."}} +{"id":"int-7c8ce4baece71955acd514e5c34b1e12","kind":"field_change","created_at":"2026-07-18T08:24:30.113404Z","actor":"Val Alexander","issue_id":"threads-2q5","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"MERGED coven PR #393 2026-07-15 (daemon main; verified via git log 04e0ce7 ancestor of main). WARD-C6 CompactionLedger events append in store::vacuum_store_path with pre/post thread commitments; tests vacuum_appends_compaction_ledger_for_warded_surface, vacuum_without_warded_surfaces_appends_no_compaction_ledger, compaction_ledger_keeps_ward_audit_append_only. Full evidence in bead comments. Closing stale in_progress claim from coordinator session."}} +{"id":"int-45fee6c5015cd238bdf2b4f49197d1c5","kind":"field_change","created_at":"2026-07-18T08:24:30.930406Z","actor":"Val Alexander","issue_id":"threads-jq4","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"MERGED coven-threads PR #2 2026-07-15 (5b4a51a on main), tag v0.1.3 cut at that commit and coven pin bumped (30474c3). Surfaces content map + export_af lossy one-way Letta exporter, no .af import path, C7 conformance in tests/c7_roundtrip.rs. Full evidence in bead comments. Closing stale in_progress claim from coordinator session."}} +{"id":"int-96e600594080674425edaa10b7a4934c","kind":"field_change","created_at":"2026-07-18T12:03:18.346804Z","actor":"Val Alexander","issue_id":"threads-k9s","extra":{"field":"status","new_value":"open","old_value":"in_progress"}} +{"id":"int-3992491181076736d0b8c9d8072dd308","kind":"field_change","created_at":"2026-07-18T12:03:18.909381Z","actor":"Val Alexander","issue_id":"threads-3xd","extra":{"field":"status","new_value":"open","old_value":"in_progress"}} +{"id":"int-019bebf2d54257621bcb1e7d3bf06f74","kind":"field_change","created_at":"2026-07-18T12:03:19.445615Z","actor":"Val Alexander","issue_id":"threads-uqx","extra":{"field":"status","new_value":"open","old_value":"in_progress"}} +{"id":"int-1805bcc20e8bca84ac7425b1877298a9","kind":"field_change","created_at":"2026-07-18T14:42:35.772492Z","actor":"Val Alexander","issue_id":"threads-k9s","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"COMPLETE, both halves merged. Spec: §2.7 DegradedFamiliarView + §4.R12 + route-1 WeaveListView shape (coven-threads main 6476a67 + 6829e5d). Daemon: coven#422 merged e04a7af8 (reopened after accidental close; branch restored from verified head e5dd8fc5) — degraded entries with const reason + sanitized single-line errors, recovery-log breadcrumbs kept, no-ward familiars skipped. Cave: coven-cave#3415 merged 037bca078 — WeaveListView {weaves, degraded}, R12 blocked rail rows with exact copy 'ward unreadable — protection not verifiable', traceForDegradedFamiliar drawer, degraded fixture, forward-compat unknown reasons; typecheck + test:app 774/774. Fail-visible gap from threads-vmf closed end to end."}} +{"id":"int-c49d97f567aa70cc2b8d289e925f4114","kind":"field_change","created_at":"2026-07-18T14:45:05.028849Z","actor":"Val Alexander","issue_id":"threads-0ip","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"MERGED coven PR #426 (7ad55434), all checks green. Root causes fixed, not papered over: (1) windows pty_runner codex_json_batch_shim flake — side-effect files (args.txt/stdin.txt) were read before reliably visible on windows; now condition-based read_side_effect_file polling with deadline; (2) ubuntu smoke session-kill timeout — record_session_exit's natural-exit path could overwrite a concurrent 'killed' status; now update_session_status_if_current CAS (UPDATE ... WHERE status='running') with unit test conditionally_updates_session_status_only_from_expected_current_status; daemon.rs exit recorder uses it. Coordinator-reviewed the product diff. No assertions weakened, no quarantines needed."}} +{"id":"int-eefa76ec3968a5d984f3da51e113add6","kind":"field_change","created_at":"2026-07-18T21:24:21.080221Z","actor":"Val Alexander","issue_id":"threads-3ym","extra":{"field":"status","new_value":"in_progress","old_value":"open"}} diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..71ba5e2 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,27 @@ +# Changelog + +All notable changes to `coven-threads-core` are documented here. + +## [0.1.4] — unreleased + +### Added + +- `AuditEventType::ApplyAudit` variant + `"apply_audit"` tag in `WARD_AUDIT_SCHEMA_SQL`'s CHECK constraint (coven-threads#5). + - Enables the coven daemon to persist Gate-4 Tier-2 applied-write records into `ward.audit` (downstream: coven#414). +- `WardAuditRecord::detail` field — nullable JSON column for event-type-specific payload. + - For `ApplyAudit` events: `{"prev_sha256":"","bytes_written":N}`. +- `WardAuditRecord::for_apply(...)` constructor — builds the audit row for a Gate-4 applied write. +- `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. +- `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. +- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). SQLite cannot `ALTER` a CHECK; the daemon should run this migration at startup when `sqlite_master` shows the old schema (i.e. no `apply_audit` in the `ward_audit` DDL). +- Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. +- New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `migration_sql_contains_apply_audit_and_detail_column`. + +### Design notes + +- **Column approach (Option A):** `diff_hash` carries `next_sha256`; `prev_sha256` + `bytes_written` ride in the new `detail` TEXT column as JSON. This avoids a second table rebuild by not adding typed hash columns while keeping the data query-accessible via SQLite JSON functions. +- **Migration approach (Option B-lite):** exports `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can perform a quiet guarded rebuild on upgrade; does not add a rusqlite dependency to coven-threads-core. + +## [0.1.3] — prior release + +Initial audit module with seven `AuditEventType` variants (`proposal_submitted`, `proposal_approved`, `proposal_rejected`, `proposal_vetoed`, `ward_updated`, `validation_verdict`, `compaction_ledger`). diff --git a/Cargo.toml b/Cargo.toml index e1c41ea..633c4b6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["crates/coven-threads-core"] [workspace.package] -version = "0.1.0" +version = "0.1.4" edition = "2021" rust-version = "1.75" license = "MIT" diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index fab7fc0..52192cf 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -14,6 +14,33 @@ //! //! WARD-C6 (compaction ledger appends to `ward.audit`) rides the same table: //! `AuditEventType::CompactionLedger`. +//! +//! **Gate-4 applied writes** are also auditable here via +//! `AuditEventType::ApplyAudit` (see §3.4, coven-threads#5). +//! +//! ## Schema versioning and migration +//! +//! `WARD_AUDIT_SCHEMA_SQL` uses `CREATE TABLE IF NOT EXISTS`, so new empty +//! stores automatically receive the current DDL including the `apply_audit` +//! CHECK tag. Stores created against v0.1.3 (which lacked the `apply_audit` +//! CHECK entry) must be rebuilt: the daemon should run +//! `WARD_AUDIT_MIGRATION_V014_SQL` once at startup when it detects the old +//! CHECK text is still in the schema. That migration rebuilds `ward_audit` +//! in a single transaction (create, copy, drop, rename) — SQLite cannot +//! `ALTER` a CHECK constraint. +//! +//! ## Where content hashes ride for applied writes +//! +//! `WardAuditRecord::diff_hash` carries `next_sha256` (the post-write content +//! hash, matching the RFC-0001 §5.6 `diff_hash` semantic for verdict rows). +//! The complementary `prev_sha256` and `bytes_written` ride in `detail` as a +//! compact JSON object `{"prev_sha256":"","bytes_written":N}`. This +//! choice: +//! - avoids new columns that would require another SQLite table rebuild; +//! - keeps `diff_hash` as the canonical single-hash field (consistent with +//! verdict rows where it holds the proposal diff hash); +//! - makes `prev_sha256` query-accessible via SQLite JSON functions when +//! needed (`json_extract(detail, '$.prev_sha256')`). use serde::{Deserialize, Serialize}; use time::OffsetDateTime; @@ -22,6 +49,11 @@ use crate::channel::Channel; use crate::ids::{FamiliarId, ProposalId, SurfaceId, ThreadId, WriterId}; use crate::validate::{MutationRequest, RejectReason, Verdict}; +/// Stable JSON key for the `detail` field of an `apply_audit` row; +pub const APPLY_AUDIT_DETAIL_KEY_PREV: &str = "prev_sha256"; +/// Stable JSON key for the bytes-written count in an `apply_audit` detail. +pub const APPLY_AUDIT_DETAIL_KEY_BYTES: &str = "bytes_written"; + /// Event types recorded in `ward.audit`. /// /// The first five are RFC-0001 §5.6's named set, verbatim. The last two are the @@ -44,6 +76,18 @@ pub enum AuditEventType { ValidationVerdict, /// WARD-C6 compaction ledger entry (§3.3, inherited by reference). CompactionLedger, + /// A Gate-4 Tier-2 write was applied and its content snapshot recorded + /// (coven-threads#5, coven#414). + /// + /// Field mapping for this event type: + /// - `diff_hash` — `next_sha256` (post-write content hash of the surface). + /// - `detail` — JSON `{"prev_sha256":"","bytes_written":N}`; + /// `prev_sha256` is the pre-write content hash, `bytes_written` is the + /// number of bytes written to the surface. + /// - `tier` — typically `"tier_2"` (logged-write tier). + /// - `decision` — `"applied"` (the write completed successfully). + /// - `files_touched` — the resolved surface ids that were written. + ApplyAudit, } impl AuditEventType { @@ -57,12 +101,17 @@ impl AuditEventType { AuditEventType::WardUpdated => "ward_updated", AuditEventType::ValidationVerdict => "validation_verdict", AuditEventType::CompactionLedger => "compaction_ledger", + AuditEventType::ApplyAudit => "apply_audit", } } } /// One row of `ward.audit` (RFC-0001 §5.6 field set). /// +/// For `ApplyAudit` events, the extra per-apply fields (`prev_sha256`, +/// `bytes_written`) are encoded in the `detail` field as a JSON object (see +/// module-level docs and [`APPLY_AUDIT_DETAIL_KEY_PREV`]). +/// /// `ward_hash` is RFC-0001 §5.6's audit-log field; at this layer it is the /// `weave_hash` of the weave the verdict was issued against — the commitment /// binding "which authority state decided this." @@ -86,7 +135,16 @@ pub struct WardAuditRecord { /// Who approved/decided, when a principal or tier actor is involved. pub approver: Option, /// Hash of the proposed diff, when the event carries one. + /// + /// For `ApplyAudit` events this is the `next_sha256` — the post-write + /// content hash of the surface. pub diff_hash: Option>, + /// Opaque JSON detail payload, event-type–specific. + /// + /// For `ApplyAudit` events: `{"prev_sha256":"","bytes_written":N}`. + /// Absent for all other event types (non-`ApplyAudit` rows leave this + /// `None`). + pub detail: Option, /// The surfaces this event touches. pub files_touched: Vec, /// The channel the triggering request arrived on, if any (§2.4). @@ -137,6 +195,7 @@ impl WardAuditRecord { decision, approver: None, diff_hash: None, + detail: None, files_touched: vec![request.surface.clone()], channel: Some(request.channel), thread_id, @@ -144,6 +203,82 @@ impl WardAuditRecord { decided_at, } } + + /// Build the audit row for a Gate-4 Tier-2 applied write (coven-threads#5). + /// + /// `next_hash` is the post-write content hash (SHA-256) of the surface; + /// it rides in `diff_hash` matching the single-hash pattern used for + /// verdict rows. + /// + /// `prev_hash` (pre-write content hash) and `bytes_written` are encoded in + /// `detail` as `{"prev_sha256":"","bytes_written":N}` because adding + /// typed columns would require an immediate SQLite table rebuild in all + /// consumers. + /// + /// `weave_hash` is the Ward/weave version the Gate-4 decision was made + /// against (RFC-0001 §5.6 `ward_hash`). + #[allow(clippy::too_many_arguments)] + pub fn for_apply( + familiar_id: FamiliarId, + weave_hash: &[u8], + surface: SurfaceId, + tier: &str, + prev_hash: Option<&[u8]>, + next_hash: Option<&[u8]>, + bytes_written: u64, + channel: Option, + submitted_at: OffsetDateTime, + decided_at: OffsetDateTime, + ) -> Self { + let prev_hex = prev_hash.map(bytes_to_hex); + let detail = serde_json::json!({ + APPLY_AUDIT_DETAIL_KEY_PREV: prev_hex.unwrap_or_default(), + APPLY_AUDIT_DETAIL_KEY_BYTES: bytes_written, + }) + .to_string(); + Self { + event_type: AuditEventType::ApplyAudit, + proposal_id: None, + familiar_id, + ward_version: None, + ward_hash: weave_hash.to_vec(), + tier: Some(tier.to_string()), + decision: "applied".to_string(), + approver: None, + diff_hash: next_hash.map(|h| h.to_vec()), + detail: Some(detail), + files_touched: vec![surface], + channel, + thread_id: None, + submitted_at, + decided_at, + } + } + + /// Decode `prev_sha256` from the `detail` JSON for `ApplyAudit` rows. + /// + /// Returns `None` for non-`ApplyAudit` events or if the field is missing. + pub fn apply_prev_sha256_hex(&self) -> Option { + let detail = self.detail.as_deref()?; + let v: serde_json::Value = serde_json::from_str(detail).ok()?; + v.get(APPLY_AUDIT_DETAIL_KEY_PREV) + .and_then(|x| x.as_str()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_string()) + } + + /// Decode `bytes_written` from the `detail` JSON for `ApplyAudit` rows. + /// + /// Returns `None` for non-`ApplyAudit` events or if the field is missing. + pub fn apply_bytes_written(&self) -> Option { + let detail = self.detail.as_deref()?; + let v: serde_json::Value = serde_json::from_str(detail).ok()?; + v.get(APPLY_AUDIT_DETAIL_KEY_BYTES).and_then(|x| x.as_u64()) + } +} + +fn bytes_to_hex(bytes: &[u8]) -> String { + bytes.iter().map(|b| format!("{b:02x}")).collect() } fn reject_tag(reason: &RejectReason) -> &'static str { @@ -163,13 +298,92 @@ fn reject_tag(reason: &RejectReason) -> &'static str { /// Idempotent (`IF NOT EXISTS` throughout) so the daemon can apply it at /// startup. Append-only is enforced *in the store*: UPDATE and DELETE abort via /// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). +/// DDL migration for v0.1.3 → v0.1.4: rebuilds `ward_audit` so its CHECK +/// constraint includes the new `apply_audit` tag. +/// +/// SQLite cannot `ALTER` a CHECK constraint on an existing table. This SQL +/// performs a safe swap in one transaction: +/// 1. Creates `ward_audit_new` with the updated CHECK (plus `detail` column). +/// 2. Copies all existing rows (NULLing `detail` for old rows). +/// 3. Drops the old table. +/// 4. Renames the new table into place. +/// 5. Re-creates indexes and append-only triggers. +/// +/// **Run condition:** execute this only when +/// `SELECT sql FROM sqlite_master WHERE type='table' AND name='ward_audit'` +/// returns a string that does NOT contain the literal `'apply_audit'` — i.e. +/// the store was created against v0.1.3. Idempotent if re-run on a v0.1.4 +/// store because `ward_audit_new` won't contain `apply_audit` in that schema. +/// The daemon should gate this migration on the presence/absence of the tag +/// in `sqlite_master`. +pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = r#" +BEGIN; + +CREATE TABLE IF NOT EXISTS ward_audit_new ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL CHECK (event_type IN ( + 'proposal_submitted','proposal_approved','proposal_rejected', + 'proposal_vetoed','ward_updated','validation_verdict', + 'compaction_ledger','apply_audit')), + proposal_id TEXT, + familiar_id TEXT NOT NULL, + ward_version TEXT, + ward_hash BLOB NOT NULL, + tier TEXT, + decision TEXT NOT NULL, + approver TEXT, + diff_hash BLOB, + detail TEXT, + files_touched TEXT NOT NULL, + channel TEXT, + thread_id TEXT, + submitted_at TEXT NOT NULL, + decided_at TEXT NOT NULL, + recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) +); + +INSERT INTO ward_audit_new ( + id, event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, detail, files_touched, channel, + thread_id, submitted_at, decided_at, recorded_at +) +SELECT + id, event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, NULL, files_touched, channel, + thread_id, submitted_at, decided_at, recorded_at +FROM ward_audit; + +DROP TABLE ward_audit; +ALTER TABLE ward_audit_new RENAME TO ward_audit; + +CREATE INDEX IF NOT EXISTS ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); +CREATE INDEX IF NOT EXISTS ward_audit_event_idx ON ward_audit (event_type, recorded_at); + +CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_update +BEFORE UPDATE ON ward_audit +BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); +END; + +CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_delete +BEFORE DELETE ON ward_audit +BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); +END; + +COMMIT; +"#; + +/// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). +/// +/// See module docs for migration notes. pub const WARD_AUDIT_SCHEMA_SQL: &str = r#" CREATE TABLE IF NOT EXISTS ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_type TEXT NOT NULL CHECK (event_type IN ( 'proposal_submitted','proposal_approved','proposal_rejected', 'proposal_vetoed','ward_updated','validation_verdict', - 'compaction_ledger')), + 'compaction_ledger','apply_audit')), proposal_id TEXT, familiar_id TEXT NOT NULL, ward_version TEXT, @@ -178,6 +392,7 @@ CREATE TABLE IF NOT EXISTS ward_audit ( decision TEXT NOT NULL, approver TEXT, diff_hash BLOB, + detail TEXT, -- event-type-specific JSON; see module docs files_touched TEXT NOT NULL, -- JSON array of surface ids channel TEXT, thread_id TEXT, @@ -266,6 +481,7 @@ mod tests { AuditEventType::WardUpdated, AuditEventType::ValidationVerdict, AuditEventType::CompactionLedger, + AuditEventType::ApplyAudit, ] { assert!( WARD_AUDIT_SCHEMA_SQL.contains(&format!("'{}'", et.tag())), @@ -284,4 +500,77 @@ mod tests { assert!(WARD_AUDIT_SCHEMA_SQL.contains("ward_audit_append_only_delete")); assert!(WARD_AUDIT_SCHEMA_SQL.contains("append-only (RFC-0001 §5.6)")); } + + #[test] + fn for_apply_produces_correct_shape() { + let now = OffsetDateTime::now_utc(); + let familiar_id = FamiliarId::new(); + let surface = SurfaceId::new("SOUL.md"); + let prev = [0xaa_u8; 32]; + let next = [0xbb_u8; 32]; + + let record = WardAuditRecord::for_apply( + familiar_id, + &[0xcc; 32], + surface.clone(), + "tier_2", + Some(&prev), + Some(&next), + 42, + Some(Channel::Mutation), + now, + now, + ); + assert_eq!(record.event_type, AuditEventType::ApplyAudit); + assert_eq!(record.decision, "applied"); + assert_eq!(record.tier.as_deref(), Some("tier_2")); + assert_eq!(record.diff_hash, Some(next.to_vec())); + assert_eq!(record.files_touched, vec![surface]); + + // detail decoding helpers + let prev_hex = record.apply_prev_sha256_hex().unwrap(); + assert_eq!(prev_hex.len(), 64); + assert!(prev_hex.chars().all(|c| c.is_ascii_hexdigit())); + assert_eq!(record.apply_bytes_written(), Some(42)); + } + + #[test] + fn for_apply_roundtrips_json() { + let now = OffsetDateTime::now_utc(); + let record = WardAuditRecord::for_apply( + FamiliarId::new(), + &[1; 32], + SurfaceId::new("SOUL.md"), + "tier_2", + None, + None, + 0, + None, + now, + now, + ); + let json = serde_json::to_string(&record).unwrap(); + let back: WardAuditRecord = serde_json::from_str(&json).unwrap(); + assert_eq!(record, back); + } + + #[test] + fn migration_sql_contains_apply_audit_and_detail_column() { + assert!( + WARD_AUDIT_MIGRATION_V014_SQL.contains("'apply_audit'"), + "migration CHECK must contain apply_audit tag" + ); + assert!( + WARD_AUDIT_MIGRATION_V014_SQL.contains("detail"), + "migration must add detail column" + ); + assert!( + WARD_AUDIT_MIGRATION_V014_SQL.contains("BEGIN;"), + "migration must be a transaction" + ); + assert!( + WARD_AUDIT_MIGRATION_V014_SQL.contains("COMMIT;"), + "migration must be a transaction" + ); + } } From db20c623169a441c17efa8e37979d7e27fbd47b6 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sat, 18 Jul 2026 16:27:00 -0500 Subject: [PATCH 06/25] review(audit): add user_version ladder + explicit detail invariant doc Addresses two Sage review notes (neither was blocking): 1. WARD_AUDIT_MIGRATION_V014_SQL now stamps PRAGMA user_version = 14 inside the transaction. Future migrations gate on 'user_version < N' instead of substring-sniffing sqlite_master DDL. Migration doc-comment updated to name the gate rule and explain the intentional non-idempotence of the CREATE step (double-run fails loudly, not silently). 2. WardAuditRecord::detail doc-comment now names the full invariant for ApplyAudit rows (prev_sha256 hex 64 chars + bytes_written u64) and explicitly states other event types MAY leave it None or use their own documented shape. Turns the implicit decoder contract into a searchable spec. CHANGELOG updated. Tests still green, clippy clean. --- .beads/interactions.jsonl | 1 + CHANGELOG.md | 2 +- Cargo.lock | 2 +- crates/coven-threads-core/src/audit.rs | 35 ++++++++++++++++++-------- 4 files changed, 28 insertions(+), 12 deletions(-) diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index 359a09c..54e4c21 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -36,3 +36,4 @@ {"id":"int-1805bcc20e8bca84ac7425b1877298a9","kind":"field_change","created_at":"2026-07-18T14:42:35.772492Z","actor":"Val Alexander","issue_id":"threads-k9s","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"COMPLETE, both halves merged. Spec: §2.7 DegradedFamiliarView + §4.R12 + route-1 WeaveListView shape (coven-threads main 6476a67 + 6829e5d). Daemon: coven#422 merged e04a7af8 (reopened after accidental close; branch restored from verified head e5dd8fc5) — degraded entries with const reason + sanitized single-line errors, recovery-log breadcrumbs kept, no-ward familiars skipped. Cave: coven-cave#3415 merged 037bca078 — WeaveListView {weaves, degraded}, R12 blocked rail rows with exact copy 'ward unreadable — protection not verifiable', traceForDegradedFamiliar drawer, degraded fixture, forward-compat unknown reasons; typecheck + test:app 774/774. Fail-visible gap from threads-vmf closed end to end."}} {"id":"int-c49d97f567aa70cc2b8d289e925f4114","kind":"field_change","created_at":"2026-07-18T14:45:05.028849Z","actor":"Val Alexander","issue_id":"threads-0ip","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"MERGED coven PR #426 (7ad55434), all checks green. Root causes fixed, not papered over: (1) windows pty_runner codex_json_batch_shim flake — side-effect files (args.txt/stdin.txt) were read before reliably visible on windows; now condition-based read_side_effect_file polling with deadline; (2) ubuntu smoke session-kill timeout — record_session_exit's natural-exit path could overwrite a concurrent 'killed' status; now update_session_status_if_current CAS (UPDATE ... WHERE status='running') with unit test conditionally_updates_session_status_only_from_expected_current_status; daemon.rs exit recorder uses it. Coordinator-reviewed the product diff. No assertions weakened, no quarantines needed."}} {"id":"int-eefa76ec3968a5d984f3da51e113add6","kind":"field_change","created_at":"2026-07-18T21:24:21.080221Z","actor":"Val Alexander","issue_id":"threads-3ym","extra":{"field":"status","new_value":"in_progress","old_value":"open"}} +{"id":"int-17b3b70454cfdb7bbdaca849bc30ad2f","kind":"field_change","created_at":"2026-07-18T21:25:07.575924Z","actor":"Val Alexander","issue_id":"threads-3ym","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"PR #7 opened: https://github.com/OpenCoven/coven-threads/pull/7. 75+17+14 tests green, clippy clean. Awaiting merge + v0.1.4 tag."}} diff --git a/CHANGELOG.md b/CHANGELOG.md index 71ba5e2..bc34d24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ All notable changes to `coven-threads-core` are documented here. - `WardAuditRecord::for_apply(...)` constructor — builds the audit row for a Gate-4 applied write. - `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. -- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). SQLite cannot `ALTER` a CHECK; the daemon should run this migration at startup when `sqlite_master` shows the old schema (i.e. no `apply_audit` in the `ward_audit` DDL). +- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). Bumps `PRAGMA user_version = 14` inside the transaction so future migrations can gate on `user_version < N` instead of substring-sniffing `sqlite_master` DDL. The daemon should run this migration when `PRAGMA user_version < 14`. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `migration_sql_contains_apply_audit_and_detail_column`. diff --git a/Cargo.lock b/Cargo.lock index a5a514b..4caca61 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -67,7 +67,7 @@ checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" [[package]] name = "coven-threads-core" -version = "0.1.0" +version = "0.1.4" dependencies = [ "blake3", "serde", diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 52192cf..19ecd5f 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -141,9 +141,15 @@ pub struct WardAuditRecord { pub diff_hash: Option>, /// Opaque JSON detail payload, event-type–specific. /// - /// For `ApplyAudit` events: `{"prev_sha256":"","bytes_written":N}`. - /// Absent for all other event types (non-`ApplyAudit` rows leave this - /// `None`). + /// **Invariant for `ApplyAudit` rows:** `detail` MUST be a JSON object + /// containing exactly: + /// - `"prev_sha256"` — hex string (64 lower-case ASCII chars), the + /// pre-write SHA-256 of the surface content, or `""` when unknown. + /// - `"bytes_written"` — u64, number of bytes written to the surface. + /// + /// Other event types MAY leave `detail` as `None`, or use event-type- + /// specific shapes documented at their construction site. Callers MUST + /// NOT assume `detail` is non-null for non-`ApplyAudit` rows. pub detail: Option, /// The surfaces this event touches. pub files_touched: Vec, @@ -309,13 +315,17 @@ fn reject_tag(reason: &RejectReason) -> &'static str { /// 4. Renames the new table into place. /// 5. Re-creates indexes and append-only triggers. /// -/// **Run condition:** execute this only when -/// `SELECT sql FROM sqlite_master WHERE type='table' AND name='ward_audit'` -/// returns a string that does NOT contain the literal `'apply_audit'` — i.e. -/// the store was created against v0.1.3. Idempotent if re-run on a v0.1.4 -/// store because `ward_audit_new` won't contain `apply_audit` in that schema. -/// The daemon should gate this migration on the presence/absence of the tag -/// in `sqlite_master`. +/// **Run condition:** execute this only when `PRAGMA user_version` returns a +/// value less than `14` — i.e. the store was created before this migration. +/// `PRAGMA user_version` is `0` for stores that predate any explicit versioning +/// (all v0.1.3 stores). After this migration runs, `user_version` is `14` +/// (matching `0.1.4`), giving future migrations a clean integer ladder to gate +/// on instead of substring-sniffing `sqlite_master` DDL. +/// +/// Idempotent in the sense that if `user_version >= 14` the daemon must not +/// re-run it; the `ward_audit_new` table name is not guarded with `IF NOT +/// EXISTS`, so a double-run would fail on the `CREATE` step (intentional: +/// prevents silent data loss from a buggy migration controller). pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = r#" BEGIN; @@ -371,6 +381,11 @@ BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; +-- Stamp schema version so future migrations can gate on `PRAGMA user_version` +-- instead of substring-sniffing the CHECK DDL. Gate rule for callers: +-- run this migration when `PRAGMA user_version < 14`. +PRAGMA user_version = 14; + COMMIT; "#; From a46377952b97b3496bba9844fec322782256fe7d Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 21:57:08 -0500 Subject: [PATCH 07/25] docs: design ApplyAudit migration repair Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...-19-apply-audit-migration-repair-design.md | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md new file mode 100644 index 0000000..fac1043 --- /dev/null +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -0,0 +1,44 @@ +# ApplyAudit Migration Repair + +## Goal + +Repair PR #7 so upgrading a v0.1.3 `ward_audit` table to v0.1.4 cannot +silently erase immutable `detail` evidence, while keeping the change scoped to +the audit schema contract. + +## Design + +The v0.1.4 migration will first add the nullable `detail` column to the legacy +table, then rebuild the table and copy `detail` into the replacement table. +This gives genuine v0.1.3 rows a `NULL` detail value without hard-coding data +loss into the copy step. + +Fresh v0.1.4 schema initialization will stamp `PRAGMA user_version = 14`. +Running the legacy migration against a current schema will fail when it tries +to add an already-present `detail` column, before dropping or rewriting any +table. Re-running the migration after a successful upgrade will likewise fail +closed. + +## Tests + +Executable SQLite tests will cover: + +1. A representative v0.1.3 table upgrades successfully, preserves every + existing field, gains `detail`, accepts `apply_audit`, retains append-only + triggers, and ends at `user_version = 14`. +2. A fresh v0.1.4 schema starts at `user_version = 14` and preserves populated + `ApplyAudit` detail. +3. Applying the legacy migration to a current schema returns an error and + leaves the existing audit row unchanged. +4. Re-running the migration after a successful legacy upgrade returns an error + and leaves migrated rows unchanged. + +Tests will use an in-memory bundled SQLite dependency available only to the +crate's test target. + +## Scope + +This repair does not add daemon migration orchestration or change the +`WardAuditRecord` wire shape. The daemon remains responsible for choosing +schema initialization for new stores and the v0.1.4 migration for legacy +stores. From 290ca297038b8e442338d391a1c66a8f1ee96e77 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 22:27:47 -0500 Subject: [PATCH 08/25] docs: plan ApplyAudit migration repair Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...2026-07-19-apply-audit-migration-repair.md | 311 ++++++++++++++++++ 1 file changed, 311 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md new file mode 100644 index 0000000..5b80ebd --- /dev/null +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -0,0 +1,311 @@ +# ApplyAudit Migration Repair Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the v0.1.4 `ward_audit` migration preserve immutable evidence and fail closed when invoked against an already-current schema. + +**Architecture:** Keep migration ownership in `audit.rs`. The legacy migration adds `detail` before rebuilding and copies that column verbatim; the current schema stamps version 14. In-memory SQLite tests execute both schema paths and verify data preservation, append-only enforcement, versioning, and safe failure. + +**Tech Stack:** Rust 2021, SQLite, `rusqlite` test dependency, Cargo test. + +--- + +## File Map + +- Modify `crates/coven-threads-core/Cargo.toml` to add bundled SQLite for tests only. +- Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, stamp fresh schemas, and add executable migration tests. +- Modify `Cargo.lock` through Cargo after adding the test dependency. + +### Task 1: Add executable migration regressions + +**Files:** +- Modify: `crates/coven-threads-core/Cargo.toml` +- Modify: `crates/coven-threads-core/src/audit.rs:435-590` +- Modify: `Cargo.lock` + +- [ ] **Step 1: Add the test-only SQLite dependency** + +Append this section to `crates/coven-threads-core/Cargo.toml`: + +```toml +[dev-dependencies] +rusqlite = { version = "0.31", features = ["bundled"] } +``` + +- [ ] **Step 2: Add a representative legacy schema fixture** + +Inside `audit.rs`'s `tests` module, import `rusqlite::{params, Connection}` and add: + +```rust +const LEGACY_WARD_AUDIT_SCHEMA_SQL: &str = r#" +CREATE TABLE ward_audit ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL CHECK (event_type IN ( + 'proposal_submitted','proposal_approved','proposal_rejected', + 'proposal_vetoed','ward_updated','validation_verdict', + 'compaction_ledger')), + proposal_id TEXT, + familiar_id TEXT NOT NULL, + ward_version TEXT, + ward_hash BLOB NOT NULL, + tier TEXT, + decision TEXT NOT NULL, + approver TEXT, + diff_hash BLOB, + files_touched TEXT NOT NULL, + channel TEXT, + thread_id TEXT, + submitted_at TEXT NOT NULL, + decided_at TEXT NOT NULL, + recorded_at TEXT NOT NULL +); + +CREATE TRIGGER ward_audit_append_only_update +BEFORE UPDATE ON ward_audit +BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); +END; + +CREATE TRIGGER ward_audit_append_only_delete +BEFORE DELETE ON ward_audit +BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); +END; +"#; + +fn insert_audit_row(connection: &Connection, event_type: &str, detail: Option<&str>) -> i64 { + connection + .execute( + "INSERT INTO ward_audit ( + event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, detail, files_touched, + channel, thread_id, submitted_at, decided_at, recorded_at + ) VALUES ( + ?1, 'proposal-1', 'familiar-1', '0.1.4', X'0102', + 'tier_2', 'applied', 'principal:val', X'0304', ?2, '[\"SOUL.md\"]', + 'mutation', 'thread-1', '2026-07-19T00:00:00Z', + '2026-07-19T00:00:01Z', '2026-07-19T00:00:02Z' + )", + params![event_type, detail], + ) + .unwrap(); + connection.last_insert_rowid() +} +``` + +For the legacy test, insert its pre-migration row with a dedicated statement +that omits `detail`, because the v0.1.3 table does not have that column. + +- [ ] **Step 3: Add the failing fresh-schema version test** + +```rust +#[test] +fn fresh_schema_is_stamped_v014() { + let connection = Connection::open_in_memory().unwrap(); + connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + let version: i64 = connection + .query_row("PRAGMA user_version", [], |row| row.get(0)) + .unwrap(); + assert_eq!(version, 14); +} +``` + +- [ ] **Step 4: Add the failing current-schema preservation test** + +```rust +#[test] +fn legacy_migration_rejects_current_schema_without_erasing_detail() { + let connection = Connection::open_in_memory().unwrap(); + connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + let detail = r#"{"prev_sha256":"abcd","bytes_written":42}"#; + let id = insert_audit_row(&connection, "apply_audit", Some(detail)); + + let error = connection + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("current schema must reject the legacy migration"); + assert!(error.to_string().contains("duplicate column name: detail")); + let _ = connection.execute_batch("ROLLBACK;"); + + let preserved: String = connection + .query_row( + "SELECT detail FROM ward_audit WHERE id = ?1", + [id], + |row| row.get(0), + ) + .unwrap(); + assert_eq!(preserved, detail); +} +``` + +- [ ] **Step 5: Add the legacy-upgrade and rerun tests** + +The legacy-upgrade test must: + +1. Execute `LEGACY_WARD_AUDIT_SCHEMA_SQL`. +2. Insert a fully populated legacy row without `detail`. +3. Execute `WARD_AUDIT_MIGRATION_V014_SQL`. +4. Assert the row's original values remain unchanged and `detail IS NULL`. +5. Assert `PRAGMA user_version = 14`. +6. Insert an `apply_audit` row with populated `detail`. +7. Assert both append-only triggers reject `UPDATE` and `DELETE`. + +The rerun test must execute the migration successfully once, invoke it again, +assert the duplicate-`detail` error, roll back the failed transaction, and +confirm the original migrated row still exists unchanged. + +- [ ] **Step 6: Run the focused tests and confirm the regression** + +Run: + +```bash +cargo test -p coven-threads-core audit::tests -- --nocapture +``` + +Expected before the SQL repair: `fresh_schema_is_stamped_v014` fails with +`left: 0, right: 14`, and +`legacy_migration_rejects_current_schema_without_erasing_detail` fails because +the migration succeeds and replaces `detail` with `NULL`. + +### Task 2: Make the migration preserve evidence and fail closed + +**Files:** +- Modify: `crates/coven-threads-core/src/audit.rs:302-433` + +- [ ] **Step 1: Guard the legacy migration before destructive work** + +Immediately after `BEGIN;` in `WARD_AUDIT_MIGRATION_V014_SQL`, add: + +```sql +ALTER TABLE ward_audit ADD COLUMN detail TEXT; +``` + +This succeeds only for the actual v0.1.3 shape. A current schema or a repeated +upgrade fails before `DROP TABLE ward_audit`. + +- [ ] **Step 2: Make replacement-table creation strict** + +Change: + +```sql +CREATE TABLE IF NOT EXISTS ward_audit_new ( +``` + +to: + +```sql +CREATE TABLE ward_audit_new ( +``` + +An abandoned or conflicting migration table must abort instead of being reused. + +- [ ] **Step 3: Copy immutable detail instead of nulling it** + +Change the migration's `SELECT` projection from: + +```sql +tier, decision, approver, diff_hash, NULL, files_touched, channel, +``` + +to: + +```sql +tier, decision, approver, diff_hash, detail, files_touched, channel, +``` + +- [ ] **Step 4: Stamp fresh v0.1.4 schemas** + +Append this statement to `WARD_AUDIT_SCHEMA_SQL`, after trigger creation: + +```sql +PRAGMA user_version = 14; +``` + +- [ ] **Step 5: Correct the migration documentation** + +Update the module and constant comments to state: + +- New stores receive current DDL and `user_version = 14`. +- The legacy migration is only for the exact v0.1.3 shape. +- It adds `detail`, copies every row including `detail`, and stamps version 14. +- Current-schema invocation or rerun fails before destructive work. +- Callers must roll back a failed migration transaction. + +- [ ] **Step 6: Run focused tests** + +Run: + +```bash +cargo test -p coven-threads-core audit::tests -- --nocapture +``` + +Expected: all audit tests pass. + +- [ ] **Step 7: Commit the repair** + +```bash +git add crates/coven-threads-core/Cargo.toml \ + crates/coven-threads-core/src/audit.rs Cargo.lock +git commit -m "fix(audit): preserve detail during v0.1.4 migration" \ + -m "Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" +``` + +### Task 3: Validate and prepare PR #7 update + +**Files:** +- Verify: all changed files +- Update through CLI: Bead `threads-44o`, PR #7 + +- [ ] **Step 1: Run repository quality gates** + +Run: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test --workspace +``` + +Expected: all commands exit successfully with no warnings. + +- [ ] **Step 2: Review the final diff** + +Run: + +```bash +git diff origin/cody/apply-audit-v014...HEAD --check +git diff origin/cody/apply-audit-v014...HEAD --stat +git status --short --branch +``` + +Expected: only the design, plan, manifest/lockfile, and audit migration files +are changed; the worktree is clean after commits. + +- [ ] **Step 3: Record verification in Beads** + +Update `threads-44o` with the branch, worktree, commit SHA, focused test result, +and full quality-gate result. Do not close the Bead until PR #7 is updated and +the fix is accepted or merged. + +- [ ] **Step 4: Re-check remote branch drift** + +Run: + +```bash +git fetch origin cody/apply-audit-v014 +git log --oneline --left-right HEAD...origin/cody/apply-audit-v014 +``` + +Expected: no remote-only commits. If remote-only commits exist, stop and +reconcile before pushing. + +- [ ] **Step 5: Update PR #7 and publish review evidence** + +Push the worktree tip to the existing PR branch only if it is a fast-forward: + +```bash +git push origin HEAD:cody/apply-audit-v014 +``` + +Then comment on PR #7 with the root cause, the fail-closed migration design, +and the exact quality gates that passed. From 2bbba525a32675bd9cdfd59ab67475178a7f4fb7 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 22:43:49 -0500 Subject: [PATCH 09/25] fix(audit): preserve detail during v0.1.4 migration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Cargo.lock | 112 +++++++ crates/coven-threads-core/Cargo.toml | 3 + crates/coven-threads-core/src/audit.rs | 414 +++++++++++++++++++++---- 3 files changed, 476 insertions(+), 53 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 4caca61..1091649 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,6 +2,18 @@ # It is not intended for manual editing. version = 3 +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "once_cell", + "version_check", + "zerocopy", +] + [[package]] name = "arrayref" version = "0.3.9" @@ -14,6 +26,12 @@ version = "0.7.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + [[package]] name = "blake3" version = "1.8.5" @@ -70,6 +88,7 @@ name = "coven-threads-core" version = "0.1.4" dependencies = [ "blake3", + "rusqlite", "serde", "serde_json", "sha2", @@ -125,6 +144,18 @@ dependencies = [ "crypto-common", ] +[[package]] +name = "fallible-iterator" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649" + +[[package]] +name = "fallible-streaming-iterator" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" + [[package]] name = "find-msvc-tools" version = "0.1.9" @@ -176,6 +207,24 @@ dependencies = [ "r-efi", ] +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" +dependencies = [ + "ahash", +] + +[[package]] +name = "hashlink" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af" +dependencies = [ + "hashbrown", +] + [[package]] name = "itoa" version = "1.0.18" @@ -199,6 +248,17 @@ version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" +[[package]] +name = "libsqlite3-sys" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c10584274047cb335c23d3e61bcef8e323adae7c5c8c760540f73610177fc3f" +dependencies = [ + "cc", + "pkg-config", + "vcpkg", +] + [[package]] name = "memchr" version = "2.8.3" @@ -223,6 +283,12 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + [[package]] name = "powerfmt" version = "0.2.0" @@ -253,6 +319,20 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" +[[package]] +name = "rusqlite" +version = "0.31.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b838eba278d213a8beaf485bd313fd580ca4505a00d5871caeb1457c55322cae" +dependencies = [ + "bitflags", + "fallible-iterator", + "fallible-streaming-iterator", + "hashlink", + "libsqlite3-sys", + "smallvec", +] + [[package]] name = "rustversion" version = "1.0.23" @@ -325,6 +405,12 @@ version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + [[package]] name = "syn" version = "2.0.119" @@ -410,6 +496,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + [[package]] name = "version_check" version = "0.9.5" @@ -461,6 +553,26 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "zerocopy" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "zmij" version = "1.0.23" diff --git a/crates/coven-threads-core/Cargo.toml b/crates/coven-threads-core/Cargo.toml index 345d0e5..94c465c 100644 --- a/crates/coven-threads-core/Cargo.toml +++ b/crates/coven-threads-core/Cargo.toml @@ -19,3 +19,6 @@ uuid.workspace = true blake3.workspace = true sha2.workspace = true time.workspace = true + +[dev-dependencies] +rusqlite = { version = "0.31", features = ["bundled"] } diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 19ecd5f..958da08 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -20,14 +20,15 @@ //! //! ## Schema versioning and migration //! -//! `WARD_AUDIT_SCHEMA_SQL` uses `CREATE TABLE IF NOT EXISTS`, so new empty -//! stores automatically receive the current DDL including the `apply_audit` -//! CHECK tag. Stores created against v0.1.3 (which lacked the `apply_audit` -//! CHECK entry) must be rebuilt: the daemon should run -//! `WARD_AUDIT_MIGRATION_V014_SQL` once at startup when it detects the old -//! CHECK text is still in the schema. That migration rebuilds `ward_audit` -//! in a single transaction (create, copy, drop, rename) — SQLite cannot -//! `ALTER` a CHECK constraint. +//! `WARD_AUDIT_SCHEMA_SQL` creates the fresh v0.1.4 shape and stamps +//! `PRAGMA user_version = 14`, so new empty stores are versioned immediately. +//! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact v0.1.3 legacy +//! shape (no `detail` column, no `apply_audit` CHECK tag). It first adds +//! `detail`, then rebuilds `ward_audit` in one transaction so the current +//! CHECK set is installed and any existing `detail` values are preserved. +//! Running that migration against a current schema, or rerunning it after a +//! successful upgrade, fails before destructive work; callers must explicitly +//! roll back the failed transaction before continuing. //! //! ## Where content hashes ride for applied writes //! @@ -301,35 +302,36 @@ fn reject_tag(reason: &RejectReason) -> &'static str { /// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). /// -/// Idempotent (`IF NOT EXISTS` throughout) so the daemon can apply it at -/// startup. Append-only is enforced *in the store*: UPDATE and DELETE abort via +/// Append-only is enforced *in the store*: UPDATE and DELETE abort via /// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). -/// DDL migration for v0.1.3 → v0.1.4: rebuilds `ward_audit` so its CHECK -/// constraint includes the new `apply_audit` tag. +/// DDL migration for v0.1.3 → v0.1.4: rebuilds `ward_audit` from the exact +/// legacy shape so its CHECK constraint includes `apply_audit` and its copy +/// path preserves `detail`. /// /// SQLite cannot `ALTER` a CHECK constraint on an existing table. This SQL /// performs a safe swap in one transaction: -/// 1. Creates `ward_audit_new` with the updated CHECK (plus `detail` column). -/// 2. Copies all existing rows (NULLing `detail` for old rows). -/// 3. Drops the old table. -/// 4. Renames the new table into place. -/// 5. Re-creates indexes and append-only triggers. +/// 1. Adds the legacy `detail` column so the old table matches the copy shape. +/// 2. Creates `ward_audit_new` with the updated CHECK. +/// 3. Copies all existing rows, preserving `detail`. +/// 4. Drops the old table. +/// 5. Renames the new table into place. +/// 6. Re-creates indexes and append-only triggers. /// -/// **Run condition:** execute this only when `PRAGMA user_version` returns a -/// value less than `14` — i.e. the store was created before this migration. -/// `PRAGMA user_version` is `0` for stores that predate any explicit versioning -/// (all v0.1.3 stores). After this migration runs, `user_version` is `14` -/// (matching `0.1.4`), giving future migrations a clean integer ladder to gate -/// on instead of substring-sniffing `sqlite_master` DDL. +/// **Run condition:** execute this only for the exact v0.1.3 legacy shape. +/// Fresh v0.1.4 schemas already stamp `PRAGMA user_version = 14`; legacy +/// v0.1.3 stores start at `0` and are stamped `14` only after this rebuild. /// -/// Idempotent in the sense that if `user_version >= 14` the daemon must not -/// re-run it; the `ward_audit_new` table name is not guarded with `IF NOT -/// EXISTS`, so a double-run would fail on the `CREATE` step (intentional: -/// prevents silent data loss from a buggy migration controller). +/// Current-schema invocation and post-upgrade reruns fail before any +/// destructive step: a current schema aborts on `ALTER TABLE ... ADD COLUMN +/// detail`, while a rerun after a successful upgrade aborts on +/// `CREATE TABLE ward_audit_new`. Callers must `ROLLBACK` failed migration +/// transactions before continuing. pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = r#" BEGIN; -CREATE TABLE IF NOT EXISTS ward_audit_new ( +ALTER TABLE ward_audit ADD COLUMN detail TEXT; + +CREATE TABLE ward_audit_new ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_type TEXT NOT NULL CHECK (event_type IN ( 'proposal_submitted','proposal_approved','proposal_rejected', @@ -359,7 +361,7 @@ INSERT INTO ward_audit_new ( ) SELECT id, event_type, proposal_id, familiar_id, ward_version, ward_hash, - tier, decision, approver, diff_hash, NULL, files_touched, channel, + tier, decision, approver, diff_hash, detail, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at FROM ward_audit; @@ -381,9 +383,6 @@ BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; --- Stamp schema version so future migrations can gate on `PRAGMA user_version` --- instead of substring-sniffing the CHECK DDL. Gate rule for callers: --- run this migration when `PRAGMA user_version < 14`. PRAGMA user_version = 14; COMMIT; @@ -391,7 +390,8 @@ COMMIT; /// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). /// -/// See module docs for migration notes. +/// See module docs for migration notes. Fresh schemas stamp +/// `PRAGMA user_version = 14`. pub const WARD_AUDIT_SCHEMA_SQL: &str = r#" CREATE TABLE IF NOT EXISTS ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, @@ -430,12 +430,237 @@ BEFORE DELETE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; + +PRAGMA user_version = 14; "#; #[cfg(test)] mod tests { use super::*; use crate::ids::{SurfaceId, WriterId}; + use rusqlite::{params, Connection}; + + const FIXED_SUBMITTED_AT: &str = "2026-07-19T00:00:00.000Z"; + const FIXED_DECIDED_AT: &str = "2026-07-19T00:01:00.000Z"; + const FIXED_RECORDED_AT: &str = "2026-07-19T00:02:00.000Z"; + const FIXED_WARD_HASH: [u8; 32] = *b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; + const FIXED_DIFF_HASH: [u8; 32] = *b"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"; + const FIXED_PREV_DETAIL: &str = r#"{"prev_sha256":"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc","bytes_written":42}"#; + const FIXED_FILES_TOUCHED: &str = r#"["SOUL.md"]"#; + + /// Representative v0.1.3 `ward_audit` schema: no `detail` column and no + /// `apply_audit` event tag, but still append-only. + const LEGACY_WARD_AUDIT_SCHEMA_SQL: &str = r#" +CREATE TABLE IF NOT EXISTS ward_audit ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL CHECK (event_type IN ( + 'proposal_submitted','proposal_approved','proposal_rejected', + 'proposal_vetoed','ward_updated','validation_verdict', + 'compaction_ledger')), + proposal_id TEXT, + familiar_id TEXT NOT NULL, + ward_version TEXT, + ward_hash BLOB NOT NULL, + tier TEXT, + decision TEXT NOT NULL, + approver TEXT, + diff_hash BLOB, + files_touched TEXT NOT NULL, + channel TEXT, + thread_id TEXT, + submitted_at TEXT NOT NULL, + decided_at TEXT NOT NULL, + recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) +); + +CREATE INDEX IF NOT EXISTS ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); +CREATE INDEX IF NOT EXISTS ward_audit_event_idx ON ward_audit (event_type, recorded_at); + +CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_update +BEFORE UPDATE ON ward_audit +BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); +END; + +CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_delete +BEFORE DELETE ON ward_audit +BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); +END; +"#; + + #[derive(Debug, PartialEq, Eq)] + struct StoredAuditRow { + id: i64, + event_type: String, + proposal_id: Option, + familiar_id: String, + ward_version: Option, + ward_hash: Vec, + tier: Option, + decision: String, + approver: Option, + diff_hash: Option>, + detail: Option, + files_touched: String, + channel: Option, + thread_id: Option, + submitted_at: String, + decided_at: String, + recorded_at: String, + } + + fn open_conn(schema_sql: &str) -> Connection { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(schema_sql).unwrap(); + conn + } + + fn user_version(conn: &Connection) -> i64 { + conn.query_row("PRAGMA user_version;", [], |row| row.get(0)) + .unwrap() + } + + fn load_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { + conn.query_row( + r#" + SELECT id, event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, detail, files_touched, + channel, thread_id, submitted_at, decided_at, recorded_at + FROM ward_audit + WHERE id = ?1 + "#, + params![id], + |row| { + Ok(StoredAuditRow { + id: row.get(0)?, + event_type: row.get(1)?, + proposal_id: row.get(2)?, + familiar_id: row.get(3)?, + ward_version: row.get(4)?, + ward_hash: row.get(5)?, + tier: row.get(6)?, + decision: row.get(7)?, + approver: row.get(8)?, + diff_hash: row.get(9)?, + detail: row.get(10)?, + files_touched: row.get(11)?, + channel: row.get(12)?, + thread_id: row.get(13)?, + submitted_at: row.get(14)?, + decided_at: row.get(15)?, + recorded_at: row.get(16)?, + }) + }, + ) + .unwrap() + } + + fn load_legacy_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { + conn.query_row( + r#" + SELECT id, event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, files_touched, channel, + thread_id, submitted_at, decided_at, recorded_at + FROM ward_audit + WHERE id = ?1 + "#, + params![id], + |row| { + Ok(StoredAuditRow { + id: row.get(0)?, + event_type: row.get(1)?, + proposal_id: row.get(2)?, + familiar_id: row.get(3)?, + ward_version: row.get(4)?, + ward_hash: row.get(5)?, + tier: row.get(6)?, + decision: row.get(7)?, + approver: row.get(8)?, + diff_hash: row.get(9)?, + detail: None, + files_touched: row.get(10)?, + channel: row.get(11)?, + thread_id: row.get(12)?, + submitted_at: row.get(13)?, + decided_at: row.get(14)?, + recorded_at: row.get(15)?, + }) + }, + ) + .unwrap() + } + + fn insert_current_apply_audit_row(conn: &Connection) -> i64 { + conn.execute( + r#" + INSERT INTO ward_audit ( + event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, detail, files_touched, + channel, thread_id, submitted_at, decided_at, recorded_at + ) VALUES ( + ?1, ?2, ?3, ?4, ?5, + ?6, ?7, ?8, ?9, ?10, ?11, + ?12, ?13, ?14, ?15, ?16 + ) + "#, + params![ + "apply_audit", + Some("proposal-current"), + "familiar-current", + Some("0.1.4"), + FIXED_WARD_HASH.as_ref(), + Some("tier_2"), + "applied", + Option::::None, + Some(FIXED_DIFF_HASH.as_ref()), + FIXED_PREV_DETAIL, + FIXED_FILES_TOUCHED, + Some("mutation"), + Option::::None, + FIXED_SUBMITTED_AT, + FIXED_DECIDED_AT, + FIXED_RECORDED_AT, + ], + ) + .unwrap(); + conn.last_insert_rowid() + } + + fn insert_legacy_ward_updated_row(conn: &Connection) -> i64 { + conn.execute( + r#" + INSERT INTO ward_audit ( + event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, files_touched, channel, + thread_id, submitted_at, decided_at, recorded_at + ) VALUES ( + ?1, ?2, ?3, ?4, ?5, + ?6, ?7, ?8, ?9, ?10, ?11, + ?12, ?13, ?14, ?15 + ) + "#, + params![ + "ward_updated", + Some("proposal-legacy"), + "familiar-legacy", + Some("0.1.3"), + FIXED_WARD_HASH.as_ref(), + Some("tier_1"), + "updated", + Some("writer:legacy"), + Some(FIXED_DIFF_HASH.as_ref()), + FIXED_FILES_TOUCHED, + Some("mutation"), + Some("thread-legacy"), + FIXED_SUBMITTED_AT, + FIXED_DECIDED_AT, + FIXED_RECORDED_AT, + ], + ) + .unwrap(); + conn.last_insert_rowid() + } fn request() -> MutationRequest { MutationRequest { @@ -519,15 +744,12 @@ mod tests { #[test] fn for_apply_produces_correct_shape() { let now = OffsetDateTime::now_utc(); - let familiar_id = FamiliarId::new(); - let surface = SurfaceId::new("SOUL.md"); let prev = [0xaa_u8; 32]; let next = [0xbb_u8; 32]; - let record = WardAuditRecord::for_apply( - familiar_id, + FamiliarId::new(), &[0xcc; 32], - surface.clone(), + SurfaceId::new("SOUL.md"), "tier_2", Some(&prev), Some(&next), @@ -539,8 +761,8 @@ mod tests { assert_eq!(record.event_type, AuditEventType::ApplyAudit); assert_eq!(record.decision, "applied"); assert_eq!(record.tier.as_deref(), Some("tier_2")); - assert_eq!(record.diff_hash, Some(next.to_vec())); - assert_eq!(record.files_touched, vec![surface]); + assert_eq!(record.diff_hash, Some(vec![0xbb; 32])); + assert_eq!(record.files_touched, vec![SurfaceId::new("SOUL.md")]); // detail decoding helpers let prev_hex = record.apply_prev_sha256_hex().unwrap(); @@ -570,22 +792,108 @@ mod tests { } #[test] - fn migration_sql_contains_apply_audit_and_detail_column() { - assert!( - WARD_AUDIT_MIGRATION_V014_SQL.contains("'apply_audit'"), - "migration CHECK must contain apply_audit tag" - ); - assert!( - WARD_AUDIT_MIGRATION_V014_SQL.contains("detail"), - "migration must add detail column" - ); + fn fresh_schema_stamps_user_version_14() { + let conn = open_conn(WARD_AUDIT_SCHEMA_SQL); + assert_eq!(user_version(&conn), 14); + } + + #[test] + fn migration_rejects_current_schema_rows_with_detail() { + let conn = open_conn(WARD_AUDIT_SCHEMA_SQL); + let row_id = insert_current_apply_audit_row(&conn); + let before = load_audit_row(&conn, row_id); + let before_version = user_version(&conn); + assert_eq!(before_version, 14); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("current-schema migration must fail on duplicate detail"); assert!( - WARD_AUDIT_MIGRATION_V014_SQL.contains("BEGIN;"), - "migration must be a transaction" + err.to_string().contains("duplicate column name: detail"), + "unexpected migration error: {err}" ); + let _ = conn.execute_batch("ROLLBACK;"); + + let after = load_audit_row(&conn, row_id); + assert_eq!(after, before); + assert_eq!(user_version(&conn), before_version); + } + + #[test] + fn legacy_schema_upgrades_and_preserves_append_only_behavior() { + let conn = open_conn(LEGACY_WARD_AUDIT_SCHEMA_SQL); + let row_id = insert_legacy_ward_updated_row(&conn); + let before = load_legacy_audit_row(&conn, row_id); + assert_eq!(before.event_type, "ward_updated"); + assert_eq!(before.decision, "updated"); + assert_eq!(user_version(&conn), 0); + + conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); + assert_eq!(user_version(&conn), 14); + + let after = load_audit_row(&conn, row_id); + assert_eq!(after.id, row_id); + assert_eq!(after.event_type, "ward_updated"); + assert_eq!(after.proposal_id.as_deref(), Some("proposal-legacy")); + assert_eq!(after.familiar_id, "familiar-legacy"); + assert_eq!(after.ward_version.as_deref(), Some("0.1.3")); + assert_eq!(after.ward_hash, FIXED_WARD_HASH.to_vec()); + assert_eq!(after.tier.as_deref(), Some("tier_1")); + assert_eq!(after.decision, "updated"); + assert_eq!(after.approver.as_deref(), Some("writer:legacy")); + assert_eq!(after.diff_hash, Some(FIXED_DIFF_HASH.to_vec())); + assert_eq!(after.detail, None); + assert_eq!(after.files_touched, FIXED_FILES_TOUCHED); + assert_eq!(after.channel.as_deref(), Some("mutation")); + assert_eq!(after.thread_id.as_deref(), Some("thread-legacy")); + assert_eq!(after.submitted_at, FIXED_SUBMITTED_AT); + assert_eq!(after.decided_at, FIXED_DECIDED_AT); + assert_eq!(after.recorded_at, FIXED_RECORDED_AT); + + let apply_row_id = insert_current_apply_audit_row(&conn); + let apply_row = load_audit_row(&conn, apply_row_id); + assert_eq!(apply_row.event_type, "apply_audit"); + assert_eq!(apply_row.detail.as_deref(), Some(FIXED_PREV_DETAIL)); + + let update_err = conn + .execute( + "UPDATE ward_audit SET decision = 'changed' WHERE id = ?1", + params![row_id], + ) + .unwrap_err(); + assert!(update_err.to_string().contains("append-only")); + + let delete_err = conn + .execute("DELETE FROM ward_audit WHERE id = ?1", params![row_id]) + .unwrap_err(); + assert!(delete_err.to_string().contains("append-only")); + } + + #[test] + fn rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows() { + let conn = open_conn(LEGACY_WARD_AUDIT_SCHEMA_SQL); + let legacy_row_id = insert_legacy_ward_updated_row(&conn); + + conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); + assert_eq!(user_version(&conn), 14); + + let apply_row_id = insert_current_apply_audit_row(&conn); + let legacy_before = load_audit_row(&conn, legacy_row_id); + let apply_before = load_audit_row(&conn, apply_row_id); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("rerunning the migration must fail on duplicate detail"); assert!( - WARD_AUDIT_MIGRATION_V014_SQL.contains("COMMIT;"), - "migration must be a transaction" + err.to_string().contains("duplicate column name: detail"), + "unexpected migration error: {err}" ); + let _ = conn.execute_batch("ROLLBACK;"); + + let legacy_after = load_audit_row(&conn, legacy_row_id); + let apply_after = load_audit_row(&conn, apply_row_id); + assert_eq!(legacy_after, legacy_before); + assert_eq!(apply_after, apply_before); + assert_eq!(user_version(&conn), 14); } } From 0641b0081a8bcfbf50c06e2c38d0a883a96b6b0f Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 22:45:22 -0500 Subject: [PATCH 10/25] docs(audit): correct migration rerun guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/coven-threads-core/src/audit.rs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 958da08..dc8ba94 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -323,9 +323,11 @@ fn reject_tag(reason: &RejectReason) -> &'static str { /// /// Current-schema invocation and post-upgrade reruns fail before any /// destructive step: a current schema aborts on `ALTER TABLE ... ADD COLUMN -/// detail`, while a rerun after a successful upgrade aborts on -/// `CREATE TABLE ward_audit_new`. Callers must `ROLLBACK` failed migration -/// transactions before continuing. +/// detail` with a duplicate-column error, while a rerun after a successful +/// upgrade fails on the same `ALTER TABLE` because the column already exists. +/// Callers must `ROLLBACK` failed migration transactions before continuing; +/// the migration remains fail-closed because no later step can run after that +/// early abort. pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = r#" BEGIN; From 2fbb5e3ab1d11ea53184c721c3e399bd0002c93b Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:02:18 -0500 Subject: [PATCH 11/25] fix(audit): keep migration versioning table-local Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 4 +- crates/coven-threads-core/src/audit.rs | 148 ++++++++++++++---- ...2026-07-19-apply-audit-migration-repair.md | 115 ++++++++------ ...-19-apply-audit-migration-repair-design.md | 36 +++-- 4 files changed, 210 insertions(+), 93 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc34d24..85ad693 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,9 +13,9 @@ All notable changes to `coven-threads-core` are documented here. - `WardAuditRecord::for_apply(...)` constructor — builds the audit row for a Gate-4 applied write. - `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. -- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). Bumps `PRAGMA user_version = 14` inside the transaction so future migrations can gate on `user_version < N` instead of substring-sniffing `sqlite_master` DDL. The daemon should run this migration when `PRAGMA user_version < 14`. +- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). Future daemon integration must gate this SQL by exact `ward_audit` table shape: legacy means the table exists, `detail` is absent, and the CHECK does not contain `apply_audit`; current means `detail` exists and `apply_audit` is accepted. The SQL preserves database-wide `PRAGMA user_version`. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. -- New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `migration_sql_contains_apply_audit_and_detail_column`. +- New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, and `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`. ### Design notes diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index dc8ba94..999e2dc 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -18,14 +18,20 @@ //! **Gate-4 applied writes** are also auditable here via //! `AuditEventType::ApplyAudit` (see §3.4, coven-threads#5). //! -//! ## Schema versioning and migration +//! ## Schema shape and migration gating //! -//! `WARD_AUDIT_SCHEMA_SQL` creates the fresh v0.1.4 shape and stamps -//! `PRAGMA user_version = 14`, so new empty stores are versioned immediately. +//! `WARD_AUDIT_SCHEMA_SQL` creates the fresh v0.1.4 `ward_audit` shape without +//! mutating database-wide `PRAGMA user_version`. //! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact v0.1.3 legacy -//! shape (no `detail` column, no `apply_audit` CHECK tag). It first adds -//! `detail`, then rebuilds `ward_audit` in one transaction so the current -//! CHECK set is installed and any existing `detail` values are preserved. +//! shape. Callers must inspect the `ward_audit` table itself before execution: +//! - exact legacy v0.1.3: the table exists, `detail` is absent, and the table +//! CHECK does not contain `apply_audit`; +//! - current v0.1.4: `detail` exists and the table CHECK accepts +//! `apply_audit`; +//! - partial/unknown shapes: fail closed and do not run the migration. +//! The migration first adds `detail`, then rebuilds `ward_audit` in one +//! transaction so the current CHECK set is installed and every existing row is +//! preserved. //! Running that migration against a current schema, or rerunning it after a //! successful upgrade, fails before destructive work; callers must explicitly //! roll back the failed transaction before continuing. @@ -317,9 +323,11 @@ fn reject_tag(reason: &RejectReason) -> &'static str { /// 5. Renames the new table into place. /// 6. Re-creates indexes and append-only triggers. /// -/// **Run condition:** execute this only for the exact v0.1.3 legacy shape. -/// Fresh v0.1.4 schemas already stamp `PRAGMA user_version = 14`; legacy -/// v0.1.3 stores start at `0` and are stamped `14` only after this rebuild. +/// **Caller gate:** inspect `ward_audit` itself before executing this SQL. +/// Run it only when the table exists, `detail` is absent, and the table CHECK +/// does not contain `apply_audit`. If `detail` exists and the table accepts +/// `apply_audit`, the schema is already current. Any partial/unknown shape +/// must fail closed instead of running the migration. /// /// Current-schema invocation and post-upgrade reruns fail before any /// destructive step: a current schema aborts on `ALTER TABLE ... ADD COLUMN @@ -327,7 +335,8 @@ fn reject_tag(reason: &RejectReason) -> &'static str { /// upgrade fails on the same `ALTER TABLE` because the column already exists. /// Callers must `ROLLBACK` failed migration transactions before continuing; /// the migration remains fail-closed because no later step can run after that -/// early abort. +/// early abort. This SQL does not read or write database-wide +/// `PRAGMA user_version`. pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = r#" BEGIN; @@ -385,15 +394,14 @@ BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; -PRAGMA user_version = 14; - COMMIT; "#; /// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). /// -/// See module docs for migration notes. Fresh schemas stamp -/// `PRAGMA user_version = 14`. +/// See module docs for shape detection and migration gating. This DDL creates +/// the current `ward_audit` shape without mutating database-wide +/// `PRAGMA user_version`. pub const WARD_AUDIT_SCHEMA_SQL: &str = r#" CREATE TABLE IF NOT EXISTS ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, @@ -432,8 +440,6 @@ BEFORE DELETE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; - -PRAGMA user_version = 14; "#; #[cfg(test)] @@ -512,17 +518,74 @@ END; recorded_at: String, } - fn open_conn(schema_sql: &str) -> Connection { - let conn = Connection::open_in_memory().unwrap(); - conn.execute_batch(schema_sql).unwrap(); - conn - } - fn user_version(conn: &Connection) -> i64 { conn.query_row("PRAGMA user_version;", [], |row| row.get(0)) .unwrap() } + fn set_user_version(conn: &Connection, version: i64) { + conn.pragma_update(None, "user_version", version).unwrap(); + } + + fn ward_audit_table_sql(conn: &Connection) -> String { + conn.query_row( + "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'ward_audit';", + [], + |row| row.get(0), + ) + .unwrap() + } + + fn ward_audit_has_detail_column(conn: &Connection) -> bool { + conn.query_row( + "SELECT EXISTS(SELECT 1 FROM pragma_table_info('ward_audit') WHERE name = 'detail');", + [], + |row| row.get::<_, i64>(0), + ) + .unwrap() + == 1 + } + + fn ward_audit_accepts_apply_audit(conn: &Connection) -> bool { + ward_audit_table_sql(conn).contains("'apply_audit'") + } + + fn assert_legacy_ward_audit_shape(conn: &Connection) { + assert!( + !ward_audit_has_detail_column(conn), + "legacy schema must not expose detail" + ); + assert!( + !ward_audit_accepts_apply_audit(conn), + "legacy schema CHECK must not mention apply_audit" + ); + } + + fn assert_current_ward_audit_shape(conn: &Connection) { + assert!( + ward_audit_has_detail_column(conn), + "current schema must expose detail" + ); + assert!( + ward_audit_accepts_apply_audit(conn), + "current schema CHECK must accept apply_audit" + ); + } + + fn assert_fresh_schema_preserves_user_version(initial_version: i64) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, initial_version); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + assert_eq!(user_version(&conn), initial_version); + assert_current_ward_audit_shape(&conn); + + let row_id = insert_current_apply_audit_row(&conn); + let row = load_audit_row(&conn, row_id); + assert_eq!(row.event_type, "apply_audit"); + assert_eq!(row.detail.as_deref(), Some(FIXED_PREV_DETAIL)); + } + fn load_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { conn.query_row( r#" @@ -794,18 +857,25 @@ END; } #[test] - fn fresh_schema_stamps_user_version_14() { - let conn = open_conn(WARD_AUDIT_SCHEMA_SQL); - assert_eq!(user_version(&conn), 14); + fn fresh_schema_preserves_user_version_zero_and_creates_current_shape() { + assert_fresh_schema_preserves_user_version(0); + } + + #[test] + fn fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape() { + assert_fresh_schema_preserves_user_version(99); } #[test] - fn migration_rejects_current_schema_rows_with_detail() { - let conn = open_conn(WARD_AUDIT_SCHEMA_SQL); + fn migration_rejects_current_schema_rows_with_detail_and_preserves_state() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_current_ward_audit_shape(&conn); let row_id = insert_current_apply_audit_row(&conn); let before = load_audit_row(&conn, row_id); let before_version = user_version(&conn); - assert_eq!(before_version, 14); + assert_eq!(before_version, 37); let err = conn .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) @@ -818,20 +888,25 @@ END; let after = load_audit_row(&conn, row_id); assert_eq!(after, before); + assert_current_ward_audit_shape(&conn); assert_eq!(user_version(&conn), before_version); } #[test] fn legacy_schema_upgrades_and_preserves_append_only_behavior() { - let conn = open_conn(LEGACY_WARD_AUDIT_SCHEMA_SQL); + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_legacy_ward_audit_shape(&conn); let row_id = insert_legacy_ward_updated_row(&conn); let before = load_legacy_audit_row(&conn, row_id); assert_eq!(before.event_type, "ward_updated"); assert_eq!(before.decision, "updated"); - assert_eq!(user_version(&conn), 0); + assert_eq!(user_version(&conn), 37); conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); - assert_eq!(user_version(&conn), 14); + assert_current_ward_audit_shape(&conn); + assert_eq!(user_version(&conn), 37); let after = load_audit_row(&conn, row_id); assert_eq!(after.id, row_id); @@ -873,11 +948,15 @@ END; #[test] fn rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows() { - let conn = open_conn(LEGACY_WARD_AUDIT_SCHEMA_SQL); + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_legacy_ward_audit_shape(&conn); let legacy_row_id = insert_legacy_ward_updated_row(&conn); conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); - assert_eq!(user_version(&conn), 14); + assert_current_ward_audit_shape(&conn); + assert_eq!(user_version(&conn), 37); let apply_row_id = insert_current_apply_audit_row(&conn); let legacy_before = load_audit_row(&conn, legacy_row_id); @@ -896,6 +975,7 @@ END; let apply_after = load_audit_row(&conn, apply_row_id); assert_eq!(legacy_after, legacy_before); assert_eq!(apply_after, apply_before); - assert_eq!(user_version(&conn), 14); + assert_current_ward_audit_shape(&conn); + assert_eq!(user_version(&conn), 37); } } diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 5b80ebd..703ebc6 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -2,9 +2,9 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Make the v0.1.4 `ward_audit` migration preserve immutable evidence and fail closed when invoked against an already-current schema. +**Goal:** Make the v0.1.4 `ward_audit` migration preserve immutable evidence, preserve database-wide version state, and fail closed when invoked outside the exact legacy schema shape. -**Architecture:** Keep migration ownership in `audit.rs`. The legacy migration adds `detail` before rebuilding and copies that column verbatim; the current schema stamps version 14. In-memory SQLite tests execute both schema paths and verify data preservation, append-only enforcement, versioning, and safe failure. +**Architecture:** Keep migration ownership in `audit.rs`. The legacy migration adds `detail` before rebuilding and copies that column verbatim; callers gate execution by exact `ward_audit` table shape instead of `PRAGMA user_version` or a separate marker. In-memory SQLite tests execute both schema paths and verify data preservation, append-only enforcement, database-wide `user_version` preservation, and safe failure. **Tech Stack:** Rust 2021, SQLite, `rusqlite` test dependency, Cargo test. @@ -12,25 +12,20 @@ ## File Map -- Modify `crates/coven-threads-core/Cargo.toml` to add bundled SQLite for tests only. -- Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, stamp fresh schemas, and add executable migration tests. -- Modify `Cargo.lock` through Cargo after adding the test dependency. +- Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, document exact-shape gating, and add executable migration tests. +- Modify `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` to document exact table-shape gating and database-wide version preservation. +- Modify `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` to align the implementation plan with the approved table-local versioning revision. +- Modify `CHANGELOG.md` to describe table-local migration gating and the updated regression coverage. ### Task 1: Add executable migration regressions **Files:** -- Modify: `crates/coven-threads-core/Cargo.toml` - Modify: `crates/coven-threads-core/src/audit.rs:435-590` -- Modify: `Cargo.lock` -- [ ] **Step 1: Add the test-only SQLite dependency** +- [ ] **Step 1: Reuse the existing test-only SQLite dependency** -Append this section to `crates/coven-threads-core/Cargo.toml`: - -```toml -[dev-dependencies] -rusqlite = { version = "0.31", features = ["bundled"] } -``` +Use the existing bundled `rusqlite` dev-dependency already present for audit +tests; no production dependency or extra migration metadata is needed. - [ ] **Step 2: Add a representative legacy schema fixture** @@ -96,27 +91,44 @@ fn insert_audit_row(connection: &Connection, event_type: &str, detail: Option<&s For the legacy test, insert its pre-migration row with a dedicated statement that omits `detail`, because the v0.1.3 table does not have that column. -- [ ] **Step 3: Add the failing fresh-schema version test** +- [ ] **Step 3: Add failing fresh-schema preservation tests** ```rust #[test] -fn fresh_schema_is_stamped_v014() { +fn fresh_schema_preserves_user_version_zero() { + let connection = Connection::open_in_memory().unwrap(); + connection.execute_batch("PRAGMA user_version = 0;").unwrap(); + connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + let version: i64 = connection + .query_row("PRAGMA user_version", [], |row| row.get(0)) + .unwrap(); + assert_eq!(version, 0); +} + +#[test] +fn fresh_schema_preserves_user_version_ninety_nine() { let connection = Connection::open_in_memory().unwrap(); + connection.execute_batch("PRAGMA user_version = 99;").unwrap(); connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); let version: i64 = connection .query_row("PRAGMA user_version", [], |row| row.get(0)) .unwrap(); - assert_eq!(version, 14); + assert_eq!(version, 99); } ``` +Each test should also assert the resulting table exposes `detail` and accepts +`apply_audit`. + - [ ] **Step 4: Add the failing current-schema preservation test** ```rust #[test] fn legacy_migration_rejects_current_schema_without_erasing_detail() { let connection = Connection::open_in_memory().unwrap(); + connection.execute_batch("PRAGMA user_version = 37;").unwrap(); connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); let detail = r#"{"prev_sha256":"abcd","bytes_written":42}"#; let id = insert_audit_row(&connection, "apply_audit", Some(detail)); @@ -135,6 +147,11 @@ fn legacy_migration_rejects_current_schema_without_erasing_detail() { ) .unwrap(); assert_eq!(preserved, detail); + + let version: i64 = connection + .query_row("PRAGMA user_version", [], |row| row.get(0)) + .unwrap(); + assert_eq!(version, 37); } ``` @@ -143,16 +160,20 @@ fn legacy_migration_rejects_current_schema_without_erasing_detail() { The legacy-upgrade test must: 1. Execute `LEGACY_WARD_AUDIT_SCHEMA_SQL`. -2. Insert a fully populated legacy row without `detail`. -3. Execute `WARD_AUDIT_MIGRATION_V014_SQL`. -4. Assert the row's original values remain unchanged and `detail IS NULL`. -5. Assert `PRAGMA user_version = 14`. -6. Insert an `apply_audit` row with populated `detail`. -7. Assert both append-only triggers reject `UPDATE` and `DELETE`. +2. Preset an unrelated database-wide `PRAGMA user_version` value (for example + `37`). +3. Insert a fully populated legacy row without `detail`. +4. Execute `WARD_AUDIT_MIGRATION_V014_SQL`. +5. Assert the row's original values remain unchanged and `detail IS NULL`. +6. Assert the rebuilt table now exposes `detail`, accepts `apply_audit`, and + preserves `PRAGMA user_version = 37`. +7. Insert an `apply_audit` row with populated `detail`. +8. Assert both append-only triggers reject `UPDATE` and `DELETE`. The rerun test must execute the migration successfully once, invoke it again, assert the duplicate-`detail` error, roll back the failed transaction, and -confirm the original migrated row still exists unchanged. +confirm the original migrated row and preserved database-wide `user_version` +still exist unchanged. - [ ] **Step 6: Run the focused tests and confirm the regression** @@ -162,10 +183,10 @@ Run: cargo test -p coven-threads-core audit::tests -- --nocapture ``` -Expected before the SQL repair: `fresh_schema_is_stamped_v014` fails with -`left: 0, right: 14`, and -`legacy_migration_rejects_current_schema_without_erasing_detail` fails because -the migration succeeds and replaces `detail` with `NULL`. +Expected before the SQL repair: the fresh-schema preservation tests fail because +`WARD_AUDIT_SCHEMA_SQL` overwrites the pre-set database-wide `user_version`, +and the current/legacy/rerun migration tests fail because +`WARD_AUDIT_MIGRATION_V014_SQL` rewrites the same unrelated value to `14`. ### Task 2: Make the migration preserve evidence and fail closed @@ -213,21 +234,24 @@ to: tier, decision, approver, diff_hash, detail, files_touched, channel, ``` -- [ ] **Step 4: Stamp fresh v0.1.4 schemas** +- [ ] **Step 4: Preserve database-wide version state** -Append this statement to `WARD_AUDIT_SCHEMA_SQL`, after trigger creation: - -```sql -PRAGMA user_version = 14; -``` +Remove any `PRAGMA user_version = 14;` statements from +`WARD_AUDIT_SCHEMA_SQL` and `WARD_AUDIT_MIGRATION_V014_SQL`. - [ ] **Step 5: Correct the migration documentation** Update the module and constant comments to state: -- New stores receive current DDL and `user_version = 14`. -- The legacy migration is only for the exact v0.1.3 shape. -- It adds `detail`, copies every row including `detail`, and stamps version 14. +- New stores receive the current DDL without mutating database-wide + `user_version`. +- The legacy migration is only for the exact v0.1.3 shape: `ward_audit` + exists, `detail` is absent, and the table CHECK does not contain + `apply_audit`. +- Current v0.1.4 or partial/unknown shapes fail closed rather than running the + migration. +- The migration adds `detail`, copies every row including `detail`, and + preserves database-wide `user_version`. - Current-schema invocation or rerun fails before destructive work. - Callers must roll back a failed migration transaction. @@ -244,9 +268,11 @@ Expected: all audit tests pass. - [ ] **Step 7: Commit the repair** ```bash -git add crates/coven-threads-core/Cargo.toml \ - crates/coven-threads-core/src/audit.rs Cargo.lock -git commit -m "fix(audit): preserve detail during v0.1.4 migration" \ +git add CHANGELOG.md \ + crates/coven-threads-core/src/audit.rs \ + docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ + docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +git commit -m "fix(audit): keep migration versioning table-local" \ -m "Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" ``` @@ -262,11 +288,10 @@ Run: ```bash cargo fmt --all -- --check -cargo clippy --workspace --all-targets -- -D warnings -cargo test --workspace +cargo test -p coven-threads-core audit::tests -- --nocapture ``` -Expected: all commands exit successfully with no warnings. +Expected: both commands exit successfully. - [ ] **Step 2: Review the final diff** @@ -278,8 +303,8 @@ git diff origin/cody/apply-audit-v014...HEAD --stat git status --short --branch ``` -Expected: only the design, plan, manifest/lockfile, and audit migration files -are changed; the worktree is clean after commits. +Expected: only the changelog, design, plan, and audit migration files are +changed; the worktree is clean after commits. - [ ] **Step 3: Record verification in Beads** diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index fac1043..ea02757 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -8,16 +8,26 @@ the audit schema contract. ## Design -The v0.1.4 migration will first add the nullable `detail` column to the legacy -table, then rebuild the table and copy `detail` into the replacement table. -This gives genuine v0.1.3 rows a `NULL` detail value without hard-coding data -loss into the copy step. +The v0.1.4 migration remains a table-local repair. Future daemon integration +must inspect `ward_audit` itself before execution: + +- exact legacy v0.1.3: the table exists, `detail` is absent, and the table + CHECK does not contain `apply_audit`; +- current v0.1.4: `detail` exists and the table accepts `apply_audit`; +- partial/unknown shapes: fail closed and do not run the migration. + +For the exact legacy shape, the migration first adds the nullable `detail` +column to the legacy table, then rebuilds the table and copies `detail` into +the replacement table. This gives genuine v0.1.3 rows a `NULL` detail value +without hard-coding data loss into the copy step. The SQL never reads or +writes database-wide `PRAGMA user_version`, so unrelated shared-store version +state is preserved. -Fresh v0.1.4 schema initialization will stamp `PRAGMA user_version = 14`. Running the legacy migration against a current schema will fail when it tries to add an already-present `detail` column, before dropping or rewriting any table. Re-running the migration after a successful upgrade will likewise fail -closed. +closed. Callers remain responsible for rolling back a failed migration +transaction before continuing. ## Tests @@ -25,13 +35,15 @@ Executable SQLite tests will cover: 1. A representative v0.1.3 table upgrades successfully, preserves every existing field, gains `detail`, accepts `apply_audit`, retains append-only - triggers, and ends at `user_version = 14`. -2. A fresh v0.1.4 schema starts at `user_version = 14` and preserves populated + triggers, and preserves a pre-set unrelated `user_version` (for example + `37`). +2. A fresh v0.1.4 schema preserves pre-set database-wide `user_version` values + `0` and `99` while creating the current table shape and preserving populated `ApplyAudit` detail. 3. Applying the legacy migration to a current schema returns an error and - leaves the existing audit row unchanged. + leaves the existing audit row and database-wide `user_version` unchanged. 4. Re-running the migration after a successful legacy upgrade returns an error - and leaves migrated rows unchanged. + and leaves migrated rows and database-wide `user_version` unchanged. Tests will use an in-memory bundled SQLite dependency available only to the crate's test target. @@ -40,5 +52,5 @@ crate's test target. This repair does not add daemon migration orchestration or change the `WardAuditRecord` wire shape. The daemon remains responsible for choosing -schema initialization for new stores and the v0.1.4 migration for legacy -stores. +schema initialization for new stores and the exact-shape-gated v0.1.4 +migration for legacy stores. From 6054dcb470988d057305f7c62ac1a7da119d2ecd Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:03:54 -0500 Subject: [PATCH 12/25] docs: keep migration repair scope focused Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 4 ++-- .../plans/2026-07-19-apply-audit-migration-repair.md | 4 +--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85ad693..bc34d24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,9 +13,9 @@ All notable changes to `coven-threads-core` are documented here. - `WardAuditRecord::for_apply(...)` constructor — builds the audit row for a Gate-4 applied write. - `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. -- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). Future daemon integration must gate this SQL by exact `ward_audit` table shape: legacy means the table exists, `detail` is absent, and the CHECK does not contain `apply_audit`; current means `detail` exists and `apply_audit` is accepted. The SQL preserves database-wide `PRAGMA user_version`. +- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). Bumps `PRAGMA user_version = 14` inside the transaction so future migrations can gate on `user_version < N` instead of substring-sniffing `sqlite_master` DDL. The daemon should run this migration when `PRAGMA user_version < 14`. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. -- New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, and `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`. +- New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `migration_sql_contains_apply_audit_and_detail_column`. ### Design notes diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 703ebc6..ddc35a1 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -15,7 +15,6 @@ - Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, document exact-shape gating, and add executable migration tests. - Modify `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` to document exact table-shape gating and database-wide version preservation. - Modify `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` to align the implementation plan with the approved table-local versioning revision. -- Modify `CHANGELOG.md` to describe table-local migration gating and the updated regression coverage. ### Task 1: Add executable migration regressions @@ -268,8 +267,7 @@ Expected: all audit tests pass. - [ ] **Step 7: Commit the repair** ```bash -git add CHANGELOG.md \ - crates/coven-threads-core/src/audit.rs \ +git add crates/coven-threads-core/src/audit.rs \ docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md git commit -m "fix(audit): keep migration versioning table-local" \ From a884b150abe4f1c6fbb63c703a6903c03dfd9cd1 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:05:25 -0500 Subject: [PATCH 13/25] docs: correct migration repair file scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../plans/2026-07-19-apply-audit-migration-repair.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index ddc35a1..f11bc7d 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -301,8 +301,9 @@ git diff origin/cody/apply-audit-v014...HEAD --stat git status --short --branch ``` -Expected: only the changelog, design, plan, and audit migration files are -changed; the worktree is clean after commits. +Expected: only the design doc, plan doc, `crates/coven-threads-core/Cargo.toml`, +`Cargo.lock`, and `crates/coven-threads-core/src/audit.rs` are changed; the +worktree is clean after commits. - [ ] **Step 3: Record verification in Beads** From 5c1ba0ffed68c343bd57dbf075f340e67e6c5217 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:10:34 -0500 Subject: [PATCH 14/25] docs: align migration plan with table-local gating Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/coven-threads-core/src/audit.rs | 2 +- ...2026-07-19-apply-audit-migration-repair.md | 285 +++++++++--------- ...-19-apply-audit-migration-repair-design.md | 6 +- 3 files changed, 150 insertions(+), 143 deletions(-) diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 999e2dc..4c31c50 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -519,7 +519,7 @@ END; } fn user_version(conn: &Connection) -> i64 { - conn.query_row("PRAGMA user_version;", [], |row| row.get(0)) + conn.pragma_query_value(None, "user_version", |row| row.get(0)) .unwrap() } diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index f11bc7d..64d2e40 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -2,37 +2,49 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Make the v0.1.4 `ward_audit` migration preserve immutable evidence, preserve database-wide version state, and fail closed when invoked outside the exact legacy schema shape. +**Goal:** Make the v0.1.4 `ward_audit` migration preserve immutable evidence, preserve unrelated database-wide version state, and fail closed when invoked outside the exact legacy schema shape. -**Architecture:** Keep migration ownership in `audit.rs`. The legacy migration adds `detail` before rebuilding and copies that column verbatim; callers gate execution by exact `ward_audit` table shape instead of `PRAGMA user_version` or a separate marker. In-memory SQLite tests execute both schema paths and verify data preservation, append-only enforcement, database-wide `user_version` preservation, and safe failure. +**Architecture:** Keep migration ownership in `audit.rs`. The legacy repair adds `detail` before rebuilding and copies that column verbatim; callers decide between fresh schema creation and legacy repair by inspecting the exact `ward_audit` table shape (`detail` column presence plus whether the table CHECK admits `apply_audit`). Ward SQL never writes database-wide `user_version`; the tests preserve and inspect unrelated store-wide values only through rusqlite pragma helpers. -**Tech Stack:** Rust 2021, SQLite, `rusqlite` test dependency, Cargo test. +**Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` dev-dependency for in-memory tests. --- ## File Map -- Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, document exact-shape gating, and add executable migration tests. -- Modify `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` to document exact table-shape gating and database-wide version preservation. -- Modify `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` to align the implementation plan with the approved table-local versioning revision. +- Modify `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` to document exact `ward_audit` shape gating and explicit preservation of unrelated database-wide `user_version`. +- Modify `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` to mirror the landed migration design, exact test names, and final diff expectations. +- Modify `crates/coven-threads-core/Cargo.toml` to add bundled `rusqlite` under `[dev-dependencies]` for executable SQLite migration tests. +- Modify `Cargo.lock` to resolve that test-only dependency. +- Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, document fail-closed table-shape gating, and add executable migration tests. ### Task 1: Add executable migration regressions **Files:** -- Modify: `crates/coven-threads-core/src/audit.rs:435-590` +- Modify: `crates/coven-threads-core/Cargo.toml` +- Modify: `Cargo.lock` +- Modify: `crates/coven-threads-core/src/audit.rs:450-980` -- [ ] **Step 1: Reuse the existing test-only SQLite dependency** +- [ ] **Step 1: Add the bundled SQLite test dependency** -Use the existing bundled `rusqlite` dev-dependency already present for audit -tests; no production dependency or extra migration metadata is needed. +Add the test-only dependency that backs the in-memory migration coverage: -- [ ] **Step 2: Add a representative legacy schema fixture** +```toml +[dev-dependencies] +rusqlite = { version = "0.31", features = ["bundled"] } +``` + +Then refresh `Cargo.lock` so the final changed-file set includes both +`crates/coven-threads-core/Cargo.toml` and `Cargo.lock`. -Inside `audit.rs`'s `tests` module, import `rusqlite::{params, Connection}` and add: +- [ ] **Step 2: Add representative legacy fixtures and pragma helpers** + +Inside `audit.rs`'s `tests` module, add the representative v0.1.3 fixture, +shared constants, and helper functions that match the final tests: ```rust const LEGACY_WARD_AUDIT_SCHEMA_SQL: &str = r#" -CREATE TABLE ward_audit ( +CREATE TABLE IF NOT EXISTS ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_type TEXT NOT NULL CHECK (event_type IN ( 'proposal_submitted','proposal_approved','proposal_rejected', @@ -51,130 +63,125 @@ CREATE TABLE ward_audit ( thread_id TEXT, submitted_at TEXT NOT NULL, decided_at TEXT NOT NULL, - recorded_at TEXT NOT NULL + recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); -CREATE TRIGGER ward_audit_append_only_update +CREATE INDEX IF NOT EXISTS ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); +CREATE INDEX IF NOT EXISTS ward_audit_event_idx ON ward_audit (event_type, recorded_at); + +CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_update BEFORE UPDATE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; -CREATE TRIGGER ward_audit_append_only_delete +CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_delete BEFORE DELETE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; "#; -fn insert_audit_row(connection: &Connection, event_type: &str, detail: Option<&str>) -> i64 { - connection - .execute( - "INSERT INTO ward_audit ( - event_type, proposal_id, familiar_id, ward_version, ward_hash, - tier, decision, approver, diff_hash, detail, files_touched, - channel, thread_id, submitted_at, decided_at, recorded_at - ) VALUES ( - ?1, 'proposal-1', 'familiar-1', '0.1.4', X'0102', - 'tier_2', 'applied', 'principal:val', X'0304', ?2, '[\"SOUL.md\"]', - 'mutation', 'thread-1', '2026-07-19T00:00:00Z', - '2026-07-19T00:00:01Z', '2026-07-19T00:00:02Z' - )", - params![event_type, detail], - ) - .unwrap(); - connection.last_insert_rowid() +fn user_version(conn: &Connection) -> i64 { + conn.pragma_query_value(None, "user_version", |row| row.get(0)) + .unwrap() +} + +fn set_user_version(conn: &Connection, version: i64) { + conn.pragma_update(None, "user_version", version).unwrap(); } ``` -For the legacy test, insert its pre-migration row with a dedicated statement -that omits `detail`, because the v0.1.3 table does not have that column. +Keep the legacy insert path separate from the current-schema insert helper: the +legacy row must omit `detail`, while the current helper inserts an +`apply_audit` row with populated `detail`. -- [ ] **Step 3: Add failing fresh-schema preservation tests** +- [ ] **Step 3: Add the fresh-schema preservation tests** + +Match the final test names exactly by routing both cases through the shared +helper: ```rust +fn assert_fresh_schema_preserves_user_version(initial_version: i64) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, initial_version); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + assert_eq!(user_version(&conn), initial_version); + assert_current_ward_audit_shape(&conn); + + let row_id = insert_current_apply_audit_row(&conn); + let row = load_audit_row(&conn, row_id); + assert_eq!(row.event_type, "apply_audit"); + assert_eq!(row.detail.as_deref(), Some(FIXED_PREV_DETAIL)); +} + #[test] -fn fresh_schema_preserves_user_version_zero() { - let connection = Connection::open_in_memory().unwrap(); - connection.execute_batch("PRAGMA user_version = 0;").unwrap(); - connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - - let version: i64 = connection - .query_row("PRAGMA user_version", [], |row| row.get(0)) - .unwrap(); - assert_eq!(version, 0); +fn fresh_schema_preserves_user_version_zero_and_creates_current_shape() { + assert_fresh_schema_preserves_user_version(0); } #[test] -fn fresh_schema_preserves_user_version_ninety_nine() { - let connection = Connection::open_in_memory().unwrap(); - connection.execute_batch("PRAGMA user_version = 99;").unwrap(); - connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - - let version: i64 = connection - .query_row("PRAGMA user_version", [], |row| row.get(0)) - .unwrap(); - assert_eq!(version, 99); +fn fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape() { + assert_fresh_schema_preserves_user_version(99); } ``` -Each test should also assert the resulting table exposes `detail` and accepts -`apply_audit`. +These tests must verify both outcomes: the schema keeps the unrelated +store-wide `user_version` unchanged, and the resulting table accepts +`apply_audit` with populated `detail`. + +- [ ] **Step 4: Add the current-schema rejection test** -- [ ] **Step 4: Add the failing current-schema preservation test** +Use the current test name and assertions: ```rust #[test] -fn legacy_migration_rejects_current_schema_without_erasing_detail() { - let connection = Connection::open_in_memory().unwrap(); - connection.execute_batch("PRAGMA user_version = 37;").unwrap(); - connection.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - let detail = r#"{"prev_sha256":"abcd","bytes_written":42}"#; - let id = insert_audit_row(&connection, "apply_audit", Some(detail)); - - let error = connection +fn migration_rejects_current_schema_rows_with_detail_and_preserves_state() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_current_ward_audit_shape(&conn); + let row_id = insert_current_apply_audit_row(&conn); + let before = load_audit_row(&conn, row_id); + let before_version = user_version(&conn); + + let err = conn .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) - .expect_err("current schema must reject the legacy migration"); - assert!(error.to_string().contains("duplicate column name: detail")); - let _ = connection.execute_batch("ROLLBACK;"); - - let preserved: String = connection - .query_row( - "SELECT detail FROM ward_audit WHERE id = ?1", - [id], - |row| row.get(0), - ) - .unwrap(); - assert_eq!(preserved, detail); - - let version: i64 = connection - .query_row("PRAGMA user_version", [], |row| row.get(0)) - .unwrap(); - assert_eq!(version, 37); + .expect_err("current-schema migration must fail on duplicate detail"); + assert!(err.to_string().contains("duplicate column name: detail")); + let _ = conn.execute_batch("ROLLBACK;"); + + assert_eq!(load_audit_row(&conn, row_id), before); + assert_current_ward_audit_shape(&conn); + assert_eq!(user_version(&conn), before_version); } ``` - [ ] **Step 5: Add the legacy-upgrade and rerun tests** -The legacy-upgrade test must: - -1. Execute `LEGACY_WARD_AUDIT_SCHEMA_SQL`. -2. Preset an unrelated database-wide `PRAGMA user_version` value (for example - `37`). -3. Insert a fully populated legacy row without `detail`. -4. Execute `WARD_AUDIT_MIGRATION_V014_SQL`. -5. Assert the row's original values remain unchanged and `detail IS NULL`. -6. Assert the rebuilt table now exposes `detail`, accepts `apply_audit`, and - preserves `PRAGMA user_version = 37`. -7. Insert an `apply_audit` row with populated `detail`. -8. Assert both append-only triggers reject `UPDATE` and `DELETE`. - -The rerun test must execute the migration successfully once, invoke it again, -assert the duplicate-`detail` error, roll back the failed transaction, and -confirm the original migrated row and preserved database-wide `user_version` -still exist unchanged. - -- [ ] **Step 6: Run the focused tests and confirm the regression** +Add the two remaining migration tests with their final names: + +1. `legacy_schema_upgrades_and_preserves_append_only_behavior` + - seed the exact legacy table shape; + - preset an unrelated database-wide `user_version` to `37` via + `set_user_version(&conn, 37)`; + - insert one legacy `ward_updated` row without `detail`; + - run `WARD_AUDIT_MIGRATION_V014_SQL`; + - assert the migrated row preserves every original field, now reports + `detail == None`, the rebuilt table accepts `apply_audit`, and the + unrelated `user_version` is still `37`; + - insert a current `apply_audit` row and assert its populated `detail` + survives; + - assert both append-only triggers reject `UPDATE` and `DELETE`. +2. `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows` + - migrate the legacy fixture once successfully; + - insert a current `apply_audit` row after the successful upgrade; + - rerun `WARD_AUDIT_MIGRATION_V014_SQL`, assert the duplicate-`detail` + failure, roll back the failed transaction, and confirm both rows and the + unrelated `user_version` remain unchanged. + +- [ ] **Step 6: Run the focused tests** Run: @@ -182,15 +189,15 @@ Run: cargo test -p coven-threads-core audit::tests -- --nocapture ``` -Expected before the SQL repair: the fresh-schema preservation tests fail because -`WARD_AUDIT_SCHEMA_SQL` overwrites the pre-set database-wide `user_version`, -and the current/legacy/rerun migration tests fail because -`WARD_AUDIT_MIGRATION_V014_SQL` rewrites the same unrelated value to `14`. +Expected after Tasks 1 and 2: all audit tests pass, including the five migration +coverage tests above. ### Task 2: Make the migration preserve evidence and fail closed **Files:** -- Modify: `crates/coven-threads-core/src/audit.rs:302-433` +- Modify: `crates/coven-threads-core/src/audit.rs:309-430` +- Modify: `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` +- Modify: `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` - [ ] **Step 1: Guard the legacy migration before destructive work** @@ -200,8 +207,8 @@ Immediately after `BEGIN;` in `WARD_AUDIT_MIGRATION_V014_SQL`, add: ALTER TABLE ward_audit ADD COLUMN detail TEXT; ``` -This succeeds only for the actual v0.1.3 shape. A current schema or a repeated -upgrade fails before `DROP TABLE ward_audit`. +This succeeds only for the exact v0.1.3 shape. A current schema or repeated +upgrade fails before `DROP TABLE ward_audit` can run. - [ ] **Step 2: Make replacement-table creation strict** @@ -235,24 +242,25 @@ tier, decision, approver, diff_hash, detail, files_touched, channel, - [ ] **Step 4: Preserve database-wide version state** -Remove any `PRAGMA user_version = 14;` statements from -`WARD_AUDIT_SCHEMA_SQL` and `WARD_AUDIT_MIGRATION_V014_SQL`. +Remove the database-wide version writes from `WARD_AUDIT_SCHEMA_SQL` and +`WARD_AUDIT_MIGRATION_V014_SQL`. Fresh DDL and the legacy repair must leave +unrelated `user_version` state untouched; migration choice belongs solely to the +observed `ward_audit` table shape. - [ ] **Step 5: Correct the migration documentation** -Update the module and constant comments to state: +Update the module docs, constant comments, and design doc to state: -- New stores receive the current DDL without mutating database-wide - `user_version`. -- The legacy migration is only for the exact v0.1.3 shape: `ward_audit` - exists, `detail` is absent, and the table CHECK does not contain - `apply_audit`. -- Current v0.1.4 or partial/unknown shapes fail closed rather than running the - migration. -- The migration adds `detail`, copies every row including `detail`, and - preserves database-wide `user_version`. -- Current-schema invocation or rerun fails before destructive work. -- Callers must roll back a failed migration transaction. +- new stores receive the current DDL without mutating database-wide + `user_version`; +- the legacy repair is only for the exact v0.1.3 shape: `ward_audit` exists, + `detail` is absent, and the table CHECK does not contain `apply_audit`; +- current v0.1.4 or partial/unknown shapes fail closed rather than running the + repair; +- the migration adds `detail`, copies every row including `detail`, and + preserves unrelated database-wide `user_version`; +- current-schema invocation or rerun fails before destructive work; +- callers must roll back a failed migration transaction before continuing. - [ ] **Step 6: Run focused tests** @@ -267,9 +275,11 @@ Expected: all audit tests pass. - [ ] **Step 7: Commit the repair** ```bash -git add crates/coven-threads-core/src/audit.rs \ - docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ - docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +git add docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ + docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md \ + crates/coven-threads-core/Cargo.toml \ + Cargo.lock \ + crates/coven-threads-core/src/audit.rs git commit -m "fix(audit): keep migration versioning table-local" \ -m "Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" ``` @@ -277,8 +287,11 @@ git commit -m "fix(audit): keep migration versioning table-local" \ ### Task 3: Validate and prepare PR #7 update **Files:** -- Verify: all changed files -- Update through CLI: Bead `threads-44o`, PR #7 +- Verify: `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` +- Verify: `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` +- Verify: `crates/coven-threads-core/Cargo.toml` +- Verify: `Cargo.lock` +- Verify: `crates/coven-threads-core/src/audit.rs` - [ ] **Step 1: Run repository quality gates** @@ -296,8 +309,8 @@ Expected: both commands exit successfully. Run: ```bash -git diff origin/cody/apply-audit-v014...HEAD --check -git diff origin/cody/apply-audit-v014...HEAD --stat +git diff origin/main...HEAD --check +git diff origin/main...HEAD --stat git status --short --branch ``` @@ -305,13 +318,7 @@ Expected: only the design doc, plan doc, `crates/coven-threads-core/Cargo.toml`, `Cargo.lock`, and `crates/coven-threads-core/src/audit.rs` are changed; the worktree is clean after commits. -- [ ] **Step 3: Record verification in Beads** - -Update `threads-44o` with the branch, worktree, commit SHA, focused test result, -and full quality-gate result. Do not close the Bead until PR #7 is updated and -the fix is accepted or merged. - -- [ ] **Step 4: Re-check remote branch drift** +- [ ] **Step 3: Re-check remote branch drift** Run: @@ -323,7 +330,7 @@ git log --oneline --left-right HEAD...origin/cody/apply-audit-v014 Expected: no remote-only commits. If remote-only commits exist, stop and reconcile before pushing. -- [ ] **Step 5: Update PR #7 and publish review evidence** +- [ ] **Step 4: Update PR #7 and publish review evidence** Push the worktree tip to the existing PR branch only if it is a fast-forward: @@ -331,5 +338,5 @@ Push the worktree tip to the existing PR branch only if it is a fast-forward: git push origin HEAD:cody/apply-audit-v014 ``` -Then comment on PR #7 with the root cause, the fail-closed migration design, -and the exact quality gates that passed. +Then update PR #7 with the root cause, the fail-closed migration design, and +the exact quality gates that passed. diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index ea02757..ada5e6d 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -19,9 +19,9 @@ must inspect `ward_audit` itself before execution: For the exact legacy shape, the migration first adds the nullable `detail` column to the legacy table, then rebuilds the table and copies `detail` into the replacement table. This gives genuine v0.1.3 rows a `NULL` detail value -without hard-coding data loss into the copy step. The SQL never reads or -writes database-wide `PRAGMA user_version`, so unrelated shared-store version -state is preserved. +without hard-coding data loss into the copy step. Ward SQL never reads or +writes database-wide `user_version`, so unrelated shared-store version state is +preserved. Running the legacy migration against a current schema will fail when it tries to add an already-present `detail` column, before dropping or rewriting any From b0900895ce93972697bc66995fd3660e49cc55bd Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:21:28 -0500 Subject: [PATCH 15/25] docs(audit): align PR 7 migration guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 21 ++++++++++++++++--- ...2026-07-19-apply-audit-migration-repair.md | 19 +++++++++++------ 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc34d24..06bf212 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,14 +13,29 @@ All notable changes to `coven-threads-core` are documented here. - `WardAuditRecord::for_apply(...)` constructor — builds the audit row for a Gate-4 applied write. - `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. -- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL to rebuild `ward_audit` for stores created against v0.1.3 (which lacked the `apply_audit` CHECK entry and the `detail` column). Bumps `PRAGMA user_version = 14` inside the transaction so future migrations can gate on `user_version < N` instead of substring-sniffing `sqlite_master` DDL. The daemon should run this migration when `PRAGMA user_version < 14`. +- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving + rebuild of `ward_audit` after exact table-shape inspection. The daemon should + run it only for the legacy v0.1.3 shape (`detail` absent and the table CHECK + does not admit `apply_audit`); current schemas skip it, and partial/unknown + shapes fail closed. The rebuild first adds `detail` to the legacy table, then + copies rows into the replacement table without discarding evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. -- New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `migration_sql_contains_apply_audit_and_detail_column`. +- New tests: `for_apply_produces_correct_shape`, + `for_apply_roundtrips_json`, + `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, + `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, + `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, + `legacy_schema_upgrades_and_preserves_append_only_behavior`, and + `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`. ### Design notes - **Column approach (Option A):** `diff_hash` carries `next_sha256`; `prev_sha256` + `bytes_written` ride in the new `detail` TEXT column as JSON. This avoids a second table rebuild by not adding typed hash columns while keeping the data query-accessible via SQLite JSON functions. -- **Migration approach (Option B-lite):** exports `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can perform a quiet guarded rebuild on upgrade; does not add a rusqlite dependency to coven-threads-core. +- **Migration approach (Option B-lite):** exports + `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can perform a quiet guarded + rebuild on upgrade after exact `ward_audit` shape inspection; production + dependencies stay unchanged, while `coven-threads-core` now carries bundled + `rusqlite` as a dev-dependency for executable migration tests. ## [0.1.3] — prior release diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 64d2e40..4909c0f 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -14,6 +14,7 @@ - Modify `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` to document exact `ward_audit` shape gating and explicit preservation of unrelated database-wide `user_version`. - Modify `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` to mirror the landed migration design, exact test names, and final diff expectations. +- Modify `CHANGELOG.md` to align the v0.1.4 migration notes with the repaired implementation and executable migration regressions. - Modify `crates/coven-threads-core/Cargo.toml` to add bundled `rusqlite` under `[dev-dependencies]` for executable SQLite migration tests. - Modify `Cargo.lock` to resolve that test-only dependency. - Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, document fail-closed table-shape gating, and add executable migration tests. @@ -195,6 +196,7 @@ coverage tests above. ### Task 2: Make the migration preserve evidence and fail closed **Files:** +- Modify: `CHANGELOG.md` - Modify: `crates/coven-threads-core/src/audit.rs:309-430` - Modify: `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` - Modify: `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` @@ -275,7 +277,8 @@ Expected: all audit tests pass. - [ ] **Step 7: Commit the repair** ```bash -git add docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ +git add CHANGELOG.md \ + docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md \ crates/coven-threads-core/Cargo.toml \ Cargo.lock \ @@ -287,6 +290,7 @@ git commit -m "fix(audit): keep migration versioning table-local" \ ### Task 3: Validate and prepare PR #7 update **Files:** +- Verify: `CHANGELOG.md` - Verify: `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` - Verify: `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` - Verify: `crates/coven-threads-core/Cargo.toml` @@ -309,14 +313,17 @@ Expected: both commands exit successfully. Run: ```bash -git diff origin/main...HEAD --check -git diff origin/main...HEAD --stat +git diff origin/cody/apply-audit-v014...HEAD --check +git diff origin/cody/apply-audit-v014...HEAD --stat git status --short --branch ``` -Expected: only the design doc, plan doc, `crates/coven-threads-core/Cargo.toml`, -`Cargo.lock`, and `crates/coven-threads-core/src/audit.rs` are changed; the -worktree is clean after commits. +Expected: only `CHANGELOG.md`, +`docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md`, +`docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md`, +`crates/coven-threads-core/Cargo.toml`, `Cargo.lock`, and +`crates/coven-threads-core/src/audit.rs` are changed; the worktree is clean +after commits. - [ ] **Step 3: Re-check remote branch drift** From c6d279e3e424da18d29bf20f18ebd37f65b6da00 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:38:52 -0500 Subject: [PATCH 16/25] fix(audit): require exact schema fingerprint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 32 +- crates/coven-threads-core/src/audit.rs | 617 ++++++++++++++---- crates/coven-threads-core/src/lib.rs | 6 +- ...2026-07-19-apply-audit-migration-repair.md | 429 +++--------- ...-19-apply-audit-migration-repair-design.md | 100 +-- 5 files changed, 681 insertions(+), 503 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 06bf212..369852c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,29 +13,41 @@ All notable changes to `coven-threads-core` are documented here. - `WardAuditRecord::for_apply(...)` constructor — builds the audit row for a Gate-4 applied write. - `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. +- `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable state tags (`missing`, + `legacy_v013`, `current_v014`, `unknown`) — reusable table-local schema + fingerprint contract for daemon callers. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving - rebuild of `ward_audit` after exact table-shape inspection. The daemon should - run it only for the legacy v0.1.3 shape (`detail` absent and the table CHECK - does not admit `apply_audit`); current schemas skip it, and partial/unknown - shapes fail closed. The rebuild first adds `detail` to the legacy table, then - copies rows into the replacement table without discarding evidence. + rebuild of `ward_audit` guarded by the exact `legacy_v013` fingerprint. The + daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, + migrate only `legacy_v013`, continue on `current_v014`, and fail closed on + `unknown`. The migration independently re-checks `legacy_v013` before any + `ALTER`, then copies rows into the replacement table without discarding + evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, + `schema_state_query_returns_missing_on_empty_db`, + `exact_legacy_fixture_returns_legacy_v013`, + `exact_current_schema_returns_current_v014`, `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, + `legacy_plus_extra_column_and_data_is_unknown_and_guard_preserves_state`, + `current_schema_missing_append_only_trigger_is_unknown_and_update_succeeds`, + `current_schema_with_extra_index_is_unknown`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, and - `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`. + `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, and + `post_alter_failure_rollback_restores_legacy_schema_after_create_conflict`. ### Design notes - **Column approach (Option A):** `diff_hash` carries `next_sha256`; `prev_sha256` + `bytes_written` ride in the new `detail` TEXT column as JSON. This avoids a second table rebuild by not adding typed hash columns while keeping the data query-accessible via SQLite JSON functions. - **Migration approach (Option B-lite):** exports - `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can perform a quiet guarded - rebuild on upgrade after exact `ward_audit` shape inspection; production - dependencies stay unchanged, while `coven-threads-core` now carries bundled - `rusqlite` as a dev-dependency for executable migration tests. + `WARD_AUDIT_SCHEMA_STATE_SQL` and `WARD_AUDIT_MIGRATION_V014_SQL` so the + daemon can perform a quiet, fail-closed upgrade after querying the exact + `ward_audit` schema fingerprint. Production dependencies stay unchanged, + while `coven-threads-core` now carries bundled `rusqlite` as a + dev-dependency for executable migration tests. ## [0.1.3] — prior release diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 4c31c50..c4fe20f 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -22,19 +22,23 @@ //! //! `WARD_AUDIT_SCHEMA_SQL` creates the fresh v0.1.4 `ward_audit` shape without //! mutating database-wide `PRAGMA user_version`. -//! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact v0.1.3 legacy -//! shape. Callers must inspect the `ward_audit` table itself before execution: -//! - exact legacy v0.1.3: the table exists, `detail` is absent, and the table -//! CHECK does not contain `apply_audit`; -//! - current v0.1.4: `detail` exists and the table CHECK accepts -//! `apply_audit`; -//! - partial/unknown shapes: fail closed and do not run the migration. -//! The migration first adds `detail`, then rebuilds `ward_audit` in one -//! transaction so the current CHECK set is installed and every existing row is -//! preserved. -//! Running that migration against a current schema, or rerunning it after a -//! successful upgrade, fails before destructive work; callers must explicitly -//! roll back the failed transaction before continuing. +//! `WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable, table-local fingerprint query +//! that returns one of four stable tags: +//! - `missing` — `ward_audit` does not exist; initialize with +//! `WARD_AUDIT_SCHEMA_SQL`; +//! - `legacy_v013` — the table exactly matches the v0.1.3 legacy fingerprint; +//! run `WARD_AUDIT_MIGRATION_V014_SQL`; +//! - `current_v014` — the table exactly matches the v0.1.4 current +//! fingerprint; continue without schema work; +//! - `unknown` — every other shape, including extra or missing columns, +//! indexes, triggers, or `event_type` CHECK drift; fail closed. +//! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` +//! fingerprint. It independently re-checks that fingerprint inside the +//! transaction before any `ALTER`, then adds `detail` and rebuilds `ward_audit` +//! so the current CHECK set is installed and every existing row is preserved. +//! If a later migration step fails after `ALTER TABLE ... ADD COLUMN detail`, +//! callers must explicitly roll back the failed transaction before continuing +//! so SQLite restores the untouched legacy table. //! //! ## Where content hashes ride for applied writes //! @@ -306,40 +310,234 @@ fn reject_tag(reason: &RejectReason) -> &'static str { } } -/// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). +/// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] when `ward_audit` is +/// absent. +pub const WARD_AUDIT_SCHEMA_STATE_MISSING: &str = "missing"; +/// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact +/// v0.1.3 legacy schema fingerprint. +pub const WARD_AUDIT_SCHEMA_STATE_LEGACY_V013: &str = "legacy_v013"; +/// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact +/// v0.1.4 current schema fingerprint. +pub const WARD_AUDIT_SCHEMA_STATE_CURRENT_V014: &str = "current_v014"; +/// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for every other +/// `ward_audit` shape. +pub const WARD_AUDIT_SCHEMA_STATE_UNKNOWN: &str = "unknown"; + +macro_rules! ward_audit_schema_state_ctes_sql { + () => { + r#" +WITH + ward_audit_table AS ( + SELECT + sql, + replace( + replace( + replace(replace(lower(sql), ' ', ''), char(10), ''), + char(13), + '' + ), + char(9), + '' + ) AS normalized_sql + FROM sqlite_master + WHERE type = 'table' AND name = 'ward_audit' + ), + ward_audit_exists AS ( + SELECT EXISTS(SELECT 1 FROM ward_audit_table) AS ok + ), + ward_audit_column_fingerprint AS ( + SELECT COALESCE( + group_concat( + printf( + '%d|%s|%s|%d|%s|%d', + cid, + name, + type, + "notnull", + COALESCE(dflt_value, ''), + pk + ), + '||' + ), + '' + ) AS fp + FROM ( + SELECT cid, name, type, "notnull", dflt_value, pk + FROM pragma_table_info('ward_audit') + ORDER BY cid + ) + ), + ward_audit_explicit_indexes AS ( + SELECT name, "unique", origin, partial + FROM pragma_index_list('ward_audit') + WHERE origin = 'c' AND name NOT LIKE 'sqlite_autoindex_%' + ), + ward_audit_index_fingerprint AS ( + SELECT COALESCE(group_concat(item, '||'), '') AS fp + FROM ( + SELECT printf( + '%s|%d|%s|%d|%s', + idx.name, + idx."unique", + idx.origin, + idx.partial, + COALESCE(( + SELECT group_concat(printf('%d:%s', seqno, name), ',') + FROM ( + SELECT seqno, name + FROM pragma_index_info(idx.name) + ORDER BY seqno + ) + ), '') + ) AS item + FROM ward_audit_explicit_indexes AS idx + ORDER BY idx.name + ) + ), + ward_audit_trigger_fingerprint AS ( + SELECT COALESCE(group_concat(item, '||'), '') AS fp + FROM ( + SELECT printf( + '%s|%s', + name, + replace( + replace( + replace(replace(lower(COALESCE(sql, '')), ' ', ''), char(10), ''), + char(13), + '' + ), + char(9), + '' + ) + ) AS item + FROM sqlite_master + WHERE type = 'trigger' AND tbl_name = 'ward_audit' + ORDER BY name + ) + ), + ward_audit_shape AS ( + SELECT + (SELECT ok FROM ward_audit_exists) AS table_exists, + COALESCE((SELECT normalized_sql FROM ward_audit_table), '') AS normalized_table_sql, + (SELECT fp FROM ward_audit_column_fingerprint) AS column_fp, + (SELECT fp FROM ward_audit_index_fingerprint) AS index_fp, + (SELECT fp FROM ward_audit_trigger_fingerprint) AS trigger_fp + ) +"# + }; +} + +macro_rules! ward_audit_exact_legacy_predicate_sql { + () => { + r#" +table_exists = 1 +AND instr(normalized_table_sql, 'idintegerprimarykeyautoincrement') > 0 +AND instr( + normalized_table_sql, + 'event_typetextnotnullcheck(event_typein(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger''))' +) > 0 +AND column_fp = '0|id|INTEGER|0||1||1|event_type|TEXT|1||0||2|proposal_id|TEXT|0||0||3|familiar_id|TEXT|1||0||4|ward_version|TEXT|0||0||5|ward_hash|BLOB|1||0||6|tier|TEXT|0||0||7|decision|TEXT|1||0||8|approver|TEXT|0||0||9|diff_hash|BLOB|0||0||10|files_touched|TEXT|1||0||11|channel|TEXT|0||0||12|thread_id|TEXT|0||0||13|submitted_at|TEXT|1||0||14|decided_at|TEXT|1||0||15|recorded_at|TEXT|1|strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')|0' +AND index_fp = 'ward_audit_event_idx|0|c|0|0:event_type,1:recorded_at||ward_audit_familiar_idx|0|c|0|0:familiar_id,1:recorded_at' +AND trigger_fp = 'ward_audit_append_only_delete|createtriggerward_audit_append_only_deletebeforedeleteonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end||ward_audit_append_only_update|createtriggerward_audit_append_only_updatebeforeupdateonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end' +"# + }; +} + +macro_rules! ward_audit_exact_current_predicate_sql { + () => { + r#" +table_exists = 1 +AND instr(normalized_table_sql, 'idintegerprimarykeyautoincrement') > 0 +AND instr( + normalized_table_sql, + 'event_typetextnotnullcheck(event_typein(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger'',''apply_audit''))' +) > 0 +AND column_fp = '0|id|INTEGER|0||1||1|event_type|TEXT|1||0||2|proposal_id|TEXT|0||0||3|familiar_id|TEXT|1||0||4|ward_version|TEXT|0||0||5|ward_hash|BLOB|1||0||6|tier|TEXT|0||0||7|decision|TEXT|1||0||8|approver|TEXT|0||0||9|diff_hash|BLOB|0||0||10|detail|TEXT|0||0||11|files_touched|TEXT|1||0||12|channel|TEXT|0||0||13|thread_id|TEXT|0||0||14|submitted_at|TEXT|1||0||15|decided_at|TEXT|1||0||16|recorded_at|TEXT|1|strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')|0' +AND index_fp = 'ward_audit_event_idx|0|c|0|0:event_type,1:recorded_at||ward_audit_familiar_idx|0|c|0|0:familiar_id,1:recorded_at' +AND trigger_fp = 'ward_audit_append_only_delete|createtriggerward_audit_append_only_deletebeforedeleteonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end||ward_audit_append_only_update|createtriggerward_audit_append_only_updatebeforeupdateonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end' +"# + }; +} + +/// Table-local schema-state query for `ward.audit` inside `coven.sqlite3` +/// (§3.4). /// -/// Append-only is enforced *in the store*: UPDATE and DELETE abort via -/// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). -/// DDL migration for v0.1.3 → v0.1.4: rebuilds `ward_audit` from the exact -/// legacy shape so its CHECK constraint includes `apply_audit` and its copy -/// path preserves `detail`. +/// Callers run this exact query and branch on the stable text result: +/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — initialize with +/// [`WARD_AUDIT_SCHEMA_SQL`]; +/// - [`WARD_AUDIT_SCHEMA_STATE_LEGACY_V013`] — run +/// [`WARD_AUDIT_MIGRATION_V014_SQL`]; +/// - [`WARD_AUDIT_SCHEMA_STATE_CURRENT_V014`] — continue without schema work; +/// - [`WARD_AUDIT_SCHEMA_STATE_UNKNOWN`] — fail closed and investigate the +/// table manually. /// -/// SQLite cannot `ALTER` a CHECK constraint on an existing table. This SQL -/// performs a safe swap in one transaction: -/// 1. Adds the legacy `detail` column so the old table matches the copy shape. -/// 2. Creates `ward_audit_new` with the updated CHECK. -/// 3. Copies all existing rows, preserving `detail`. -/// 4. Drops the old table. -/// 5. Renames the new table into place. -/// 6. Re-creates indexes and append-only triggers. +/// The fingerprint is strict: ordered column metadata (including +/// `recorded_at`'s default and primary-key flags), the exact explicit index +/// set, the exact append-only trigger set, and the `event_type` CHECK +/// signature must all match. Any extra or missing column, index, trigger, or +/// CHECK drift returns `unknown`. +pub const WARD_AUDIT_SCHEMA_STATE_SQL: &str = concat!( + ward_audit_schema_state_ctes_sql!(), + r#" +SELECT CASE + WHEN table_exists = 0 THEN 'missing' + WHEN "#, + ward_audit_exact_legacy_predicate_sql!(), + r#" THEN 'legacy_v013' + WHEN "#, + ward_audit_exact_current_predicate_sql!(), + r#" THEN 'current_v014' + ELSE 'unknown' +END AS schema_state +FROM ward_audit_shape; +"#, +); + +/// DDL migration for the exact `legacy_v013` `ward.audit` fingerprint inside +/// `coven.sqlite3` (§3.4). /// -/// **Caller gate:** inspect `ward_audit` itself before executing this SQL. -/// Run it only when the table exists, `detail` is absent, and the table CHECK -/// does not contain `apply_audit`. If `detail` exists and the table accepts -/// `apply_audit`, the schema is already current. Any partial/unknown shape -/// must fail closed instead of running the migration. +/// Append-only is enforced *in the store*: UPDATE and DELETE abort via +/// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). +/// SQLite cannot `ALTER` a CHECK constraint on an existing table, so this +/// transaction: +/// 1. re-checks the exact legacy fingerprint in SQL before any mutation; +/// 2. adds the legacy `detail` column so the old table matches the copy shape; +/// 3. creates `ward_audit_new` with the updated CHECK; +/// 4. copies every existing row, preserving `detail`; +/// 5. swaps the tables; and +/// 6. re-creates the exact explicit indexes and append-only triggers. /// -/// Current-schema invocation and post-upgrade reruns fail before any -/// destructive step: a current schema aborts on `ALTER TABLE ... ADD COLUMN -/// detail` with a duplicate-column error, while a rerun after a successful -/// upgrade fails on the same `ALTER TABLE` because the column already exists. -/// Callers must `ROLLBACK` failed migration transactions before continuing; -/// the migration remains fail-closed because no later step can run after that -/// early abort. This SQL does not read or write database-wide +/// Callers should still branch on [`WARD_AUDIT_SCHEMA_STATE_SQL`] first: +/// initialize when the state is `missing`, migrate only `legacy_v013`, +/// continue on `current_v014`, and fail closed on `unknown`. This migration +/// independently guards the same `legacy_v013` fingerprint so callers cannot +/// mutate a partial or already-current schema by skipping classification. If a +/// later step fails after `ALTER TABLE ... ADD COLUMN detail`, callers must +/// `ROLLBACK` the failed transaction before continuing so SQLite restores the +/// untouched legacy table. This SQL does not read or write database-wide /// `PRAGMA user_version`. -pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = r#" +pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = concat!( + r#" BEGIN; +CREATE TEMP TABLE ward_audit_migration_guard ( + ok INTEGER NOT NULL CHECK (ok = 1) +); + +INSERT INTO ward_audit_migration_guard (ok) +"#, + ward_audit_schema_state_ctes_sql!(), + r#" +SELECT CASE + WHEN "#, + ward_audit_exact_legacy_predicate_sql!(), + r#" THEN 1 + ELSE 0 +END +FROM ward_audit_shape; + +DROP TABLE ward_audit_migration_guard; + ALTER TABLE ward_audit ADD COLUMN detail TEXT; CREATE TABLE ward_audit_new ( @@ -395,12 +593,13 @@ BEGIN END; COMMIT; -"#; +"#, +); /// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). /// -/// See module docs for shape detection and migration gating. This DDL creates -/// the current `ward_audit` shape without mutating database-wide +/// See module docs for the full schema-state contract. This DDL creates the +/// exact `current_v014` fingerprint without mutating database-wide /// `PRAGMA user_version`. pub const WARD_AUDIT_SCHEMA_SQL: &str = r#" CREATE TABLE IF NOT EXISTS ward_audit ( @@ -447,6 +646,7 @@ mod tests { use super::*; use crate::ids::{SurfaceId, WriterId}; use rusqlite::{params, Connection}; + use std::collections::BTreeSet; const FIXED_SUBMITTED_AT: &str = "2026-07-19T00:00:00.000Z"; const FIXED_DECIDED_AT: &str = "2026-07-19T00:01:00.000Z"; @@ -497,6 +697,13 @@ BEGIN END; "#; + const EXPECTED_EXPLICIT_INDEX_NAMES: &[&str] = + &["ward_audit_event_idx", "ward_audit_familiar_idx"]; + const EXPECTED_TRIGGER_NAMES: &[&str] = &[ + "ward_audit_append_only_delete", + "ward_audit_append_only_update", + ]; + #[derive(Debug, PartialEq, Eq)] struct StoredAuditRow { id: i64, @@ -527,49 +734,61 @@ END; conn.pragma_update(None, "user_version", version).unwrap(); } - fn ward_audit_table_sql(conn: &Connection) -> String { - conn.query_row( - "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'ward_audit';", - [], - |row| row.get(0), - ) - .unwrap() + fn ward_audit_schema_state(conn: &Connection) -> String { + conn.query_row(WARD_AUDIT_SCHEMA_STATE_SQL, [], |row| row.get(0)) + .unwrap() } - fn ward_audit_has_detail_column(conn: &Connection) -> bool { - conn.query_row( - "SELECT EXISTS(SELECT 1 FROM pragma_table_info('ward_audit') WHERE name = 'detail');", - [], - |row| row.get::<_, i64>(0), - ) - .unwrap() - == 1 + fn assert_schema_state(conn: &Connection, expected: &str) { + assert_eq!(ward_audit_schema_state(conn), expected); } - fn ward_audit_accepts_apply_audit(conn: &Connection) -> bool { - ward_audit_table_sql(conn).contains("'apply_audit'") + fn expected_explicit_index_names() -> BTreeSet { + EXPECTED_EXPLICIT_INDEX_NAMES + .iter() + .map(|name| (*name).to_string()) + .collect() } - fn assert_legacy_ward_audit_shape(conn: &Connection) { - assert!( - !ward_audit_has_detail_column(conn), - "legacy schema must not expose detail" - ); - assert!( - !ward_audit_accepts_apply_audit(conn), - "legacy schema CHECK must not mention apply_audit" - ); + fn expected_trigger_names() -> BTreeSet { + EXPECTED_TRIGGER_NAMES + .iter() + .map(|name| (*name).to_string()) + .collect() } - fn assert_current_ward_audit_shape(conn: &Connection) { - assert!( - ward_audit_has_detail_column(conn), - "current schema must expose detail" - ); - assert!( - ward_audit_accepts_apply_audit(conn), - "current schema CHECK must accept apply_audit" - ); + fn explicit_index_names(conn: &Connection) -> BTreeSet { + let mut stmt = conn + .prepare( + "SELECT name FROM pragma_index_list('ward_audit') WHERE origin = 'c' AND name NOT LIKE 'sqlite_autoindex_%' ORDER BY name;", + ) + .unwrap(); + stmt.query_map([], |row| row.get::<_, String>(0)) + .unwrap() + .collect::>() + .unwrap() + } + + fn trigger_names(conn: &Connection) -> BTreeSet { + let mut stmt = conn + .prepare( + "SELECT name FROM sqlite_master WHERE type = 'trigger' AND tbl_name = 'ward_audit' ORDER BY name;", + ) + .unwrap(); + stmt.query_map([], |row| row.get::<_, String>(0)) + .unwrap() + .collect::>() + .unwrap() + } + + fn has_column(conn: &Connection, name: &str) -> bool { + conn.query_row( + "SELECT EXISTS(SELECT 1 FROM pragma_table_info('ward_audit') WHERE name = ?1);", + params![name], + |row| row.get::<_, i64>(0), + ) + .unwrap() + == 1 } fn assert_fresh_schema_preserves_user_version(initial_version: i64) { @@ -578,7 +797,7 @@ END; conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); assert_eq!(user_version(&conn), initial_version); - assert_current_ward_audit_shape(&conn); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); let row_id = insert_current_apply_audit_row(&conn); let row = load_audit_row(&conn, row_id); @@ -656,6 +875,24 @@ END; .unwrap() } + fn load_legacy_extra_value(conn: &Connection, id: i64) -> Option { + conn.query_row( + "SELECT legacy_extra FROM ward_audit WHERE id = ?1;", + params![id], + |row| row.get(0), + ) + .unwrap() + } + + fn ward_audit_new_conflict_value(conn: &Connection) -> String { + conn.query_row( + "SELECT conflict FROM ward_audit_new ORDER BY rowid LIMIT 1;", + [], + |row| row.get(0), + ) + .unwrap() + } + fn insert_current_apply_audit_row(conn: &Connection) -> i64 { conn.execute( r#" @@ -727,6 +964,42 @@ END; conn.last_insert_rowid() } + fn insert_legacy_ward_updated_row_with_extra(conn: &Connection, extra: &str) -> i64 { + conn.execute( + r#" + INSERT INTO ward_audit ( + event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, files_touched, channel, + thread_id, submitted_at, decided_at, recorded_at, legacy_extra + ) VALUES ( + ?1, ?2, ?3, ?4, ?5, + ?6, ?7, ?8, ?9, ?10, ?11, + ?12, ?13, ?14, ?15, ?16 + ) + "#, + params![ + "ward_updated", + Some("proposal-legacy"), + "familiar-legacy", + Some("0.1.3"), + FIXED_WARD_HASH.as_ref(), + Some("tier_1"), + "updated", + Some("writer:legacy"), + Some(FIXED_DIFF_HASH.as_ref()), + FIXED_FILES_TOUCHED, + Some("mutation"), + Some("thread-legacy"), + FIXED_SUBMITTED_AT, + FIXED_DECIDED_AT, + FIXED_RECORDED_AT, + extra, + ], + ) + .unwrap(); + conn.last_insert_rowid() + } + fn request() -> MutationRequest { MutationRequest { surface: SurfaceId::new("SOUL.md"), @@ -856,6 +1129,26 @@ END; assert_eq!(record, back); } + #[test] + fn schema_state_query_returns_missing_on_empty_db() { + let conn = Connection::open_in_memory().unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_MISSING); + } + + #[test] + fn exact_legacy_fixture_returns_legacy_v013() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + } + + #[test] + fn exact_current_schema_returns_current_v014() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + } + #[test] fn fresh_schema_preserves_user_version_zero_and_creates_current_shape() { assert_fresh_schema_preserves_user_version(0); @@ -866,29 +1159,92 @@ END; assert_fresh_schema_preserves_user_version(99); } + #[test] + fn legacy_plus_extra_column_and_data_is_unknown_and_guard_preserves_state() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + conn.execute_batch("ALTER TABLE ward_audit ADD COLUMN legacy_extra TEXT;") + .unwrap(); + let row_id = insert_legacy_ward_updated_row_with_extra(&conn, "survivor"); + let before = load_legacy_audit_row(&conn, row_id); + let before_extra = load_legacy_extra_value(&conn, row_id); + let before_version = user_version(&conn); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("partial legacy schema must fail at the migration guard"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected migration error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_legacy_audit_row(&conn, row_id), before); + assert_eq!(load_legacy_extra_value(&conn, row_id), before_extra); + assert_eq!(user_version(&conn), before_version); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert!(has_column(&conn, "legacy_extra")); + assert!(!has_column(&conn, "detail")); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + } + + #[test] + fn current_schema_missing_append_only_trigger_is_unknown_and_update_succeeds() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + let row_id = insert_current_apply_audit_row(&conn); + + conn.execute_batch("DROP TRIGGER ward_audit_append_only_update;") + .unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + conn.execute( + "UPDATE ward_audit SET decision = 'mutated' WHERE id = ?1;", + params![row_id], + ) + .unwrap(); + assert_eq!(load_audit_row(&conn, row_id).decision, "mutated"); + } + + #[test] + fn current_schema_with_extra_index_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + + conn.execute_batch("CREATE INDEX ward_audit_decision_idx ON ward_audit (decision);") + .unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + #[test] fn migration_rejects_current_schema_rows_with_detail_and_preserves_state() { let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, 37); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_current_ward_audit_shape(&conn); let row_id = insert_current_apply_audit_row(&conn); let before = load_audit_row(&conn, row_id); let before_version = user_version(&conn); - assert_eq!(before_version, 37); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); let err = conn .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) - .expect_err("current-schema migration must fail on duplicate detail"); + .expect_err("current schema migration must fail at the legacy guard"); assert!( - err.to_string().contains("duplicate column name: detail"), + err.to_string().contains("CHECK constraint failed"), "unexpected migration error: {err}" ); - let _ = conn.execute_batch("ROLLBACK;"); + conn.execute_batch("ROLLBACK;").unwrap(); - let after = load_audit_row(&conn, row_id); - assert_eq!(after, before); - assert_current_ward_audit_shape(&conn); + assert_eq!(load_audit_row(&conn, row_id), before); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); assert_eq!(user_version(&conn), before_version); } @@ -897,35 +1253,36 @@ END; let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, 37); conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_legacy_ward_audit_shape(&conn); let row_id = insert_legacy_ward_updated_row(&conn); let before = load_legacy_audit_row(&conn, row_id); - assert_eq!(before.event_type, "ward_updated"); - assert_eq!(before.decision, "updated"); assert_eq!(user_version(&conn), 37); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); - assert_current_ward_audit_shape(&conn); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); assert_eq!(user_version(&conn), 37); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); let after = load_audit_row(&conn, row_id); assert_eq!(after.id, row_id); - assert_eq!(after.event_type, "ward_updated"); - assert_eq!(after.proposal_id.as_deref(), Some("proposal-legacy")); - assert_eq!(after.familiar_id, "familiar-legacy"); - assert_eq!(after.ward_version.as_deref(), Some("0.1.3")); - assert_eq!(after.ward_hash, FIXED_WARD_HASH.to_vec()); - assert_eq!(after.tier.as_deref(), Some("tier_1")); - assert_eq!(after.decision, "updated"); - assert_eq!(after.approver.as_deref(), Some("writer:legacy")); - assert_eq!(after.diff_hash, Some(FIXED_DIFF_HASH.to_vec())); + assert_eq!(after.event_type, before.event_type); + assert_eq!(after.proposal_id, before.proposal_id); + assert_eq!(after.familiar_id, before.familiar_id); + assert_eq!(after.ward_version, before.ward_version); + assert_eq!(after.ward_hash, before.ward_hash); + assert_eq!(after.tier, before.tier); + assert_eq!(after.decision, before.decision); + assert_eq!(after.approver, before.approver); + assert_eq!(after.diff_hash, before.diff_hash); assert_eq!(after.detail, None); - assert_eq!(after.files_touched, FIXED_FILES_TOUCHED); - assert_eq!(after.channel.as_deref(), Some("mutation")); - assert_eq!(after.thread_id.as_deref(), Some("thread-legacy")); - assert_eq!(after.submitted_at, FIXED_SUBMITTED_AT); - assert_eq!(after.decided_at, FIXED_DECIDED_AT); - assert_eq!(after.recorded_at, FIXED_RECORDED_AT); + assert_eq!(after.files_touched, before.files_touched); + assert_eq!(after.channel, before.channel); + assert_eq!(after.thread_id, before.thread_id); + assert_eq!(after.submitted_at, before.submitted_at); + assert_eq!(after.decided_at, before.decided_at); + assert_eq!(after.recorded_at, before.recorded_at); let apply_row_id = insert_current_apply_audit_row(&conn); let apply_row = load_audit_row(&conn, apply_row_id); @@ -951,11 +1308,10 @@ END; let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, 37); conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_legacy_ward_audit_shape(&conn); let legacy_row_id = insert_legacy_ward_updated_row(&conn); conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); - assert_current_ward_audit_shape(&conn); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); assert_eq!(user_version(&conn), 37); let apply_row_id = insert_current_apply_audit_row(&conn); @@ -964,18 +1320,47 @@ END; let err = conn .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) - .expect_err("rerunning the migration must fail on duplicate detail"); + .expect_err("rerunning the migration must fail at the legacy guard"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected migration error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_audit_row(&conn, legacy_row_id), legacy_before); + assert_eq!(load_audit_row(&conn, apply_row_id), apply_before); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!(user_version(&conn), 37); + } + + #[test] + fn post_alter_failure_rollback_restores_legacy_schema_after_create_conflict() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_legacy_ward_updated_row(&conn); + let before = load_legacy_audit_row(&conn, row_id); + conn.execute_batch( + "CREATE TABLE ward_audit_new (conflict TEXT NOT NULL);\nINSERT INTO ward_audit_new (conflict) VALUES ('sentinel');", + ) + .unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("conflicting replacement table must fail after ALTER TABLE succeeds"); assert!( - err.to_string().contains("duplicate column name: detail"), + err.to_string() + .contains("table ward_audit_new already exists"), "unexpected migration error: {err}" ); - let _ = conn.execute_batch("ROLLBACK;"); + conn.execute_batch("ROLLBACK;").unwrap(); - let legacy_after = load_audit_row(&conn, legacy_row_id); - let apply_after = load_audit_row(&conn, apply_row_id); - assert_eq!(legacy_after, legacy_before); - assert_eq!(apply_after, apply_before); - assert_current_ward_audit_shape(&conn); + assert_eq!(load_legacy_audit_row(&conn, row_id), before); assert_eq!(user_version(&conn), 37); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + assert!(!has_column(&conn, "detail")); + assert_eq!(ward_audit_new_conflict_value(&conn), "sentinel"); } } diff --git a/crates/coven-threads-core/src/lib.rs b/crates/coven-threads-core/src/lib.rs index 235a840..052e578 100644 --- a/crates/coven-threads-core/src/lib.rs +++ b/crates/coven-threads-core/src/lib.rs @@ -66,7 +66,11 @@ pub mod thread; pub mod validate; pub mod weave; -pub use audit::{AuditEventType, WardAuditRecord, WARD_AUDIT_SCHEMA_SQL}; +pub use audit::{ + AuditEventType, WardAuditRecord, WARD_AUDIT_MIGRATION_V014_SQL, WARD_AUDIT_SCHEMA_SQL, + WARD_AUDIT_SCHEMA_STATE_CURRENT_V014, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013, + WARD_AUDIT_SCHEMA_STATE_MISSING, WARD_AUDIT_SCHEMA_STATE_SQL, WARD_AUDIT_SCHEMA_STATE_UNKNOWN, +}; pub use channel::Channel; pub use fray::{FrayOrSnap, FrayReason, SnapReason}; pub use ids::{ diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 4909c0f..dd30075 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -1,349 +1,104 @@ # ApplyAudit Migration Repair Implementation Plan -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +**Goal:** Replace the old two-boolean `ward_audit` migration gate with a full +schema fingerprint/state contract, add an in-migration legacy guard, preserve +evidence and unrelated `user_version`, and prove rollback behavior with +executable rusqlite tests. -**Goal:** Make the v0.1.4 `ward_audit` migration preserve immutable evidence, preserve unrelated database-wide version state, and fail closed when invoked outside the exact legacy schema shape. +**Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. +Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on +`missing` / `legacy_v013` / `current_v014` / `unknown`, and embed the same +exact `legacy_v013` predicate inside `WARD_AUDIT_MIGRATION_V014_SQL` before any +`ALTER TABLE`. The migration remains table-local and never writes +database-wide `user_version`. -**Architecture:** Keep migration ownership in `audit.rs`. The legacy repair adds `detail` before rebuilding and copies that column verbatim; callers decide between fresh schema creation and legacy repair by inspecting the exact `ward_audit` table shape (`detail` column presence plus whether the table CHECK admits `apply_audit`). Ward SQL never writes database-wide `user_version`; the tests preserve and inspect unrelated store-wide values only through rusqlite pragma helpers. - -**Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` dev-dependency for in-memory tests. +**Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` for in-memory +tests. --- -## File Map - -- Modify `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` to document exact `ward_audit` shape gating and explicit preservation of unrelated database-wide `user_version`. -- Modify `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` to mirror the landed migration design, exact test names, and final diff expectations. -- Modify `CHANGELOG.md` to align the v0.1.4 migration notes with the repaired implementation and executable migration regressions. -- Modify `crates/coven-threads-core/Cargo.toml` to add bundled `rusqlite` under `[dev-dependencies]` for executable SQLite migration tests. -- Modify `Cargo.lock` to resolve that test-only dependency. -- Modify `crates/coven-threads-core/src/audit.rs` to repair the migration SQL, document fail-closed table-shape gating, and add executable migration tests. - -### Task 1: Add executable migration regressions - -**Files:** -- Modify: `crates/coven-threads-core/Cargo.toml` -- Modify: `Cargo.lock` -- Modify: `crates/coven-threads-core/src/audit.rs:450-980` - -- [ ] **Step 1: Add the bundled SQLite test dependency** - -Add the test-only dependency that backs the in-memory migration coverage: - -```toml -[dev-dependencies] -rusqlite = { version = "0.31", features = ["bundled"] } -``` - -Then refresh `Cargo.lock` so the final changed-file set includes both -`crates/coven-threads-core/Cargo.toml` and `Cargo.lock`. - -- [ ] **Step 2: Add representative legacy fixtures and pragma helpers** - -Inside `audit.rs`'s `tests` module, add the representative v0.1.3 fixture, -shared constants, and helper functions that match the final tests: - -```rust -const LEGACY_WARD_AUDIT_SCHEMA_SQL: &str = r#" -CREATE TABLE IF NOT EXISTS ward_audit ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - event_type TEXT NOT NULL CHECK (event_type IN ( - 'proposal_submitted','proposal_approved','proposal_rejected', - 'proposal_vetoed','ward_updated','validation_verdict', - 'compaction_ledger')), - proposal_id TEXT, - familiar_id TEXT NOT NULL, - ward_version TEXT, - ward_hash BLOB NOT NULL, - tier TEXT, - decision TEXT NOT NULL, - approver TEXT, - diff_hash BLOB, - files_touched TEXT NOT NULL, - channel TEXT, - thread_id TEXT, - submitted_at TEXT NOT NULL, - decided_at TEXT NOT NULL, - recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) -); - -CREATE INDEX IF NOT EXISTS ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); -CREATE INDEX IF NOT EXISTS ward_audit_event_idx ON ward_audit (event_type, recorded_at); - -CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_update -BEFORE UPDATE ON ward_audit -BEGIN - SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); -END; - -CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_delete -BEFORE DELETE ON ward_audit -BEGIN - SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); -END; -"#; - -fn user_version(conn: &Connection) -> i64 { - conn.pragma_query_value(None, "user_version", |row| row.get(0)) - .unwrap() -} - -fn set_user_version(conn: &Connection, version: i64) { - conn.pragma_update(None, "user_version", version).unwrap(); -} -``` - -Keep the legacy insert path separate from the current-schema insert helper: the -legacy row must omit `detail`, while the current helper inserts an -`apply_audit` row with populated `detail`. - -- [ ] **Step 3: Add the fresh-schema preservation tests** - -Match the final test names exactly by routing both cases through the shared -helper: - -```rust -fn assert_fresh_schema_preserves_user_version(initial_version: i64) { - let conn = Connection::open_in_memory().unwrap(); - set_user_version(&conn, initial_version); - conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - - assert_eq!(user_version(&conn), initial_version); - assert_current_ward_audit_shape(&conn); - - let row_id = insert_current_apply_audit_row(&conn); - let row = load_audit_row(&conn, row_id); - assert_eq!(row.event_type, "apply_audit"); - assert_eq!(row.detail.as_deref(), Some(FIXED_PREV_DETAIL)); -} - -#[test] -fn fresh_schema_preserves_user_version_zero_and_creates_current_shape() { - assert_fresh_schema_preserves_user_version(0); -} - -#[test] -fn fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape() { - assert_fresh_schema_preserves_user_version(99); -} -``` - -These tests must verify both outcomes: the schema keeps the unrelated -store-wide `user_version` unchanged, and the resulting table accepts -`apply_audit` with populated `detail`. - -- [ ] **Step 4: Add the current-schema rejection test** - -Use the current test name and assertions: - -```rust -#[test] -fn migration_rejects_current_schema_rows_with_detail_and_preserves_state() { - let conn = Connection::open_in_memory().unwrap(); - set_user_version(&conn, 37); - conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_current_ward_audit_shape(&conn); - let row_id = insert_current_apply_audit_row(&conn); - let before = load_audit_row(&conn, row_id); - let before_version = user_version(&conn); - - let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) - .expect_err("current-schema migration must fail on duplicate detail"); - assert!(err.to_string().contains("duplicate column name: detail")); - let _ = conn.execute_batch("ROLLBACK;"); - - assert_eq!(load_audit_row(&conn, row_id), before); - assert_current_ward_audit_shape(&conn); - assert_eq!(user_version(&conn), before_version); -} -``` - -- [ ] **Step 5: Add the legacy-upgrade and rerun tests** - -Add the two remaining migration tests with their final names: - -1. `legacy_schema_upgrades_and_preserves_append_only_behavior` - - seed the exact legacy table shape; - - preset an unrelated database-wide `user_version` to `37` via - `set_user_version(&conn, 37)`; - - insert one legacy `ward_updated` row without `detail`; - - run `WARD_AUDIT_MIGRATION_V014_SQL`; - - assert the migrated row preserves every original field, now reports - `detail == None`, the rebuilt table accepts `apply_audit`, and the - unrelated `user_version` is still `37`; - - insert a current `apply_audit` row and assert its populated `detail` - survives; - - assert both append-only triggers reject `UPDATE` and `DELETE`. -2. `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows` - - migrate the legacy fixture once successfully; - - insert a current `apply_audit` row after the successful upgrade; - - rerun `WARD_AUDIT_MIGRATION_V014_SQL`, assert the duplicate-`detail` - failure, roll back the failed transaction, and confirm both rows and the - unrelated `user_version` remain unchanged. - -- [ ] **Step 6: Run the focused tests** - -Run: - -```bash -cargo test -p coven-threads-core audit::tests -- --nocapture -``` - -Expected after Tasks 1 and 2: all audit tests pass, including the five migration -coverage tests above. - -### Task 2: Make the migration preserve evidence and fail closed - -**Files:** -- Modify: `CHANGELOG.md` -- Modify: `crates/coven-threads-core/src/audit.rs:309-430` -- Modify: `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` -- Modify: `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` - -- [ ] **Step 1: Guard the legacy migration before destructive work** - -Immediately after `BEGIN;` in `WARD_AUDIT_MIGRATION_V014_SQL`, add: - -```sql -ALTER TABLE ward_audit ADD COLUMN detail TEXT; -``` +## Final File Map -This succeeds only for the exact v0.1.3 shape. A current schema or repeated -upgrade fails before `DROP TABLE ward_audit` can run. +Relative to `origin/cody/apply-audit-v014`, the repaired diff stays scoped to +these seven files: -- [ ] **Step 2: Make replacement-table creation strict** +1. `CHANGELOG.md` +2. `Cargo.lock` +3. `crates/coven-threads-core/Cargo.toml` +4. `crates/coven-threads-core/src/audit.rs` +5. `crates/coven-threads-core/src/lib.rs` +6. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` +7. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` -Change: - -```sql -CREATE TABLE IF NOT EXISTS ward_audit_new ( -``` - -to: - -```sql -CREATE TABLE ward_audit_new ( -``` - -An abandoned or conflicting migration table must abort instead of being reused. - -- [ ] **Step 3: Copy immutable detail instead of nulling it** - -Change the migration's `SELECT` projection from: - -```sql -tier, decision, approver, diff_hash, NULL, files_touched, channel, -``` - -to: - -```sql -tier, decision, approver, diff_hash, detail, files_touched, channel, -``` - -- [ ] **Step 4: Preserve database-wide version state** - -Remove the database-wide version writes from `WARD_AUDIT_SCHEMA_SQL` and -`WARD_AUDIT_MIGRATION_V014_SQL`. Fresh DDL and the legacy repair must leave -unrelated `user_version` state untouched; migration choice belongs solely to the -observed `ward_audit` table shape. - -- [ ] **Step 5: Correct the migration documentation** - -Update the module docs, constant comments, and design doc to state: - -- new stores receive the current DDL without mutating database-wide - `user_version`; -- the legacy repair is only for the exact v0.1.3 shape: `ward_audit` exists, - `detail` is absent, and the table CHECK does not contain `apply_audit`; -- current v0.1.4 or partial/unknown shapes fail closed rather than running the - repair; -- the migration adds `detail`, copies every row including `detail`, and - preserves unrelated database-wide `user_version`; -- current-schema invocation or rerun fails before destructive work; -- callers must roll back a failed migration transaction before continuing. - -- [ ] **Step 6: Run focused tests** - -Run: - -```bash -cargo test -p coven-threads-core audit::tests -- --nocapture -``` - -Expected: all audit tests pass. - -- [ ] **Step 7: Commit the repair** - -```bash -git add CHANGELOG.md \ - docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md \ - docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md \ - crates/coven-threads-core/Cargo.toml \ - Cargo.lock \ - crates/coven-threads-core/src/audit.rs -git commit -m "fix(audit): keep migration versioning table-local" \ - -m "Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" -``` - -### Task 3: Validate and prepare PR #7 update - -**Files:** -- Verify: `CHANGELOG.md` -- Verify: `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` -- Verify: `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` -- Verify: `crates/coven-threads-core/Cargo.toml` -- Verify: `Cargo.lock` -- Verify: `crates/coven-threads-core/src/audit.rs` - -- [ ] **Step 1: Run repository quality gates** - -Run: - -```bash -cargo fmt --all -- --check -cargo test -p coven-threads-core audit::tests -- --nocapture -``` - -Expected: both commands exit successfully. - -- [ ] **Step 2: Review the final diff** - -Run: - -```bash -git diff origin/cody/apply-audit-v014...HEAD --check -git diff origin/cody/apply-audit-v014...HEAD --stat -git status --short --branch -``` - -Expected: only `CHANGELOG.md`, -`docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md`, -`docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md`, -`crates/coven-threads-core/Cargo.toml`, `Cargo.lock`, and -`crates/coven-threads-core/src/audit.rs` are changed; the worktree is clean -after commits. - -- [ ] **Step 3: Re-check remote branch drift** - -Run: - -```bash -git fetch origin cody/apply-audit-v014 -git log --oneline --left-right HEAD...origin/cody/apply-audit-v014 -``` - -Expected: no remote-only commits. If remote-only commits exist, stop and -reconcile before pushing. - -- [ ] **Step 4: Update PR #7 and publish review evidence** - -Push the worktree tip to the existing PR branch only if it is a fast-forward: +--- -```bash -git push origin HEAD:cody/apply-audit-v014 +## Task 1: Lock the behavior down with tests + +- Add schema-state tests for: + - empty DB → `missing`; + - exact legacy fixture → `legacy_v013`; + - exact current schema → `current_v014`. +- Add drift tests for: + - legacy + extra column/data → `unknown`, migration guard failure, explicit + `ROLLBACK`, and preservation of row data, extra column, indexes, triggers, + and unrelated `user_version`; + - current missing one append-only trigger → `unknown`, plus an `UPDATE` + succeeding to prove why it is unknown; + - current with an extra explicit object (index) → `unknown`. +- Update existing migration failure tests so exact current/rerun cases fail at + the new guard and always `ROLLBACK` with `unwrap()`. +- Add post-`ALTER` rollback coverage by precreating conflicting + `ward_audit_new`, forcing `CREATE TABLE ward_audit_new` to fail after the + guard and `ALTER TABLE` succeed, then rolling back and asserting the legacy + row/schema are restored. + +## Task 2: Implement the schema fingerprint/state contract + +- Add stable state tag constants. +- Add public `WARD_AUDIT_SCHEMA_STATE_SQL` that fingerprints: + - ordered column metadata (including `recorded_at` default and PK metadata); + - exact explicit index set, excluding SQLite autoindexes; + - exact append-only trigger set; and + - exact `event_type` CHECK signature, including legacy absence vs current + presence of `apply_audit`. +- Re-export the state query and tags from `crates/coven-threads-core/src/lib.rs` + for root-API callers. +- Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: + - starts a transaction; + - creates a TEMP guard table with `CHECK (ok = 1)`; + - inserts `1` only when the full exact legacy predicate holds, otherwise + inserts `0` and aborts; + - drops the guard on the success path; + - keeps `ALTER ADD detail`, strict replacement-table creation, detail copy, + and explicit index/trigger recreation; + - never writes `PRAGMA user_version`. + +## Task 3: Update the written contract and validate + +- Update `audit.rs` docs, the design doc, the plan doc, and `CHANGELOG.md` to + use the “full schema fingerprint/state contract” terminology instead of the + old two-boolean exact check. +- Document the caller contract explicitly: + - query `WARD_AUDIT_SCHEMA_STATE_SQL`; + - initialize on `missing`; + - migrate only `legacy_v013`; + - continue on `current_v014`; + - fail closed on `unknown`; + - rely on the migration’s independent legacy guard as a second line of + defense. +- Run: + - `cargo fmt` + - focused audit tests + - `cargo test --workspace` if focused tests pass + - `git diff --check` + - stale-wording searches for the old partial-gate terminology. + +## Commit + +Create a new commit (no amend) with: + +```text +fix(audit): require exact schema fingerprint + +Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` - -Then update PR #7 with the root cause, the fail-closed migration design, and -the exact quality gates that passed. diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index ada5e6d..babbc85 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -2,55 +2,77 @@ ## Goal -Repair PR #7 so upgrading a v0.1.3 `ward_audit` table to v0.1.4 cannot -silently erase immutable `detail` evidence, while keeping the change scoped to -the audit schema contract. +Repair PR #7 so `ward_audit` upgrades are gated by a full table-local schema +fingerprint/state contract, not a two-boolean proxy. The upgrade must preserve +immutable evidence, preserve unrelated shared-store `user_version`, reject +partial or drifted schemas before mutation, and remain scoped to the audit +schema contract. -## Design +## Schema-state contract -The v0.1.4 migration remains a table-local repair. Future daemon integration -must inspect `ward_audit` itself before execution: +`crates/coven-threads-core::WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable query +callers run before touching `ward_audit`. It returns one stable tag: -- exact legacy v0.1.3: the table exists, `detail` is absent, and the table - CHECK does not contain `apply_audit`; -- current v0.1.4: `detail` exists and the table accepts `apply_audit`; -- partial/unknown shapes: fail closed and do not run the migration. +- `missing` — `ward_audit` does not exist; initialize with + `WARD_AUDIT_SCHEMA_SQL`. +- `legacy_v013` — exact v0.1.3 legacy fingerprint; run + `WARD_AUDIT_MIGRATION_V014_SQL`. +- `current_v014` — exact v0.1.4 current fingerprint; continue without schema + work. +- `unknown` — every other shape; fail closed and investigate manually. -For the exact legacy shape, the migration first adds the nullable `detail` -column to the legacy table, then rebuilds the table and copies `detail` into -the replacement table. This gives genuine v0.1.3 rows a `NULL` detail value -without hard-coding data loss into the copy step. Ward SQL never reads or -writes database-wide `user_version`, so unrelated shared-store version state is -preserved. +The fingerprint is exact and table-local: -Running the legacy migration against a current schema will fail when it tries -to add an already-present `detail` column, before dropping or rewriting any -table. Re-running the migration after a successful upgrade will likewise fail -closed. Callers remain responsible for rolling back a failed migration -transaction before continuing. +- ordered column metadata from `pragma_table_info('ward_audit')`, including + name, type, `NOT NULL`, `DEFAULT`, and primary-key metadata; +- exact explicit index set (excluding SQLite internal autoindexes); +- exact append-only trigger set; +- exact `event_type` CHECK signature, distinguishing legacy absence vs current + presence of `apply_audit`. -## Tests +Extra or missing columns, indexes, triggers, or CHECK drift all classify as +`unknown`. + +## Migration design + +The v0.1.4 repair remains a table-local migration. The SQL still: + +1. `ALTER TABLE ward_audit ADD COLUMN detail TEXT`; +2. creates a strict replacement `ward_audit_new`; +3. copies every row, preserving `detail`; +4. swaps tables; and +5. re-creates the required explicit indexes and append-only triggers. -Executable SQLite tests will cover: +The change is the guardrail: immediately after `BEGIN;`, the migration creates +a TEMP guard table with `CHECK (ok = 1)` and inserts `1` only if the exact +`legacy_v013` fingerprint holds. Any missing/current/partial schema inserts `0` +instead, aborting before `ALTER TABLE`. This makes the migration fail closed +even if a caller skips the classification query. + +If a later step fails after `ALTER TABLE` (for example, `ward_audit_new` +already exists), callers must explicitly `ROLLBACK` the failed transaction +before continuing so SQLite restores the untouched legacy table. Ward SQL never +reads or writes database-wide `PRAGMA user_version`, so unrelated shared-store +version state is preserved. + +## Tests -1. A representative v0.1.3 table upgrades successfully, preserves every - existing field, gains `detail`, accepts `apply_audit`, retains append-only - triggers, and preserves a pre-set unrelated `user_version` (for example - `37`). -2. A fresh v0.1.4 schema preserves pre-set database-wide `user_version` values - `0` and `99` while creating the current table shape and preserving populated - `ApplyAudit` detail. -3. Applying the legacy migration to a current schema returns an error and - leaves the existing audit row and database-wide `user_version` unchanged. -4. Re-running the migration after a successful legacy upgrade returns an error - and leaves migrated rows and database-wide `user_version` unchanged. +Executable rusqlite tests cover: -Tests will use an in-memory bundled SQLite dependency available only to the -crate's test target. +1. schema-state classification for `missing`, exact `legacy_v013`, and exact + `current_v014`; +2. partial legacy drift (`legacy` + extra column/data) classifying as + `unknown`, rejecting at the guard, and rolling back cleanly without losing + row data, extra columns, indexes, triggers, or unrelated `user_version`; +3. current drift cases such as a missing append-only trigger or an extra index + classifying as `unknown`; +4. exact current/rerun guard failures requiring explicit `ROLLBACK`; +5. successful exact-legacy upgrade landing in `current_v014`; and +6. post-`ALTER` failure rollback restoring the full legacy table state. ## Scope This repair does not add daemon migration orchestration or change the -`WardAuditRecord` wire shape. The daemon remains responsible for choosing -schema initialization for new stores and the exact-shape-gated v0.1.4 -migration for legacy stores. +`WardAuditRecord` wire shape. The daemon remains responsible for calling +`WARD_AUDIT_SCHEMA_STATE_SQL`, choosing fresh initialization vs legacy +migration, and failing closed on `unknown`. From 5018694957b9c4b89e02a8a79023b5350bac2724 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Sun, 19 Jul 2026 23:51:46 -0500 Subject: [PATCH 17/25] fix(audit): fingerprint full table definition Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 12 +- crates/coven-threads-core/src/audit.rs | 227 +++++++++++++++--- ...2026-07-19-apply-audit-migration-repair.md | 42 ++-- ...-19-apply-audit-migration-repair-design.md | 25 +- 4 files changed, 246 insertions(+), 60 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 369852c..e9fe5fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,13 +15,16 @@ All notable changes to `coven-threads-core` are documented here. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. - `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable state tags (`missing`, `legacy_v013`, `current_v014`, `unknown`) — reusable table-local schema - fingerprint contract for daemon callers. + fingerprint contract for daemon callers, with full normalized `CREATE TABLE` + equality covering all declared table-level constraints alongside column, + explicit-index, and trigger fingerprints. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving rebuild of `ward_audit` guarded by the exact `legacy_v013` fingerprint. The daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, migrate only `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The migration independently re-checks `legacy_v013` before any - `ALTER`, then copies rows into the replacement table without discarding + `ALTER`, using the same full normalized-table + column/index/trigger + predicate, then copies rows into the replacement table without discarding evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, @@ -29,9 +32,14 @@ All notable changes to `coven-threads-core` are documented here. `schema_state_query_returns_missing_on_empty_db`, `exact_legacy_fixture_returns_legacy_v013`, `exact_current_schema_returns_current_v014`, + `fresh_and_migrated_current_schemas_share_the_same_normalized_table_sql`, `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, `legacy_plus_extra_column_and_data_is_unknown_and_guard_preserves_state`, + `legacy_schema_with_extra_table_check_is_unknown_and_guard_preserves_constraint`, + `legacy_schema_with_extra_unique_is_unknown_and_guard_preserves_constraint`, + `current_schema_with_extra_table_check_is_unknown`, + `current_schema_with_extra_unique_is_unknown`, `current_schema_missing_append_only_trigger_is_unknown_and_update_succeeds`, `current_schema_with_extra_index_is_unknown`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index c4fe20f..49f6b77 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -30,8 +30,8 @@ //! run `WARD_AUDIT_MIGRATION_V014_SQL`; //! - `current_v014` — the table exactly matches the v0.1.4 current //! fingerprint; continue without schema work; -//! - `unknown` — every other shape, including extra or missing columns, -//! indexes, triggers, or `event_type` CHECK drift; fail closed. +//! - `unknown` — every other shape, including any extra or missing declared +//! table constraint, column, index, or trigger; fail closed. //! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` //! fingerprint. It independently re-checks that fingerprint inside the //! transaction before any `ALTER`, then adds `detail` and rebuilds `ward_audit` @@ -332,7 +332,11 @@ WITH sql, replace( replace( - replace(replace(lower(sql), ' ', ''), char(10), ''), + replace( + replace(replace(sql, '"', ''), ' ', ''), + char(10), + '' + ), char(13), '' ), @@ -402,7 +406,11 @@ WITH name, replace( replace( - replace(replace(lower(COALESCE(sql, '')), ' ', ''), char(10), ''), + replace( + replace(replace(COALESCE(sql, ''), '"', ''), ' ', ''), + char(10), + '' + ), char(13), '' ), @@ -431,14 +439,10 @@ macro_rules! ward_audit_exact_legacy_predicate_sql { () => { r#" table_exists = 1 -AND instr(normalized_table_sql, 'idintegerprimarykeyautoincrement') > 0 -AND instr( - normalized_table_sql, - 'event_typetextnotnullcheck(event_typein(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger''))' -) > 0 +AND normalized_table_sql = 'CREATETABLEward_audit(idINTEGERPRIMARYKEYAUTOINCREMENT,event_typeTEXTNOTNULLCHECK(event_typeIN(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger'')),proposal_idTEXT,familiar_idTEXTNOTNULL,ward_versionTEXT,ward_hashBLOBNOTNULL,tierTEXT,decisionTEXTNOTNULL,approverTEXT,diff_hashBLOB,files_touchedTEXTNOTNULL,channelTEXT,thread_idTEXT,submitted_atTEXTNOTNULL,decided_atTEXTNOTNULL,recorded_atTEXTNOTNULLDEFAULT(strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')))' AND column_fp = '0|id|INTEGER|0||1||1|event_type|TEXT|1||0||2|proposal_id|TEXT|0||0||3|familiar_id|TEXT|1||0||4|ward_version|TEXT|0||0||5|ward_hash|BLOB|1||0||6|tier|TEXT|0||0||7|decision|TEXT|1||0||8|approver|TEXT|0||0||9|diff_hash|BLOB|0||0||10|files_touched|TEXT|1||0||11|channel|TEXT|0||0||12|thread_id|TEXT|0||0||13|submitted_at|TEXT|1||0||14|decided_at|TEXT|1||0||15|recorded_at|TEXT|1|strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')|0' AND index_fp = 'ward_audit_event_idx|0|c|0|0:event_type,1:recorded_at||ward_audit_familiar_idx|0|c|0|0:familiar_id,1:recorded_at' -AND trigger_fp = 'ward_audit_append_only_delete|createtriggerward_audit_append_only_deletebeforedeleteonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end||ward_audit_append_only_update|createtriggerward_audit_append_only_updatebeforeupdateonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end' +AND trigger_fp = 'ward_audit_append_only_delete|CREATETRIGGERward_audit_append_only_deleteBEFOREDELETEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END||ward_audit_append_only_update|CREATETRIGGERward_audit_append_only_updateBEFOREUPDATEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END' "# }; } @@ -447,14 +451,10 @@ macro_rules! ward_audit_exact_current_predicate_sql { () => { r#" table_exists = 1 -AND instr(normalized_table_sql, 'idintegerprimarykeyautoincrement') > 0 -AND instr( - normalized_table_sql, - 'event_typetextnotnullcheck(event_typein(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger'',''apply_audit''))' -) > 0 +AND normalized_table_sql = 'CREATETABLEward_audit(idINTEGERPRIMARYKEYAUTOINCREMENT,event_typeTEXTNOTNULLCHECK(event_typeIN(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger'',''apply_audit'')),proposal_idTEXT,familiar_idTEXTNOTNULL,ward_versionTEXT,ward_hashBLOBNOTNULL,tierTEXT,decisionTEXTNOTNULL,approverTEXT,diff_hashBLOB,detailTEXT,files_touchedTEXTNOTNULL,channelTEXT,thread_idTEXT,submitted_atTEXTNOTNULL,decided_atTEXTNOTNULL,recorded_atTEXTNOTNULLDEFAULT(strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')))' AND column_fp = '0|id|INTEGER|0||1||1|event_type|TEXT|1||0||2|proposal_id|TEXT|0||0||3|familiar_id|TEXT|1||0||4|ward_version|TEXT|0||0||5|ward_hash|BLOB|1||0||6|tier|TEXT|0||0||7|decision|TEXT|1||0||8|approver|TEXT|0||0||9|diff_hash|BLOB|0||0||10|detail|TEXT|0||0||11|files_touched|TEXT|1||0||12|channel|TEXT|0||0||13|thread_id|TEXT|0||0||14|submitted_at|TEXT|1||0||15|decided_at|TEXT|1||0||16|recorded_at|TEXT|1|strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')|0' AND index_fp = 'ward_audit_event_idx|0|c|0|0:event_type,1:recorded_at||ward_audit_familiar_idx|0|c|0|0:familiar_id,1:recorded_at' -AND trigger_fp = 'ward_audit_append_only_delete|createtriggerward_audit_append_only_deletebeforedeleteonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end||ward_audit_append_only_update|createtriggerward_audit_append_only_updatebeforeupdateonward_auditbeginselectraise(abort,''ward_auditisappend-only(rfc-0001§5.6)'');end' +AND trigger_fp = 'ward_audit_append_only_delete|CREATETRIGGERward_audit_append_only_deleteBEFOREDELETEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END||ward_audit_append_only_update|CREATETRIGGERward_audit_append_only_updateBEFOREUPDATEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END' "# }; } @@ -471,11 +471,14 @@ AND trigger_fp = 'ward_audit_append_only_delete|createtriggerward_audit_append_o /// - [`WARD_AUDIT_SCHEMA_STATE_UNKNOWN`] — fail closed and investigate the /// table manually. /// -/// The fingerprint is strict: ordered column metadata (including -/// `recorded_at`'s default and primary-key flags), the exact explicit index -/// set, the exact append-only trigger set, and the `event_type` CHECK -/// signature must all match. Any extra or missing column, index, trigger, or -/// CHECK drift returns `unknown`. +/// The fingerprint is strict: the full normalized `CREATE TABLE` definition, +/// ordered column metadata (including `recorded_at`'s default and primary-key +/// flags), the exact explicit index set, and the exact append-only trigger set +/// must all match. Full table-SQL equality covers every declared table-level +/// constraint (`CHECK`, `UNIQUE`, foreign-key clauses, and the `event_type` +/// list), so any extra or missing column, constraint, index, or trigger +/// returns `unknown`. Normalization removes whitespace and SQLite's renamed- +/// table double quotes without weakening literal or constraint equality. pub const WARD_AUDIT_SCHEMA_STATE_SQL: &str = concat!( ward_audit_schema_state_ctes_sql!(), r#" @@ -500,7 +503,9 @@ FROM ward_audit_shape; /// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). /// SQLite cannot `ALTER` a CHECK constraint on an existing table, so this /// transaction: -/// 1. re-checks the exact legacy fingerprint in SQL before any mutation; +/// 1. re-checks the exact legacy fingerprint in SQL before any mutation, +/// including full normalized `CREATE TABLE` equality plus column/index/ +/// trigger fingerprints; /// 2. adds the legacy `detail` column so the old table matches the copy shape; /// 3. creates `ward_audit_new` with the updated CHECK; /// 4. copies every existing row, preserving `detail`; @@ -616,12 +621,12 @@ CREATE TABLE IF NOT EXISTS ward_audit ( decision TEXT NOT NULL, approver TEXT, diff_hash BLOB, - detail TEXT, -- event-type-specific JSON; see module docs - files_touched TEXT NOT NULL, -- JSON array of surface ids + detail TEXT, + files_touched TEXT NOT NULL, channel TEXT, thread_id TEXT, - submitted_at TEXT NOT NULL, -- RFC 3339 - decided_at TEXT NOT NULL, -- RFC 3339 + submitted_at TEXT NOT NULL, + decided_at TEXT NOT NULL, recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); @@ -743,6 +748,43 @@ END; assert_eq!(ward_audit_schema_state(conn), expected); } + fn normalized_table_sql(conn: &Connection) -> String { + conn.query_row( + concat!( + ward_audit_schema_state_ctes_sql!(), + r#" +SELECT normalized_table_sql +FROM ward_audit_shape; +"# + ), + [], + |row| row.get(0), + ) + .unwrap() + } + + fn schema_sql_with_extra_table_constraint( + base_schema_sql: &str, + constraint_sql: &str, + ) -> String { + let marker = "\n);\n\nCREATE INDEX"; + let replacement = format!(",\n {constraint_sql}{marker}"); + let schema_sql = base_schema_sql.replacen(marker, &replacement, 1); + assert_ne!( + schema_sql, base_schema_sql, + "expected to inject {constraint_sql}" + ); + schema_sql + } + + fn legacy_schema_with_extra_table_constraint(constraint_sql: &str) -> String { + schema_sql_with_extra_table_constraint(LEGACY_WARD_AUDIT_SCHEMA_SQL, constraint_sql) + } + + fn current_schema_with_extra_table_constraint(constraint_sql: &str) -> String { + schema_sql_with_extra_table_constraint(WARD_AUDIT_SCHEMA_SQL, constraint_sql) + } + fn expected_explicit_index_names() -> BTreeSet { EXPECTED_EXPLICIT_INDEX_NAMES .iter() @@ -805,6 +847,39 @@ END; assert_eq!(row.detail.as_deref(), Some(FIXED_PREV_DETAIL)); } + fn assert_legacy_drift_guard_rollback_preserves_state( + constraint_sql: &str, + after_rollback: impl FnOnce(&Connection), + ) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(&legacy_schema_with_extra_table_constraint(constraint_sql)) + .unwrap(); + let row_id = insert_legacy_ward_updated_row(&conn); + let before = load_legacy_audit_row(&conn, row_id); + let before_version = user_version(&conn); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("drifted legacy schema must fail at the migration guard"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected migration error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_legacy_audit_row(&conn, row_id), before); + assert_eq!(user_version(&conn), before_version); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert!(!has_column(&conn, "detail")); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + + after_rollback(&conn); + } + fn load_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { conn.query_row( r#" @@ -929,7 +1004,11 @@ END; conn.last_insert_rowid() } - fn insert_legacy_ward_updated_row(conn: &Connection) -> i64 { + fn try_insert_legacy_ward_updated_row( + conn: &Connection, + decision: &str, + recorded_at: &str, + ) -> rusqlite::Result { conn.execute( r#" INSERT INTO ward_audit ( @@ -949,7 +1028,7 @@ END; Some("0.1.3"), FIXED_WARD_HASH.as_ref(), Some("tier_1"), - "updated", + decision, Some("writer:legacy"), Some(FIXED_DIFF_HASH.as_ref()), FIXED_FILES_TOUCHED, @@ -957,11 +1036,14 @@ END; Some("thread-legacy"), FIXED_SUBMITTED_AT, FIXED_DECIDED_AT, - FIXED_RECORDED_AT, + recorded_at, ], - ) - .unwrap(); - conn.last_insert_rowid() + )?; + Ok(conn.last_insert_rowid()) + } + + fn insert_legacy_ward_updated_row(conn: &Connection) -> i64 { + try_insert_legacy_ward_updated_row(conn, "updated", FIXED_RECORDED_AT).unwrap() } fn insert_legacy_ward_updated_row_with_extra(conn: &Connection, extra: &str) -> i64 { @@ -1149,6 +1231,27 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); } + #[test] + fn fresh_and_migrated_current_schemas_share_the_same_normalized_table_sql() { + let fresh = Connection::open_in_memory().unwrap(); + fresh.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + let migrated = Connection::open_in_memory().unwrap(); + migrated + .execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL) + .unwrap(); + migrated + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .unwrap(); + + assert_schema_state(&fresh, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&migrated, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!( + normalized_table_sql(&fresh), + normalized_table_sql(&migrated) + ); + } + #[test] fn fresh_schema_preserves_user_version_zero_and_creates_current_shape() { assert_fresh_schema_preserves_user_version(0); @@ -1192,6 +1295,66 @@ END; assert_eq!(trigger_names(&conn), expected_trigger_names()); } + #[test] + fn legacy_schema_with_extra_table_check_is_unknown_and_guard_preserves_constraint() { + assert_legacy_drift_guard_rollback_preserves_state( + "CHECK (length(decision) > 0)", + |conn| { + let err = try_insert_legacy_ward_updated_row(conn, "", FIXED_RECORDED_AT) + .expect_err( + "legacy CHECK drift must still reject empty decisions after rollback", + ); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected post-rollback CHECK error: {err}" + ); + }, + ); + } + + #[test] + fn legacy_schema_with_extra_unique_is_unknown_and_guard_preserves_constraint() { + assert_legacy_drift_guard_rollback_preserves_state( + "UNIQUE (decision, recorded_at)", + |conn| { + let err = try_insert_legacy_ward_updated_row(conn, "updated", FIXED_RECORDED_AT) + .expect_err( + "legacy UNIQUE drift must still reject duplicate decision/recorded_at rows after rollback", + ); + assert!( + err.to_string().contains("UNIQUE constraint failed"), + "unexpected post-rollback UNIQUE error: {err}" + ); + }, + ); + } + + #[test] + fn current_schema_with_extra_table_check_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(¤t_schema_with_extra_table_constraint( + "CHECK (length(decision) > 0)", + )) + .unwrap(); + + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + + #[test] + fn current_schema_with_extra_unique_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(¤t_schema_with_extra_table_constraint( + "UNIQUE (decision, recorded_at)", + )) + .unwrap(); + + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + #[test] fn current_schema_missing_append_only_trigger_is_unknown_and_update_succeeds() { let conn = Connection::open_in_memory().unwrap(); diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index dd30075..8a2c548 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -1,9 +1,10 @@ # ApplyAudit Migration Repair Implementation Plan **Goal:** Replace the old two-boolean `ward_audit` migration gate with a full -schema fingerprint/state contract, add an in-migration legacy guard, preserve -evidence and unrelated `user_version`, and prove rollback behavior with -executable rusqlite tests. +schema fingerprint/state contract built around normalized `CREATE TABLE` +equality plus column/index/trigger fingerprints, add an in-migration legacy +guard, preserve evidence and unrelated `user_version`, and prove rollback +behavior with executable rusqlite tests. **Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on @@ -19,16 +20,13 @@ tests. ## Final File Map -Relative to `origin/cody/apply-audit-v014`, the repaired diff stays scoped to -these seven files: +For this follow-up fingerprint repair, the diff stays scoped to these four +files: 1. `CHANGELOG.md` -2. `Cargo.lock` -3. `crates/coven-threads-core/Cargo.toml` -4. `crates/coven-threads-core/src/audit.rs` -5. `crates/coven-threads-core/src/lib.rs` -6. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` -7. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` +2. `crates/coven-threads-core/src/audit.rs` +3. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` +4. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` --- @@ -42,6 +40,12 @@ these seven files: - legacy + extra column/data → `unknown`, migration guard failure, explicit `ROLLBACK`, and preservation of row data, extra column, indexes, triggers, and unrelated `user_version`; + - legacy/current with an extra table-level `CHECK (length(decision) > 0)` → + `unknown`, with the legacy guard failing before mutation and rollback + preserving the original CHECK behavior; + - legacy/current with an extra table-level `UNIQUE (decision, recorded_at)` + → `unknown`, with the legacy guard failing before mutation and rollback + preserving the original UNIQUE behavior; - current missing one append-only trigger → `unknown`, plus an `UPDATE` succeeding to prove why it is unknown; - current with an extra explicit object (index) → `unknown`. @@ -51,16 +55,21 @@ these seven files: `ward_audit_new`, forcing `CREATE TABLE ward_audit_new` to fail after the guard and `ALTER TABLE` succeed, then rolling back and asserting the legacy row/schema are restored. +- Add a normalization-equivalence test proving fresh current and successfully + migrated current both classify `current_v014` under the same normalized table + definition. ## Task 2: Implement the schema fingerprint/state contract - Add stable state tag constants. - Add public `WARD_AUDIT_SCHEMA_STATE_SQL` that fingerprints: + - full normalized `CREATE TABLE ward_audit (...)` equality, covering all + table-level constraints; - ordered column metadata (including `recorded_at` default and PK metadata); - exact explicit index set, excluding SQLite autoindexes; - exact append-only trigger set; and - - exact `event_type` CHECK signature, including legacy absence vs current - presence of `apply_audit`. + - legacy/current event-list differences inside the normalized table + definition, including the presence of `apply_audit`. - Re-export the state query and tags from `crates/coven-threads-core/src/lib.rs` for root-API callers. - Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: @@ -76,8 +85,9 @@ these seven files: ## Task 3: Update the written contract and validate - Update `audit.rs` docs, the design doc, the plan doc, and `CHANGELOG.md` to - use the “full schema fingerprint/state contract” terminology instead of the - old two-boolean exact check. + say that full normalized `CREATE TABLE` equality covers every declared + table-level constraint, alongside column/index/trigger fingerprints, instead + of the old partial exact-check wording. - Document the caller contract explicitly: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - initialize on `missing`; @@ -98,7 +108,7 @@ these seven files: Create a new commit (no amend) with: ```text -fix(audit): require exact schema fingerprint +fix(audit): fingerprint full table definition Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index babbc85..4bcfe6e 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -23,15 +23,16 @@ callers run before touching `ward_audit`. It returns one stable tag: The fingerprint is exact and table-local: +- full normalized `CREATE TABLE ward_audit (...)` equality; - ordered column metadata from `pragma_table_info('ward_audit')`, including name, type, `NOT NULL`, `DEFAULT`, and primary-key metadata; -- exact explicit index set (excluding SQLite internal autoindexes); -- exact append-only trigger set; -- exact `event_type` CHECK signature, distinguishing legacy absence vs current - presence of `apply_audit`. +- exact explicit index set (excluding SQLite internal autoindexes); and +- exact append-only trigger set. -Extra or missing columns, indexes, triggers, or CHECK drift all classify as -`unknown`. +The full normalized table definition covers every declared table-level +constraint — the `event_type` CHECK list, extra `CHECK` clauses, `UNIQUE` +clauses, and foreign-key clauses — so any extra or missing constraint, column, +index, or trigger classifies as `unknown`. ## Migration design @@ -64,11 +65,15 @@ Executable rusqlite tests cover: 2. partial legacy drift (`legacy` + extra column/data) classifying as `unknown`, rejecting at the guard, and rolling back cleanly without losing row data, extra columns, indexes, triggers, or unrelated `user_version`; -3. current drift cases such as a missing append-only trigger or an extra index +3. legacy/current drift caused by extra table-level `CHECK` or `UNIQUE` + constraints classifying as `unknown`, with legacy guard failures requiring + explicit `ROLLBACK` and preserving the original constraint behavior; +4. current drift cases such as a missing append-only trigger or an extra index classifying as `unknown`; -4. exact current/rerun guard failures requiring explicit `ROLLBACK`; -5. successful exact-legacy upgrade landing in `current_v014`; and -6. post-`ALTER` failure rollback restoring the full legacy table state. +5. exact current/rerun guard failures requiring explicit `ROLLBACK`; +6. successful exact-legacy upgrade landing in `current_v014`, with fresh and + migrated current schemas sharing the same normalized table definition; and +7. post-`ALTER` failure rollback restoring the full legacy table state. ## Scope From 1b5631734e653e904fd1d6a30b358ac6425bff9c Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 00:07:24 -0500 Subject: [PATCH 18/25] fix(audit): compare exact stored schema SQL Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 21 +- crates/coven-threads-core/src/audit.rs | 487 ++++++++++++++---- ...2026-07-19-apply-audit-migration-repair.md | 44 +- ...-19-apply-audit-migration-repair-design.md | 41 +- 4 files changed, 462 insertions(+), 131 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9fe5fe..a95a0ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,24 +15,27 @@ All notable changes to `coven-threads-core` are documented here. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. - `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable state tags (`missing`, `legacy_v013`, `current_v014`, `unknown`) — reusable table-local schema - fingerprint contract for daemon callers, with full normalized `CREATE TABLE` - equality covering all declared table-level constraints alongside column, - explicit-index, and trigger fingerprints. + fingerprint contract for daemon callers, using exact stored + `sqlite_master.sql` fingerprints for the `ward_audit` table, explicit + indexes, and append-only triggers, plus ordered column metadata. Only the + controlled fresh/migrated v0.1.4 table SQL variants and the shipped v0.1.3 + table SQL are accepted; no whitespace-destroying normalization is applied. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving rebuild of `ward_audit` guarded by the exact `legacy_v013` fingerprint. The daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, migrate only `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The migration independently re-checks `legacy_v013` before any - `ALTER`, using the same full normalized-table + column/index/trigger - predicate, then copies rows into the replacement table without discarding - evidence. + `ALTER`, using the same exact stored-table + column/index/trigger predicate, + then copies rows into the replacement table without discarding evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `schema_state_query_returns_missing_on_empty_db`, `exact_legacy_fixture_returns_legacy_v013`, `exact_current_schema_returns_current_v014`, - `fresh_and_migrated_current_schemas_share_the_same_normalized_table_sql`, + `fresh_and_migrated_current_schemas_use_controlled_exact_sql_variants`, + `current_schema_with_spaced_event_type_literal_is_unknown`, + `legacy_schema_with_spaced_event_type_literal_is_unknown_and_guard_preserves_state`, `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, `legacy_plus_extra_column_and_data_is_unknown_and_guard_preserves_state`, @@ -42,6 +45,10 @@ All notable changes to `coven-threads-core` are documented here. `current_schema_with_extra_unique_is_unknown`, `current_schema_missing_append_only_trigger_is_unknown_and_update_succeeds`, `current_schema_with_extra_index_is_unknown`, + `current_schema_with_desc_index_is_unknown`, + `current_schema_with_collated_index_is_unknown`, + `current_schema_with_altered_trigger_error_literal_is_unknown`, + `current_schema_with_altered_trigger_body_is_unknown`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, and `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, and diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 49f6b77..52ca4c3 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -32,6 +32,13 @@ //! fingerprint; continue without schema work; //! - `unknown` — every other shape, including any extra or missing declared //! table constraint, column, index, or trigger; fail closed. +//! The fingerprint uses exact `sqlite_master.sql` text for the table, explicit +//! indexes, and append-only triggers, plus ordered `pragma_table_info` +//! metadata. It does **not** normalize whitespace: the only accepted +//! `current_v014` table-SQL variants are the fresh `CREATE TABLE ward_audit` +//! form and the quoted `CREATE TABLE "ward_audit"` form SQLite stores after +//! the exact legacy migration path, and the `legacy_v013` fingerprint includes +//! the inline comments preserved from the shipped v0.1.3 DDL. //! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` //! fingerprint. It independently re-checks that fingerprint inside the //! transaction before any `ALTER`, then adds `detail` and rebuilds `ward_audit` @@ -328,21 +335,7 @@ macro_rules! ward_audit_schema_state_ctes_sql { r#" WITH ward_audit_table AS ( - SELECT - sql, - replace( - replace( - replace( - replace(replace(sql, '"', ''), ' ', ''), - char(10), - '' - ), - char(13), - '' - ), - char(9), - '' - ) AS normalized_sql + SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'ward_audit' ), @@ -371,53 +364,19 @@ WITH ORDER BY cid ) ), - ward_audit_explicit_indexes AS ( - SELECT name, "unique", origin, partial - FROM pragma_index_list('ward_audit') - WHERE origin = 'c' AND name NOT LIKE 'sqlite_autoindex_%' - ), ward_audit_index_fingerprint AS ( SELECT COALESCE(group_concat(item, '||'), '') AS fp FROM ( - SELECT printf( - '%s|%d|%s|%d|%s', - idx.name, - idx."unique", - idx.origin, - idx.partial, - COALESCE(( - SELECT group_concat(printf('%d:%s', seqno, name), ',') - FROM ( - SELECT seqno, name - FROM pragma_index_info(idx.name) - ORDER BY seqno - ) - ), '') - ) AS item - FROM ward_audit_explicit_indexes AS idx - ORDER BY idx.name + SELECT printf('%s|%s', name, COALESCE(sql, '')) AS item + FROM sqlite_master + WHERE type = 'index' AND tbl_name = 'ward_audit' AND sql IS NOT NULL + ORDER BY name ) ), ward_audit_trigger_fingerprint AS ( SELECT COALESCE(group_concat(item, '||'), '') AS fp FROM ( - SELECT printf( - '%s|%s', - name, - replace( - replace( - replace( - replace(replace(COALESCE(sql, ''), '"', ''), ' ', ''), - char(10), - '' - ), - char(13), - '' - ), - char(9), - '' - ) - ) AS item + SELECT printf('%s|%s', name, COALESCE(sql, '')) AS item FROM sqlite_master WHERE type = 'trigger' AND tbl_name = 'ward_audit' ORDER BY name @@ -426,7 +385,7 @@ WITH ward_audit_shape AS ( SELECT (SELECT ok FROM ward_audit_exists) AS table_exists, - COALESCE((SELECT normalized_sql FROM ward_audit_table), '') AS normalized_table_sql, + COALESCE((SELECT sql FROM ward_audit_table), '') AS table_sql, (SELECT fp FROM ward_audit_column_fingerprint) AS column_fp, (SELECT fp FROM ward_audit_index_fingerprint) AS index_fp, (SELECT fp FROM ward_audit_trigger_fingerprint) AS trigger_fp @@ -435,27 +394,145 @@ WITH }; } +macro_rules! ward_audit_exact_legacy_table_sql_sql { + () => { + r#"'CREATE TABLE ward_audit ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL CHECK (event_type IN ( + ''proposal_submitted'',''proposal_approved'',''proposal_rejected'', + ''proposal_vetoed'',''ward_updated'',''validation_verdict'', + ''compaction_ledger'')), + proposal_id TEXT, + familiar_id TEXT NOT NULL, + ward_version TEXT, + ward_hash BLOB NOT NULL, + tier TEXT, + decision TEXT NOT NULL, + approver TEXT, + diff_hash BLOB, + files_touched TEXT NOT NULL, -- JSON array of surface ids + channel TEXT, + thread_id TEXT, + submitted_at TEXT NOT NULL, -- RFC 3339 + decided_at TEXT NOT NULL, -- RFC 3339 + recorded_at TEXT NOT NULL DEFAULT (strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')) +)'"# + }; +} + +macro_rules! ward_audit_exact_current_fresh_table_sql_sql { + () => { + r#"'CREATE TABLE ward_audit ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL CHECK (event_type IN ( + ''proposal_submitted'',''proposal_approved'',''proposal_rejected'', + ''proposal_vetoed'',''ward_updated'',''validation_verdict'', + ''compaction_ledger'',''apply_audit'')), + proposal_id TEXT, + familiar_id TEXT NOT NULL, + ward_version TEXT, + ward_hash BLOB NOT NULL, + tier TEXT, + decision TEXT NOT NULL, + approver TEXT, + diff_hash BLOB, + detail TEXT, + files_touched TEXT NOT NULL, + channel TEXT, + thread_id TEXT, + submitted_at TEXT NOT NULL, + decided_at TEXT NOT NULL, + recorded_at TEXT NOT NULL DEFAULT (strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')) +)'"# + }; +} + +macro_rules! ward_audit_exact_current_migrated_table_sql_sql { + () => { + r#"'CREATE TABLE "ward_audit" ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL CHECK (event_type IN ( + ''proposal_submitted'',''proposal_approved'',''proposal_rejected'', + ''proposal_vetoed'',''ward_updated'',''validation_verdict'', + ''compaction_ledger'',''apply_audit'')), + proposal_id TEXT, + familiar_id TEXT NOT NULL, + ward_version TEXT, + ward_hash BLOB NOT NULL, + tier TEXT, + decision TEXT NOT NULL, + approver TEXT, + diff_hash BLOB, + detail TEXT, + files_touched TEXT NOT NULL, + channel TEXT, + thread_id TEXT, + submitted_at TEXT NOT NULL, + decided_at TEXT NOT NULL, + recorded_at TEXT NOT NULL DEFAULT (strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')) +)'"# + }; +} + +macro_rules! ward_audit_exact_explicit_index_fp_sql { + () => { + r#"'ward_audit_event_idx|CREATE INDEX ward_audit_event_idx ON ward_audit (event_type, recorded_at)||ward_audit_familiar_idx|CREATE INDEX ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at)'"# + }; +} + +macro_rules! ward_audit_exact_trigger_fp_sql { + () => { + r#"'ward_audit_append_only_delete|CREATE TRIGGER ward_audit_append_only_delete +BEFORE DELETE ON ward_audit +BEGIN + SELECT RAISE(ABORT, ''ward_audit is append-only (RFC-0001 §5.6)''); +END||ward_audit_append_only_update|CREATE TRIGGER ward_audit_append_only_update +BEFORE UPDATE ON ward_audit +BEGIN + SELECT RAISE(ABORT, ''ward_audit is append-only (RFC-0001 §5.6)''); +END'"# + }; +} + macro_rules! ward_audit_exact_legacy_predicate_sql { () => { - r#" + concat!( + r#" table_exists = 1 -AND normalized_table_sql = 'CREATETABLEward_audit(idINTEGERPRIMARYKEYAUTOINCREMENT,event_typeTEXTNOTNULLCHECK(event_typeIN(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger'')),proposal_idTEXT,familiar_idTEXTNOTNULL,ward_versionTEXT,ward_hashBLOBNOTNULL,tierTEXT,decisionTEXTNOTNULL,approverTEXT,diff_hashBLOB,files_touchedTEXTNOTNULL,channelTEXT,thread_idTEXT,submitted_atTEXTNOTNULL,decided_atTEXTNOTNULL,recorded_atTEXTNOTNULLDEFAULT(strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')))' +AND table_sql = "#, + ward_audit_exact_legacy_table_sql_sql!(), + r#" AND column_fp = '0|id|INTEGER|0||1||1|event_type|TEXT|1||0||2|proposal_id|TEXT|0||0||3|familiar_id|TEXT|1||0||4|ward_version|TEXT|0||0||5|ward_hash|BLOB|1||0||6|tier|TEXT|0||0||7|decision|TEXT|1||0||8|approver|TEXT|0||0||9|diff_hash|BLOB|0||0||10|files_touched|TEXT|1||0||11|channel|TEXT|0||0||12|thread_id|TEXT|0||0||13|submitted_at|TEXT|1||0||14|decided_at|TEXT|1||0||15|recorded_at|TEXT|1|strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')|0' -AND index_fp = 'ward_audit_event_idx|0|c|0|0:event_type,1:recorded_at||ward_audit_familiar_idx|0|c|0|0:familiar_id,1:recorded_at' -AND trigger_fp = 'ward_audit_append_only_delete|CREATETRIGGERward_audit_append_only_deleteBEFOREDELETEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END||ward_audit_append_only_update|CREATETRIGGERward_audit_append_only_updateBEFOREUPDATEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END' +AND index_fp = "#, + ward_audit_exact_explicit_index_fp_sql!(), + r#" +AND trigger_fp = "#, + ward_audit_exact_trigger_fp_sql!(), + r#" "# + ) }; } macro_rules! ward_audit_exact_current_predicate_sql { () => { - r#" + concat!( + r#" table_exists = 1 -AND normalized_table_sql = 'CREATETABLEward_audit(idINTEGERPRIMARYKEYAUTOINCREMENT,event_typeTEXTNOTNULLCHECK(event_typeIN(''proposal_submitted'',''proposal_approved'',''proposal_rejected'',''proposal_vetoed'',''ward_updated'',''validation_verdict'',''compaction_ledger'',''apply_audit'')),proposal_idTEXT,familiar_idTEXTNOTNULL,ward_versionTEXT,ward_hashBLOBNOTNULL,tierTEXT,decisionTEXTNOTNULL,approverTEXT,diff_hashBLOB,detailTEXT,files_touchedTEXTNOTNULL,channelTEXT,thread_idTEXT,submitted_atTEXTNOTNULL,decided_atTEXTNOTNULL,recorded_atTEXTNOTNULLDEFAULT(strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')))' +AND table_sql IN ("#, + ward_audit_exact_current_fresh_table_sql_sql!(), + r#", "#, + ward_audit_exact_current_migrated_table_sql_sql!(), + r#") AND column_fp = '0|id|INTEGER|0||1||1|event_type|TEXT|1||0||2|proposal_id|TEXT|0||0||3|familiar_id|TEXT|1||0||4|ward_version|TEXT|0||0||5|ward_hash|BLOB|1||0||6|tier|TEXT|0||0||7|decision|TEXT|1||0||8|approver|TEXT|0||0||9|diff_hash|BLOB|0||0||10|detail|TEXT|0||0||11|files_touched|TEXT|1||0||12|channel|TEXT|0||0||13|thread_id|TEXT|0||0||14|submitted_at|TEXT|1||0||15|decided_at|TEXT|1||0||16|recorded_at|TEXT|1|strftime(''%Y-%m-%dT%H:%M:%fZ'',''now'')|0' -AND index_fp = 'ward_audit_event_idx|0|c|0|0:event_type,1:recorded_at||ward_audit_familiar_idx|0|c|0|0:familiar_id,1:recorded_at' -AND trigger_fp = 'ward_audit_append_only_delete|CREATETRIGGERward_audit_append_only_deleteBEFOREDELETEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END||ward_audit_append_only_update|CREATETRIGGERward_audit_append_only_updateBEFOREUPDATEONward_auditBEGINSELECTRAISE(ABORT,''ward_auditisappend-only(RFC-0001§5.6)'');END' +AND index_fp = "#, + ward_audit_exact_explicit_index_fp_sql!(), + r#" +AND trigger_fp = "#, + ward_audit_exact_trigger_fp_sql!(), + r#" "# + ) }; } @@ -471,14 +548,18 @@ AND trigger_fp = 'ward_audit_append_only_delete|CREATETRIGGERward_audit_append_o /// - [`WARD_AUDIT_SCHEMA_STATE_UNKNOWN`] — fail closed and investigate the /// table manually. /// -/// The fingerprint is strict: the full normalized `CREATE TABLE` definition, -/// ordered column metadata (including `recorded_at`'s default and primary-key -/// flags), the exact explicit index set, and the exact append-only trigger set -/// must all match. Full table-SQL equality covers every declared table-level -/// constraint (`CHECK`, `UNIQUE`, foreign-key clauses, and the `event_type` -/// list), so any extra or missing column, constraint, index, or trigger -/// returns `unknown`. Normalization removes whitespace and SQLite's renamed- -/// table double quotes without weakening literal or constraint equality. +/// The fingerprint is strict: the exact `sqlite_master.sql` stored for +/// `ward_audit`, ordered column metadata (including `recorded_at`'s default and +/// primary-key flags), the exact explicit index SQL set, and the exact +/// append-only trigger SQL set must all match. Full stored-table-SQL equality +/// covers every declared table-level constraint (`CHECK`, `UNIQUE`, foreign-key +/// clauses, and the `event_type` list), so any extra or missing column, +/// constraint, index, or trigger returns `unknown`. No whitespace-destroying +/// normalization is applied: the only accepted `current_v014` table SQL +/// variants are the fresh `CREATE TABLE ward_audit (...)` form and SQLite's +/// quoted `CREATE TABLE "ward_audit" (...)` form produced by the exact legacy +/// migration path, while the `legacy_v013` fingerprint intentionally includes +/// the inline comments preserved from the shipped v0.1.3 DDL. pub const WARD_AUDIT_SCHEMA_STATE_SQL: &str = concat!( ward_audit_schema_state_ctes_sql!(), r#" @@ -504,7 +585,7 @@ FROM ward_audit_shape; /// SQLite cannot `ALTER` a CHECK constraint on an existing table, so this /// transaction: /// 1. re-checks the exact legacy fingerprint in SQL before any mutation, -/// including full normalized `CREATE TABLE` equality plus column/index/ +/// including exact stored `sqlite_master.sql` equality plus column/index/ /// trigger fingerprints; /// 2. adds the legacy `detail` column so the old table matches the copy shape; /// 3. creates `ward_audit_new` with the updated CHECK; @@ -661,8 +742,9 @@ mod tests { const FIXED_PREV_DETAIL: &str = r#"{"prev_sha256":"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc","bytes_written":42}"#; const FIXED_FILES_TOUCHED: &str = r#"["SOUL.md"]"#; - /// Representative v0.1.3 `ward_audit` schema: no `detail` column and no - /// `apply_audit` event tag, but still append-only. + /// Exact shipped v0.1.3 `ward_audit` DDL from `origin/main` / the PR base. + /// Keep the inline comments: SQLite preserves them in `sqlite_master.sql`, + /// and the legacy fingerprint intentionally matches that stored text. const LEGACY_WARD_AUDIT_SCHEMA_SQL: &str = r#" CREATE TABLE IF NOT EXISTS ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, @@ -678,11 +760,11 @@ CREATE TABLE IF NOT EXISTS ward_audit ( decision TEXT NOT NULL, approver TEXT, diff_hash BLOB, - files_touched TEXT NOT NULL, + files_touched TEXT NOT NULL, -- JSON array of surface ids channel TEXT, thread_id TEXT, - submitted_at TEXT NOT NULL, - decided_at TEXT NOT NULL, + submitted_at TEXT NOT NULL, -- RFC 3339 + decided_at TEXT NOT NULL, -- RFC 3339 recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); @@ -748,15 +830,48 @@ END; assert_eq!(ward_audit_schema_state(conn), expected); } - fn normalized_table_sql(conn: &Connection) -> String { + fn sql_literal_value(conn: &Connection, literal_sql: &str) -> String { + conn.query_row(&format!("SELECT {literal_sql};"), [], |row| row.get(0)) + .unwrap() + } + + fn stored_table_sql(conn: &Connection) -> String { conn.query_row( - concat!( - ward_audit_schema_state_ctes_sql!(), - r#" -SELECT normalized_table_sql -FROM ward_audit_shape; -"# - ), + "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'ward_audit';", + [], + |row| row.get(0), + ) + .unwrap() + } + + fn explicit_index_sql_fingerprint(conn: &Connection) -> String { + conn.query_row( + r#" + SELECT COALESCE(group_concat(item, '||'), '') + FROM ( + SELECT printf('%s|%s', name, sql) AS item + FROM sqlite_master + WHERE type = 'index' AND tbl_name = 'ward_audit' AND sql IS NOT NULL + ORDER BY name + ); + "#, + [], + |row| row.get(0), + ) + .unwrap() + } + + fn trigger_sql_fingerprint(conn: &Connection) -> String { + conn.query_row( + r#" + SELECT COALESCE(group_concat(item, '||'), '') + FROM ( + SELECT printf('%s|%s', name, sql) AS item + FROM sqlite_master + WHERE type = 'trigger' AND tbl_name = 'ward_audit' + ORDER BY name + ); + "#, [], |row| row.get(0), ) @@ -785,6 +900,21 @@ FROM ward_audit_shape; schema_sql_with_extra_table_constraint(WARD_AUDIT_SCHEMA_SQL, constraint_sql) } + fn drift_event_type_literal( + schema_sql: &str, + exact_literal: &str, + drifted_literal: &str, + ) -> String { + let exact = format!("'{exact_literal}'"); + let drifted = format!("'{drifted_literal}'"); + let drifted_schema_sql = schema_sql.replacen(&exact, &drifted, 1); + assert_ne!( + drifted_schema_sql, schema_sql, + "expected to drift {exact_literal} to {drifted_literal}" + ); + drifted_schema_sql + } + fn expected_explicit_index_names() -> BTreeSet { EXPECTED_EXPLICIT_INDEX_NAMES .iter() @@ -847,14 +977,13 @@ FROM ward_audit_shape; assert_eq!(row.detail.as_deref(), Some(FIXED_PREV_DETAIL)); } - fn assert_legacy_drift_guard_rollback_preserves_state( - constraint_sql: &str, + fn assert_legacy_schema_guard_rollback_preserves_state( + schema_sql: &str, after_rollback: impl FnOnce(&Connection), ) { let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, 37); - conn.execute_batch(&legacy_schema_with_extra_table_constraint(constraint_sql)) - .unwrap(); + conn.execute_batch(schema_sql).unwrap(); let row_id = insert_legacy_ward_updated_row(&conn); let before = load_legacy_audit_row(&conn, row_id); let before_version = user_version(&conn); @@ -880,6 +1009,14 @@ FROM ward_audit_shape; after_rollback(&conn); } + fn assert_legacy_drift_guard_rollback_preserves_state( + constraint_sql: &str, + after_rollback: impl FnOnce(&Connection), + ) { + let schema_sql = legacy_schema_with_extra_table_constraint(constraint_sql); + assert_legacy_schema_guard_rollback_preserves_state(&schema_sql, after_rollback); + } + fn load_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { conn.query_row( r#" @@ -1222,6 +1359,18 @@ FROM ward_audit_shape; let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + assert_eq!( + stored_table_sql(&conn), + sql_literal_value(&conn, ward_audit_exact_legacy_table_sql_sql!()) + ); + assert_eq!( + explicit_index_sql_fingerprint(&conn), + sql_literal_value(&conn, ward_audit_exact_explicit_index_fp_sql!()) + ); + assert_eq!( + trigger_sql_fingerprint(&conn), + sql_literal_value(&conn, ward_audit_exact_trigger_fp_sql!()) + ); } #[test] @@ -1229,10 +1378,22 @@ FROM ward_audit_shape; let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!( + stored_table_sql(&conn), + sql_literal_value(&conn, ward_audit_exact_current_fresh_table_sql_sql!()) + ); + assert_eq!( + explicit_index_sql_fingerprint(&conn), + sql_literal_value(&conn, ward_audit_exact_explicit_index_fp_sql!()) + ); + assert_eq!( + trigger_sql_fingerprint(&conn), + sql_literal_value(&conn, ward_audit_exact_trigger_fp_sql!()) + ); } #[test] - fn fresh_and_migrated_current_schemas_share_the_same_normalized_table_sql() { + fn fresh_and_migrated_current_schemas_use_controlled_exact_sql_variants() { let fresh = Connection::open_in_memory().unwrap(); fresh.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); @@ -1247,9 +1408,62 @@ FROM ward_audit_shape; assert_schema_state(&fresh, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); assert_schema_state(&migrated, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); assert_eq!( - normalized_table_sql(&fresh), - normalized_table_sql(&migrated) + stored_table_sql(&fresh), + sql_literal_value(&fresh, ward_audit_exact_current_fresh_table_sql_sql!()) + ); + assert_eq!( + stored_table_sql(&migrated), + sql_literal_value( + &migrated, + ward_audit_exact_current_migrated_table_sql_sql!() + ) + ); + assert_ne!(stored_table_sql(&fresh), stored_table_sql(&migrated)); + assert_eq!( + explicit_index_sql_fingerprint(&fresh), + sql_literal_value(&fresh, ward_audit_exact_explicit_index_fp_sql!()) + ); + assert_eq!( + explicit_index_sql_fingerprint(&migrated), + sql_literal_value(&migrated, ward_audit_exact_explicit_index_fp_sql!()) + ); + assert_eq!( + trigger_sql_fingerprint(&fresh), + sql_literal_value(&fresh, ward_audit_exact_trigger_fp_sql!()) + ); + assert_eq!( + trigger_sql_fingerprint(&migrated), + sql_literal_value(&migrated, ward_audit_exact_trigger_fp_sql!()) + ); + } + + #[test] + fn current_schema_with_spaced_event_type_literal_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(&drift_event_type_literal( + WARD_AUDIT_SCHEMA_SQL, + "apply_audit", + "apply_ audit", + )) + .unwrap(); + + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + assert!(stored_table_sql(&conn).contains("'apply_ audit'")); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + + #[test] + fn legacy_schema_with_spaced_event_type_literal_is_unknown_and_guard_preserves_state() { + let schema_sql = drift_event_type_literal( + LEGACY_WARD_AUDIT_SCHEMA_SQL, + "compaction_ledger", + "compaction_ ledger", ); + assert_legacy_schema_guard_rollback_preserves_state(&schema_sql, |conn| { + assert!(stored_table_sql(conn).contains("'compaction_ ledger'")); + assert!(!stored_table_sql(conn).contains("'compaction_ledger'")); + }); } #[test] @@ -1386,6 +1600,95 @@ FROM ward_audit_shape; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); } + #[test] + fn current_schema_with_desc_index_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + + conn.execute_batch( + "DROP INDEX ward_audit_event_idx; + CREATE INDEX ward_audit_event_idx ON ward_audit (event_type, recorded_at DESC);", + ) + .unwrap(); + + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert!( + explicit_index_sql_fingerprint(&conn).contains("recorded_at DESC"), + "expected DESC drift in index SQL" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + + #[test] + fn current_schema_with_collated_index_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + + conn.execute_batch( + "DROP INDEX ward_audit_event_idx; + CREATE INDEX ward_audit_event_idx ON ward_audit (event_type COLLATE NOCASE, recorded_at);", + ) + .unwrap(); + + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert!( + explicit_index_sql_fingerprint(&conn).contains("COLLATE NOCASE"), + "expected collation drift in index SQL" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + + #[test] + fn current_schema_with_altered_trigger_error_literal_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + + conn.execute_batch( + "DROP TRIGGER ward_audit_append_only_update; + CREATE TRIGGER ward_audit_append_only_update + BEFORE UPDATE ON ward_audit + BEGIN + SELECT RAISE(ABORT, 'ward_audit is append-only (drifted)'); + END;", + ) + .unwrap(); + + assert_eq!(trigger_names(&conn), expected_trigger_names()); + assert!( + trigger_sql_fingerprint(&conn).contains("drifted"), + "expected trigger-literal drift in trigger SQL" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + + #[test] + fn current_schema_with_altered_trigger_body_is_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + + conn.execute_batch( + "DROP TRIGGER ward_audit_append_only_delete; + CREATE TRIGGER ward_audit_append_only_delete + BEFORE DELETE ON ward_audit + BEGIN + SELECT 1; + SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); + END;", + ) + .unwrap(); + + assert_eq!(trigger_names(&conn), expected_trigger_names()); + assert!( + trigger_sql_fingerprint(&conn).contains("SELECT 1;"), + "expected trigger-body drift in trigger SQL" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + #[test] fn migration_rejects_current_schema_rows_with_detail_and_preserves_state() { let conn = Connection::open_in_memory().unwrap(); diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 8a2c548..47a3be4 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -1,10 +1,11 @@ # ApplyAudit Migration Repair Implementation Plan **Goal:** Replace the old two-boolean `ward_audit` migration gate with a full -schema fingerprint/state contract built around normalized `CREATE TABLE` -equality plus column/index/trigger fingerprints, add an in-migration legacy -guard, preserve evidence and unrelated `user_version`, and prove rollback -behavior with executable rusqlite tests. +schema fingerprint/state contract built around exact stored +`sqlite_master.sql` fingerprints for the table/indexes/triggers plus ordered +column metadata, add an in-migration legacy guard, preserve evidence and +unrelated `user_version`, and prove rollback behavior with executable +rusqlite tests. **Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on @@ -40,6 +41,9 @@ files: - legacy + extra column/data → `unknown`, migration guard failure, explicit `ROLLBACK`, and preservation of row data, extra column, indexes, triggers, and unrelated `user_version`; + - legacy/current `event_type` CHECK drift where a quoted literal gains an + internal space (for example `apply_ audit`) → `unknown`, with the legacy + guard failing before mutation and rollback preserving data; - legacy/current with an extra table-level `CHECK (length(decision) > 0)` → `unknown`, with the legacy guard failing before mutation and rollback preserving the original CHECK behavior; @@ -48,28 +52,32 @@ files: preserving the original UNIQUE behavior; - current missing one append-only trigger → `unknown`, plus an `UPDATE` succeeding to prove why it is unknown; - - current with an extra explicit object (index) → `unknown`. + - current with an extra explicit object (index), `recorded_at DESC`, or + `COLLATE NOCASE` on the explicit index SQL → `unknown`; + - current with altered append-only trigger error literal/body → `unknown`. - Update existing migration failure tests so exact current/rerun cases fail at the new guard and always `ROLLBACK` with `unwrap()`. - Add post-`ALTER` rollback coverage by precreating conflicting `ward_audit_new`, forcing `CREATE TABLE ward_audit_new` to fail after the guard and `ALTER TABLE` succeed, then rolling back and asserting the legacy row/schema are restored. -- Add a normalization-equivalence test proving fresh current and successfully - migrated current both classify `current_v014` under the same normalized table - definition. +- Add exact-stored-SQL tests proving fresh current and successfully migrated + current both classify `current_v014` while matching only their controlled + SQLite-emitted table-SQL variants, and that the shipped legacy fixture + matches `legacy_v013`. ## Task 2: Implement the schema fingerprint/state contract - Add stable state tag constants. - Add public `WARD_AUDIT_SCHEMA_STATE_SQL` that fingerprints: - - full normalized `CREATE TABLE ward_audit (...)` equality, covering all - table-level constraints; + - exact stored `sqlite_master.sql` for `ward_audit`, covering all table-level + constraints; - ordered column metadata (including `recorded_at` default and PK metadata); - - exact explicit index set, excluding SQLite autoindexes; - - exact append-only trigger set; and - - legacy/current event-list differences inside the normalized table - definition, including the presence of `apply_audit`. + - exact explicit index SQL (ordered by name, excluding SQLite autoindexes); + - exact append-only trigger SQL (ordered by name); and + - only the controlled fresh/migrated current table-SQL variants plus the + shipped legacy table-SQL variant, including the presence of `apply_audit` + and any preserved inline comments. - Re-export the state query and tags from `crates/coven-threads-core/src/lib.rs` for root-API callers. - Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: @@ -85,9 +93,9 @@ files: ## Task 3: Update the written contract and validate - Update `audit.rs` docs, the design doc, the plan doc, and `CHANGELOG.md` to - say that full normalized `CREATE TABLE` equality covers every declared - table-level constraint, alongside column/index/trigger fingerprints, instead - of the old partial exact-check wording. + say that exact stored `sqlite_master.sql` equality covers every declared + table-level constraint, alongside column/index/trigger fingerprints, and that + no whitespace-destroying normalization is allowed. - Document the caller contract explicitly: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - initialize on `missing`; @@ -108,7 +116,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): fingerprint full table definition +fix(audit): compare exact stored schema SQL Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index 4bcfe6e..59f7bbd 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -23,16 +23,24 @@ callers run before touching `ward_audit`. It returns one stable tag: The fingerprint is exact and table-local: -- full normalized `CREATE TABLE ward_audit (...)` equality; +- exact stored `sqlite_master.sql` for the `ward_audit` table; - ordered column metadata from `pragma_table_info('ward_audit')`, including name, type, `NOT NULL`, `DEFAULT`, and primary-key metadata; -- exact explicit index set (excluding SQLite internal autoindexes); and -- exact append-only trigger set. - -The full normalized table definition covers every declared table-level -constraint — the `event_type` CHECK list, extra `CHECK` clauses, `UNIQUE` -clauses, and foreign-key clauses — so any extra or missing constraint, column, -index, or trigger classifies as `unknown`. +- exact stored `sqlite_master.sql` for each explicit index (ordered by name and + fingerprinted as `name|sql`, excluding SQLite internal autoindexes); and +- exact stored `sqlite_master.sql` for each append-only trigger (ordered by + name and fingerprinted as `name|sql`). + +The full stored table definition covers every declared table-level constraint — +the `event_type` CHECK list, extra `CHECK` clauses, `UNIQUE` clauses, and +foreign-key clauses — so any extra or missing constraint, column, index, or +trigger classifies as `unknown`. No whitespace-destroying normalization is +applied: only the exact stored SQL variants SQLite emits for the shipped +schemas are accepted. For `current_v014`, that means the fresh +`CREATE TABLE ward_audit (...)` form and the quoted +`CREATE TABLE "ward_audit" (...)` form produced by the legacy rename path. For +`legacy_v013`, the fingerprint intentionally includes the inline comments +preserved from the shipped v0.1.3 DDL. ## Migration design @@ -68,12 +76,17 @@ Executable rusqlite tests cover: 3. legacy/current drift caused by extra table-level `CHECK` or `UNIQUE` constraints classifying as `unknown`, with legacy guard failures requiring explicit `ROLLBACK` and preserving the original constraint behavior; -4. current drift cases such as a missing append-only trigger or an extra index - classifying as `unknown`; -5. exact current/rerun guard failures requiring explicit `ROLLBACK`; -6. successful exact-legacy upgrade landing in `current_v014`, with fresh and - migrated current schemas sharing the same normalized table definition; and -7. post-`ALTER` failure rollback restoring the full legacy table state. +4. current/legacy `event_type` CHECK drift where a quoted literal gains an + internal space (for example `apply_ audit`) classifying as `unknown`, with + the legacy guard rejecting before mutation and rollback preserving data; +5. current drift cases such as a missing append-only trigger, an extra index, a + `recorded_at DESC` index, a `COLLATE NOCASE` index, or altered append-only + trigger SQL classifying as `unknown`; +6. exact current/rerun guard failures requiring explicit `ROLLBACK`; +7. successful exact-legacy upgrade landing in `current_v014`, with fresh and + migrated current schemas both classifying `current_v014` while matching only + their controlled exact stored SQL variants; and +8. post-`ALTER` failure rollback restoring the full legacy table state. ## Scope From f5b17214e549e81b124272581a0d5b35d17326c7 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 00:25:43 -0500 Subject: [PATCH 19/25] fix(audit): make schema initialization fail closed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 29 +- crates/coven-threads-core/src/audit.rs | 361 ++++++++++++++++-- ...2026-07-19-apply-audit-migration-repair.md | 56 ++- ...-19-apply-audit-migration-repair-design.md | 66 +++- 4 files changed, 445 insertions(+), 67 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a95a0ea..c152b8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,9 +17,18 @@ All notable changes to `coven-threads-core` are documented here. `legacy_v013`, `current_v014`, `unknown`) — reusable table-local schema fingerprint contract for daemon callers, using exact stored `sqlite_master.sql` fingerprints for the `ward_audit` table, explicit - indexes, and append-only triggers, plus ordered column metadata. Only the + indexes, and append-only triggers, plus ordered column metadata. `missing` + now means the `ward_audit` table is absent **and** the reserved main-schema + `ward_audit*` namespace is otherwise empty; absent-table collisions on any + `ward_audit*` table/index/trigger/view fail closed as `unknown`. Only the controlled fresh/migrated v0.1.4 table SQL variants and the shipped v0.1.3 table SQL are accepted; no whitespace-destroying normalization is applied. +- `WARD_AUDIT_SCHEMA_SQL` — atomic, self-guarding schema initialization for the + exact `current_v014` fingerprint. It begins a transaction, permits only + `missing` or exact `current_v014` before any mutation, uses idempotent `IF NOT + EXISTS` DDL for daemon compatibility, then requires exact `current_v014` + before `COMMIT`. Exact `legacy_v013` and every `unknown` shape fail closed; + callers must `ROLLBACK` after any init error. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving rebuild of `ward_audit` guarded by the exact `legacy_v013` fingerprint. The daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, @@ -31,11 +40,14 @@ All notable changes to `coven-threads-core` are documented here. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, `schema_state_query_returns_missing_on_empty_db`, + `fresh_schema_sql_initializes_current_schema_atomically_and_enforces_append_only`, `exact_legacy_fixture_returns_legacy_v013`, `exact_current_schema_returns_current_v014`, + `current_schema_sql_reruns_idempotently_and_preserves_rows_and_objects`, `fresh_and_migrated_current_schemas_use_controlled_exact_sql_variants`, `current_schema_with_spaced_event_type_literal_is_unknown`, `legacy_schema_with_spaced_event_type_literal_is_unknown_and_guard_preserves_state`, + `legacy_schema_sql_rejects_and_rollback_preserves_state`, `fresh_schema_preserves_user_version_zero_and_creates_current_shape`, `fresh_schema_preserves_user_version_ninety_nine_and_creates_current_shape`, `legacy_plus_extra_column_and_data_is_unknown_and_guard_preserves_state`, @@ -49,6 +61,9 @@ All notable changes to `coven-threads-core` are documented here. `current_schema_with_collated_index_is_unknown`, `current_schema_with_altered_trigger_error_literal_is_unknown`, `current_schema_with_altered_trigger_body_is_unknown`, + `absent_ward_audit_with_reserved_index_collision_is_unknown_and_schema_sql_preserves_other_objects`, + `absent_ward_audit_with_reserved_trigger_collision_is_unknown_and_schema_sql_preserves_other_objects`, + `unknown_partial_current_schema_rejects_schema_sql_and_preserves_state`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, and `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, and @@ -58,11 +73,13 @@ All notable changes to `coven-threads-core` are documented here. - **Column approach (Option A):** `diff_hash` carries `next_sha256`; `prev_sha256` + `bytes_written` ride in the new `detail` TEXT column as JSON. This avoids a second table rebuild by not adding typed hash columns while keeping the data query-accessible via SQLite JSON functions. - **Migration approach (Option B-lite):** exports - `WARD_AUDIT_SCHEMA_STATE_SQL` and `WARD_AUDIT_MIGRATION_V014_SQL` so the - daemon can perform a quiet, fail-closed upgrade after querying the exact - `ward_audit` schema fingerprint. Production dependencies stay unchanged, - while `coven-threads-core` now carries bundled `rusqlite` as a - dev-dependency for executable migration tests. + `WARD_AUDIT_SCHEMA_STATE_SQL`, `WARD_AUDIT_SCHEMA_SQL`, and + `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can classify `missing` / + `legacy_v013` / `current_v014` / `unknown`, perform a quiet fail-closed init + or upgrade, and recover cleanly with explicit rollback after any init or + migration error. Production dependencies stay unchanged, while + `coven-threads-core` now carries bundled `rusqlite` as a dev-dependency for + executable migration tests. ## [0.1.3] — prior release diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 52ca4c3..6fc18f3 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -20,18 +20,25 @@ //! //! ## Schema shape and migration gating //! -//! `WARD_AUDIT_SCHEMA_SQL` creates the fresh v0.1.4 `ward_audit` shape without -//! mutating database-wide `PRAGMA user_version`. +//! `WARD_AUDIT_SCHEMA_SQL` initializes or verifies the exact v0.1.4 +//! `ward_audit` shape in one transaction, without mutating database-wide +//! `PRAGMA user_version`. It is safe for the daemon to execute unconditionally +//! on every store open: a pre-install guard permits only `missing` or exact +//! `current_v014`, the DDL runs with `IF NOT EXISTS` compatibility, and a +//! post-install guard requires exact `current_v014` before `COMMIT`. If either +//! guard fails, callers must explicitly `ROLLBACK` before continuing. //! `WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable, table-local fingerprint query //! that returns one of four stable tags: -//! - `missing` — `ward_audit` does not exist; initialize with +//! - `missing` — `ward_audit` does not exist **and** no reserved main-schema +//! `ward_audit*` table/index/trigger/view names already exist; initialize with //! `WARD_AUDIT_SCHEMA_SQL`; //! - `legacy_v013` — the table exactly matches the v0.1.3 legacy fingerprint; //! run `WARD_AUDIT_MIGRATION_V014_SQL`; //! - `current_v014` — the table exactly matches the v0.1.4 current //! fingerprint; continue without schema work; //! - `unknown` — every other shape, including any extra or missing declared -//! table constraint, column, index, or trigger; fail closed. +//! table constraint, column, index, or trigger, plus absent-table collisions +//! anywhere else in the reserved `ward_audit*` namespace; fail closed. //! The fingerprint uses exact `sqlite_master.sql` text for the table, explicit //! indexes, and append-only triggers, plus ordered `pragma_table_info` //! metadata. It does **not** normalize whitespace: the only accepted @@ -318,7 +325,8 @@ fn reject_tag(reason: &RejectReason) -> &'static str { } /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] when `ward_audit` is -/// absent. +/// absent and the reserved main-schema `ward_audit*` namespace is otherwise +/// empty. pub const WARD_AUDIT_SCHEMA_STATE_MISSING: &str = "missing"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact /// v0.1.3 legacy schema fingerprint. @@ -327,7 +335,8 @@ pub const WARD_AUDIT_SCHEMA_STATE_LEGACY_V013: &str = "legacy_v013"; /// v0.1.4 current schema fingerprint. pub const WARD_AUDIT_SCHEMA_STATE_CURRENT_V014: &str = "current_v014"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for every other -/// `ward_audit` shape. +/// `ward_audit` shape, plus absent-table reserved-name collisions elsewhere in +/// the main-schema `ward_audit*` namespace. pub const WARD_AUDIT_SCHEMA_STATE_UNKNOWN: &str = "unknown"; macro_rules! ward_audit_schema_state_ctes_sql { @@ -382,13 +391,21 @@ WITH ORDER BY name ) ), + ward_audit_reserved_namespace_object_count AS ( + SELECT COUNT(*) AS count + FROM sqlite_master + WHERE type IN ('table', 'index', 'trigger', 'view') + AND lower(name) GLOB 'ward_audit*' + AND NOT (type = 'table' AND name = 'ward_audit') + ), ward_audit_shape AS ( SELECT (SELECT ok FROM ward_audit_exists) AS table_exists, COALESCE((SELECT sql FROM ward_audit_table), '') AS table_sql, (SELECT fp FROM ward_audit_column_fingerprint) AS column_fp, (SELECT fp FROM ward_audit_index_fingerprint) AS index_fp, - (SELECT fp FROM ward_audit_trigger_fingerprint) AS trigger_fp + (SELECT fp FROM ward_audit_trigger_fingerprint) AS trigger_fp, + (SELECT count FROM ward_audit_reserved_namespace_object_count) AS reserved_namespace_count ) "# }; @@ -536,11 +553,30 @@ AND trigger_fp = "#, }; } +macro_rules! ward_audit_schema_state_case_sql { + () => { + concat!( + r#"CASE + WHEN table_exists = 0 AND reserved_namespace_count = 0 THEN 'missing' + WHEN "#, + ward_audit_exact_legacy_predicate_sql!(), + r#" THEN 'legacy_v013' + WHEN "#, + ward_audit_exact_current_predicate_sql!(), + r#" THEN 'current_v014' + ELSE 'unknown' +END"# + ) + }; +} + /// Table-local schema-state query for `ward.audit` inside `coven.sqlite3` /// (§3.4). /// /// Callers run this exact query and branch on the stable text result: -/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — initialize with +/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — `ward_audit` is absent and the +/// reserved main-schema `ward_audit*` namespace is otherwise empty; initialize +/// with /// [`WARD_AUDIT_SCHEMA_SQL`]; /// - [`WARD_AUDIT_SCHEMA_STATE_LEGACY_V013`] — run /// [`WARD_AUDIT_MIGRATION_V014_SQL`]; @@ -554,25 +590,22 @@ AND trigger_fp = "#, /// append-only trigger SQL set must all match. Full stored-table-SQL equality /// covers every declared table-level constraint (`CHECK`, `UNIQUE`, foreign-key /// clauses, and the `event_type` list), so any extra or missing column, -/// constraint, index, or trigger returns `unknown`. No whitespace-destroying -/// normalization is applied: the only accepted `current_v014` table SQL -/// variants are the fresh `CREATE TABLE ward_audit (...)` form and SQLite's -/// quoted `CREATE TABLE "ward_audit" (...)` form produced by the exact legacy -/// migration path, while the `legacy_v013` fingerprint intentionally includes -/// the inline comments preserved from the shipped v0.1.3 DDL. +/// constraint, index, or trigger returns `unknown`. When `ward_audit` itself is +/// absent, any preexisting main-schema table/index/trigger/view whose name is +/// reserved under the `ward_audit*` namespace also returns `unknown`, so +/// initialization cannot silently skip append-only objects on another table. No +/// whitespace-destroying normalization is applied: the only accepted +/// `current_v014` table SQL variants are the fresh `CREATE TABLE ward_audit +/// (...)` form and SQLite's quoted `CREATE TABLE "ward_audit" (...)` form +/// produced by the exact legacy migration path, while the `legacy_v013` +/// fingerprint intentionally includes the inline comments preserved from the +/// shipped v0.1.3 DDL. pub const WARD_AUDIT_SCHEMA_STATE_SQL: &str = concat!( ward_audit_schema_state_ctes_sql!(), r#" -SELECT CASE - WHEN table_exists = 0 THEN 'missing' - WHEN "#, - ward_audit_exact_legacy_predicate_sql!(), - r#" THEN 'legacy_v013' - WHEN "#, - ward_audit_exact_current_predicate_sql!(), - r#" THEN 'current_v014' - ELSE 'unknown' -END AS schema_state +SELECT "#, + ward_audit_schema_state_case_sql!(), + r#" AS schema_state FROM ward_audit_shape; "#, ); @@ -682,12 +715,11 @@ COMMIT; "#, ); -/// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). -/// -/// See module docs for the full schema-state contract. This DDL creates the -/// exact `current_v014` fingerprint without mutating database-wide -/// `PRAGMA user_version`. -pub const WARD_AUDIT_SCHEMA_SQL: &str = r#" +// Shared raw current-v0.1.4 DDL body used by the guarded init SQL and drift +// tests. +macro_rules! ward_audit_current_objects_sql { + () => { + r#" CREATE TABLE IF NOT EXISTS ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_type TEXT NOT NULL CHECK (event_type IN ( @@ -725,7 +757,65 @@ BEFORE DELETE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; -"#; +"# + }; +} + +/// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). +/// +/// See module docs for the full schema-state contract. This transaction is +/// safe to run unconditionally on every store open: it permits only exact +/// `missing` or `current_v014` before any mutation, uses idempotent `IF NOT +/// EXISTS` DDL for daemon compatibility, then requires exact `current_v014` +/// before `COMMIT`. Exact `legacy_v013` and every `unknown` shape fail closed. +/// If this SQL errors, callers must explicitly `ROLLBACK` before continuing so +/// SQLite discards any uncommitted work. This DDL never mutates database-wide +/// `PRAGMA user_version`. +pub const WARD_AUDIT_SCHEMA_SQL: &str = concat!( + r#" +BEGIN; + +CREATE TEMP TABLE ward_audit_schema_pre_guard ( + ok INTEGER NOT NULL CHECK (ok = 1) +); + +INSERT INTO ward_audit_schema_pre_guard (ok) +"#, + ward_audit_schema_state_ctes_sql!(), + r#" +SELECT CASE + WHEN ("#, + ward_audit_schema_state_case_sql!(), + r#") IN ('missing', 'current_v014') THEN 1 + ELSE 0 +END +FROM ward_audit_shape; + +DROP TABLE ward_audit_schema_pre_guard; +"#, + ward_audit_current_objects_sql!(), + r#" +CREATE TEMP TABLE ward_audit_schema_post_guard ( + ok INTEGER NOT NULL CHECK (ok = 1) +); + +INSERT INTO ward_audit_schema_post_guard (ok) +"#, + ward_audit_schema_state_ctes_sql!(), + r#" +SELECT CASE + WHEN ("#, + ward_audit_schema_state_case_sql!(), + r#") = 'current_v014' THEN 1 + ELSE 0 +END +FROM ward_audit_shape; + +DROP TABLE ward_audit_schema_post_guard; + +COMMIT; +"#, +); #[cfg(test)] mod tests { @@ -897,7 +987,7 @@ END; } fn current_schema_with_extra_table_constraint(constraint_sql: &str) -> String { - schema_sql_with_extra_table_constraint(WARD_AUDIT_SCHEMA_SQL, constraint_sql) + schema_sql_with_extra_table_constraint(ward_audit_current_objects_sql!(), constraint_sql) } fn drift_event_type_literal( @@ -963,6 +1053,16 @@ END; == 1 } + fn main_schema_object_exists(conn: &Connection, object_type: &str, name: &str) -> bool { + conn.query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type = ?1 AND name = ?2);", + params![object_type, name], + |row| row.get::<_, i64>(0), + ) + .unwrap() + == 1 + } + fn assert_fresh_schema_preserves_user_version(initial_version: i64) { let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, initial_version); @@ -1354,6 +1454,34 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_MISSING); } + #[test] + fn fresh_schema_sql_initializes_current_schema_atomically_and_enforces_append_only() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 11); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_MISSING); + + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!(user_version(&conn), 11); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + + let row_id = insert_current_apply_audit_row(&conn); + let update_err = conn + .execute( + "UPDATE ward_audit SET decision = 'changed' WHERE id = ?1;", + params![row_id], + ) + .unwrap_err(); + assert!(update_err.to_string().contains("append-only")); + + let delete_err = conn + .execute("DELETE FROM ward_audit WHERE id = ?1;", params![row_id]) + .unwrap_err(); + assert!(delete_err.to_string().contains("append-only")); + } + #[test] fn exact_legacy_fixture_returns_legacy_v013() { let conn = Connection::open_in_memory().unwrap(); @@ -1392,6 +1520,27 @@ END; ); } + #[test] + fn current_schema_sql_reruns_idempotently_and_preserves_rows_and_objects() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 29); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_current_apply_audit_row(&conn); + let before = load_audit_row(&conn, row_id); + let before_table_sql = stored_table_sql(&conn); + let before_indexes = explicit_index_names(&conn); + let before_triggers = trigger_names(&conn); + + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + + assert_eq!(load_audit_row(&conn, row_id), before); + assert_eq!(stored_table_sql(&conn), before_table_sql); + assert_eq!(explicit_index_names(&conn), before_indexes); + assert_eq!(trigger_names(&conn), before_triggers); + assert_eq!(user_version(&conn), 29); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + } + #[test] fn fresh_and_migrated_current_schemas_use_controlled_exact_sql_variants() { let fresh = Connection::open_in_memory().unwrap(); @@ -1441,7 +1590,7 @@ END; fn current_schema_with_spaced_event_type_literal_is_unknown() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(&drift_event_type_literal( - WARD_AUDIT_SCHEMA_SQL, + ward_audit_current_objects_sql!(), "apply_audit", "apply_ audit", )) @@ -1466,6 +1615,34 @@ END; }); } + #[test] + fn legacy_schema_sql_rejects_and_rollback_preserves_state() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_legacy_ward_updated_row(&conn); + let before = load_legacy_audit_row(&conn, row_id); + let before_version = user_version(&conn); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("exact legacy schema must fail closed at the schema-init guard"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_legacy_audit_row(&conn, row_id), before); + assert_eq!(user_version(&conn), before_version); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + assert!(!has_column(&conn, "detail")); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + } + #[test] fn fresh_schema_preserves_user_version_zero_and_creates_current_shape() { assert_fresh_schema_preserves_user_version(0); @@ -1689,6 +1866,124 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); } + #[test] + fn absent_ward_audit_with_reserved_index_collision_is_unknown_and_schema_sql_preserves_other_objects( + ) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 23); + conn.execute_batch( + "CREATE TABLE other (id INTEGER PRIMARY KEY, note TEXT NOT NULL); + INSERT INTO other (id, note) VALUES (1, 'sentinel'); + CREATE INDEX ward_audit_event_idx ON other (note);", + ) + .unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("reserved index collision must fail closed before creating ward_audit"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit")); + assert!(main_schema_object_exists(&conn, "table", "other")); + assert!(main_schema_object_exists( + &conn, + "index", + "ward_audit_event_idx" + )); + assert_eq!( + conn.query_row("SELECT note FROM other WHERE id = 1;", [], |row| row + .get::<_, String>(0)) + .unwrap(), + "sentinel" + ); + assert_eq!(user_version(&conn), 23); + } + + #[test] + fn absent_ward_audit_with_reserved_trigger_collision_is_unknown_and_schema_sql_preserves_other_objects( + ) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 31); + conn.execute_batch( + "CREATE TABLE other (id INTEGER PRIMARY KEY, note TEXT NOT NULL); + INSERT INTO other (id, note) VALUES (1, 'sentinel'); + CREATE TRIGGER ward_audit_append_only_update + BEFORE UPDATE ON other + BEGIN + SELECT RAISE(ABORT, 'other is append-only'); + END;", + ) + .unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("reserved trigger collision must fail closed before creating ward_audit"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit")); + assert!(main_schema_object_exists(&conn, "table", "other")); + assert!(main_schema_object_exists( + &conn, + "trigger", + "ward_audit_append_only_update" + )); + let trigger_err = conn + .execute("UPDATE other SET note = 'changed' WHERE id = 1;", []) + .unwrap_err(); + assert!(trigger_err.to_string().contains("other is append-only")); + assert_eq!( + conn.query_row("SELECT note FROM other WHERE id = 1;", [], |row| row + .get::<_, String>(0)) + .unwrap(), + "sentinel" + ); + assert_eq!(user_version(&conn), 31); + } + + #[test] + fn unknown_partial_current_schema_rejects_schema_sql_and_preserves_state() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_current_apply_audit_row(&conn); + let before = load_audit_row(&conn, row_id); + + conn.execute_batch("DROP TRIGGER ward_audit_append_only_update;") + .unwrap(); + let before_triggers = trigger_names(&conn); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("partial current schema must fail closed at the schema-init guard"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_audit_row(&conn, row_id), before); + assert_eq!(trigger_names(&conn), before_triggers); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(user_version(&conn), 37); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + #[test] fn migration_rejects_current_schema_rows_with_detail_and_preserves_state() { let conn = Connection::open_in_memory().unwrap(); diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 47a3be4..0111b32 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -3,16 +3,18 @@ **Goal:** Replace the old two-boolean `ward_audit` migration gate with a full schema fingerprint/state contract built around exact stored `sqlite_master.sql` fingerprints for the table/indexes/triggers plus ordered -column metadata, add an in-migration legacy guard, preserve evidence and -unrelated `user_version`, and prove rollback behavior with executable -rusqlite tests. +column metadata, reserve the `ward_audit*` namespace on fresh installs, make +schema initialization fail closed and atomic, preserve evidence and unrelated +`user_version`, and prove rollback behavior with executable rusqlite tests. **Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on -`missing` / `legacy_v013` / `current_v014` / `unknown`, and embed the same -exact `legacy_v013` predicate inside `WARD_AUDIT_MIGRATION_V014_SQL` before any -`ALTER TABLE`. The migration remains table-local and never writes -database-wide `user_version`. +`missing` / `legacy_v013` / `current_v014` / `unknown`, reserve `missing` for +an absent table with no preexisting main-schema `ward_audit*` objects, wrap +`WARD_AUDIT_SCHEMA_SQL` in atomic pre/post guards that allow only +`missing`/`current_v014`, and embed the same exact `legacy_v013` predicate +inside `WARD_AUDIT_MIGRATION_V014_SQL` before any `ALTER TABLE`. The init and +migration paths remain table-local and never write database-wide `user_version`. **Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` for in-memory tests. @@ -36,7 +38,18 @@ files: - Add schema-state tests for: - empty DB → `missing`; - exact legacy fixture → `legacy_v013`; - - exact current schema → `current_v014`. + - exact current schema → `current_v014`; + - absent-table reserved-name collisions (`ward_audit_event_idx`, + `ward_audit_append_only_update`) → `unknown`. +- Add init-safety tests for: + - clean empty DB → `WARD_AUDIT_SCHEMA_SQL` succeeds atomically, lands on + `current_v014`, and append-only UPDATE/DELETE still abort; + - exact current schema → rerunning `WARD_AUDIT_SCHEMA_SQL` is idempotent and + preserves rows/objects; + - exact legacy schema → `WARD_AUDIT_SCHEMA_SQL` rejects, requires explicit + `ROLLBACK`, and preserves state/data; + - unknown partial current schema → `WARD_AUDIT_SCHEMA_SQL` rejects and + preserves the drifted state. - Add drift tests for: - legacy + extra column/data → `unknown`, migration guard failure, explicit `ROLLBACK`, and preservation of row data, extra column, indexes, triggers, @@ -74,12 +87,26 @@ files: constraints; - ordered column metadata (including `recorded_at` default and PK metadata); - exact explicit index SQL (ordered by name, excluding SQLite autoindexes); - - exact append-only trigger SQL (ordered by name); and + - exact append-only trigger SQL (ordered by name); + - when `ward_audit` is absent, an empty reserved main-schema + `ward_audit*` namespace across tables/indexes/triggers/views; and - only the controlled fresh/migrated current table-SQL variants plus the shipped legacy table-SQL variant, including the presence of `apply_audit` and any preserved inline comments. - Re-export the state query and tags from `crates/coven-threads-core/src/lib.rs` for root-API callers. +- Update `WARD_AUDIT_SCHEMA_SQL` so it: + - starts a transaction; + - creates a TEMP pre-install guard with `CHECK (ok = 1)`; + - inserts `1` only when the shared schema-state expression returns + `missing` or `current_v014`, otherwise inserts `0` and aborts before any + mutation; + - keeps `CREATE TABLE/INDEX/TRIGGER IF NOT EXISTS` for current daemon + compatibility; + - creates a TEMP post-install guard that requires exact `current_v014`, + aborting on any silent no-op/collision or malformed result; + - commits on success; and + - requires callers to `ROLLBACK` after any init error. - Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: - starts a transaction; - creates a TEMP guard table with `CHECK (ok = 1)`; @@ -94,16 +121,17 @@ files: - Update `audit.rs` docs, the design doc, the plan doc, and `CHANGELOG.md` to say that exact stored `sqlite_master.sql` equality covers every declared - table-level constraint, alongside column/index/trigger fingerprints, and that + table-level constraint, alongside column/index/trigger fingerprints, that the + main-schema `ward_audit*` namespace is reserved on fresh installs, and that no whitespace-destroying normalization is allowed. - Document the caller contract explicitly: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - - initialize on `missing`; + - initialize on `missing` only; - migrate only `legacy_v013`; - continue on `current_v014`; - fail closed on `unknown`; - - rely on the migration’s independent legacy guard as a second line of - defense. + - `ROLLBACK` after any init or migration error; and + - rely on the init/migration independent guards as second lines of defense. - Run: - `cargo fmt` - focused audit tests @@ -116,7 +144,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): compare exact stored schema SQL +fix(audit): make schema initialization fail closed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index 59f7bbd..1a7d7c6 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -13,7 +13,8 @@ schema contract. `crates/coven-threads-core::WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable query callers run before touching `ward_audit`. It returns one stable tag: -- `missing` — `ward_audit` does not exist; initialize with +- `missing` — `ward_audit` does not exist and no reserved main-schema + `ward_audit*` table/index/trigger/view names already exist; initialize with `WARD_AUDIT_SCHEMA_SQL`. - `legacy_v013` — exact v0.1.3 legacy fingerprint; run `WARD_AUDIT_MIGRATION_V014_SQL`. @@ -34,14 +35,39 @@ The fingerprint is exact and table-local: The full stored table definition covers every declared table-level constraint — the `event_type` CHECK list, extra `CHECK` clauses, `UNIQUE` clauses, and foreign-key clauses — so any extra or missing constraint, column, index, or -trigger classifies as `unknown`. No whitespace-destroying normalization is -applied: only the exact stored SQL variants SQLite emits for the shipped -schemas are accepted. For `current_v014`, that means the fresh +trigger classifies as `unknown`. When `ward_audit` itself is absent, any +preexisting main-schema object whose name is reserved under `ward_audit*` also +classifies as `unknown`; that prevents schema init from silently skipping +append-only objects already attached to other tables. No whitespace-destroying +normalization is applied: only the exact stored SQL variants SQLite emits for +the shipped schemas are accepted. For `current_v014`, that means the fresh `CREATE TABLE ward_audit (...)` form and the quoted `CREATE TABLE "ward_audit" (...)` form produced by the legacy rename path. For `legacy_v013`, the fingerprint intentionally includes the inline comments preserved from the shipped v0.1.3 DDL. +## Initialization design + +`WARD_AUDIT_SCHEMA_SQL` must stay compatible with the current daemon behavior: +the daemon executes it unconditionally on every store open. The safe contract is +therefore: + +1. `BEGIN;` +2. run a TEMP pre-install guard that reuses the exact schema-state CTEs and + permits only `missing` or exact `current_v014`; +3. create the `ward_audit` table, explicit indexes, and append-only triggers + with `IF NOT EXISTS`; +4. run a TEMP post-install guard that reuses the same schema-state expression + and requires exact `current_v014`; and +5. `COMMIT;` + +This makes fresh installs atomic, makes exact current reruns idempotent, and +fails closed for `legacy_v013`, `unknown`, or any malformed/silently skipped +install result. If either guard errors, callers must explicitly `ROLLBACK` +before continuing so SQLite removes any uncommitted `ward_audit` table created +on the failed path while preserving unrelated objects that caused the +collision. + ## Migration design The v0.1.4 repair remains a table-local migration. The SQL still: @@ -56,7 +82,8 @@ The change is the guardrail: immediately after `BEGIN;`, the migration creates a TEMP guard table with `CHECK (ok = 1)` and inserts `1` only if the exact `legacy_v013` fingerprint holds. Any missing/current/partial schema inserts `0` instead, aborting before `ALTER TABLE`. This makes the migration fail closed -even if a caller skips the classification query. +even if a caller skips the classification query, and keeps initialization and +migration aligned on the same exact state contract. If a later step fails after `ALTER TABLE` (for example, `ward_audit_new` already exists), callers must explicitly `ROLLBACK` the failed transaction @@ -69,28 +96,39 @@ version state is preserved. Executable rusqlite tests cover: 1. schema-state classification for `missing`, exact `legacy_v013`, and exact - `current_v014`; -2. partial legacy drift (`legacy` + extra column/data) classifying as + `current_v014`, including absent-table reserved-namespace collisions + classifying as `unknown`; +2. fresh empty-store initialization classifying `missing`, running + `WARD_AUDIT_SCHEMA_SQL` atomically to `current_v014`, and enforcing + append-only UPDATE/DELETE rejection; +3. exact current reruns of `WARD_AUDIT_SCHEMA_SQL` staying idempotent while + preserving rows, indexes, triggers, and exact stored SQL fingerprints; +4. exact legacy/current/unknown init-guard failures requiring explicit + `ROLLBACK` and preserving preexisting data/object state; +5. reserved-name collision repros (`ward_audit_event_idx` and + `ward_audit_append_only_update` on another table) failing closed before any + `ward_audit` table is created; and +6. partial legacy drift (`legacy` + extra column/data) classifying as `unknown`, rejecting at the guard, and rolling back cleanly without losing row data, extra columns, indexes, triggers, or unrelated `user_version`; -3. legacy/current drift caused by extra table-level `CHECK` or `UNIQUE` +7. legacy/current drift caused by extra table-level `CHECK` or `UNIQUE` constraints classifying as `unknown`, with legacy guard failures requiring explicit `ROLLBACK` and preserving the original constraint behavior; -4. current/legacy `event_type` CHECK drift where a quoted literal gains an +8. current/legacy `event_type` CHECK drift where a quoted literal gains an internal space (for example `apply_ audit`) classifying as `unknown`, with the legacy guard rejecting before mutation and rollback preserving data; -5. current drift cases such as a missing append-only trigger, an extra index, a +9. current drift cases such as a missing append-only trigger, an extra index, a `recorded_at DESC` index, a `COLLATE NOCASE` index, or altered append-only trigger SQL classifying as `unknown`; -6. exact current/rerun guard failures requiring explicit `ROLLBACK`; -7. successful exact-legacy upgrade landing in `current_v014`, with fresh and +10. successful exact-legacy upgrade landing in `current_v014`, with fresh and migrated current schemas both classifying `current_v014` while matching only their controlled exact stored SQL variants; and -8. post-`ALTER` failure rollback restoring the full legacy table state. +11. post-`ALTER` failure rollback restoring the full legacy table state. ## Scope This repair does not add daemon migration orchestration or change the `WardAuditRecord` wire shape. The daemon remains responsible for calling `WARD_AUDIT_SCHEMA_STATE_SQL`, choosing fresh initialization vs legacy -migration, and failing closed on `unknown`. +migration, running `WARD_AUDIT_SCHEMA_SQL` only through the allowed +`missing`/`current_v014` contract, and failing closed on `unknown`. From 35527b0ab055a6e9e4baa6f09f5af7b626c0cb18 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 00:56:06 -0500 Subject: [PATCH 20/25] fix(audit): reject temporary schema shadows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 50 +- crates/coven-threads-core/src/audit.rs | 754 ++++++++++++++---- ...2026-07-19-apply-audit-migration-repair.md | 87 +- ...-19-apply-audit-migration-repair-design.md | 88 +- 4 files changed, 756 insertions(+), 223 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c152b8d..40fd569 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,26 +16,32 @@ All notable changes to `coven-threads-core` are documented here. - `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable state tags (`missing`, `legacy_v013`, `current_v014`, `unknown`) — reusable table-local schema fingerprint contract for daemon callers, using exact stored - `sqlite_master.sql` fingerprints for the `ward_audit` table, explicit - indexes, and append-only triggers, plus ordered column metadata. `missing` - now means the `ward_audit` table is absent **and** the reserved main-schema - `ward_audit*` namespace is otherwise empty; absent-table collisions on any - `ward_audit*` table/index/trigger/view fail closed as `unknown`. Only the + `main.sqlite_master.sql` fingerprints for the durable `main.ward_audit` + table, explicit main indexes, and append-only main triggers, plus ordered + durable column metadata discovered through schema-qualified PRAGMAs. + `missing` now means `main.ward_audit` is absent, the reserved main-schema + `ward_audit` / `ward_audit_*` namespace is otherwise empty, **and** no temp + shadow/reserved temp object exists; any temp-schema table/view/index/trigger + named `ward_audit` or `ward_audit_*` fail-closes as `unknown`. Only the controlled fresh/migrated v0.1.4 table SQL variants and the shipped v0.1.3 table SQL are accepted; no whitespace-destroying normalization is applied. - `WARD_AUDIT_SCHEMA_SQL` — atomic, self-guarding schema initialization for the exact `current_v014` fingerprint. It begins a transaction, permits only `missing` or exact `current_v014` before any mutation, uses idempotent `IF NOT - EXISTS` DDL for daemon compatibility, then requires exact `current_v014` - before `COMMIT`. Exact `legacy_v013` and every `unknown` shape fail closed; - callers must `ROLLBACK` after any init error. + EXISTS` DDL for daemon compatibility, targets durable schema objects in + `main` wherever SQLite syntax permits, then requires exact `current_v014` + before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, and any + temp shadow/reserved temp object fail closed; callers must `ROLLBACK` after + any init error. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving - rebuild of `ward_audit` guarded by the exact `legacy_v013` fingerprint. The - daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, - migrate only `legacy_v013`, continue on `current_v014`, and fail closed on - `unknown`. The migration independently re-checks `legacy_v013` before any - `ALTER`, using the same exact stored-table + column/index/trigger predicate, - then copies rows into the replacement table without discarding evidence. + rebuild of `main.ward_audit` guarded by the exact `legacy_v013` fingerprint + plus temp-shadow rejection. The daemon should query + `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, migrate only + `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The + migration independently re-checks `legacy_v013` before any `ALTER`, using the + same exact stored-table + column/index/trigger predicate plus temp-shadow + rejection, then copies rows into the replacement table without discarding + evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, @@ -67,7 +73,13 @@ All notable changes to `coven-threads-core` are documented here. `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, and `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, and - `post_alter_failure_rollback_restores_legacy_schema_after_create_conflict`. + `post_alter_failure_rollback_restores_legacy_schema_after_create_conflict`, and + `schema_qualified_table_valued_pragmas_resolve_main_and_temp_separately`, + `current_main_with_temp_shadow_is_unknown_and_guards_preserve_main_and_temp_rows`, + `missing_main_with_temp_ward_audit_shadow_is_unknown_and_schema_sql_rejects_without_creating_main`, + `missing_main_with_reserved_temp_objects_is_unknown_and_schema_sql_preserves_temp_objects`, + `legacy_main_with_temp_shadow_is_unknown_and_migration_rejects_before_mutating_either_schema`, and + `unqualified_insert_targets_temp_shadow_while_schema_state_stays_unknown`. ### Design notes @@ -77,9 +89,11 @@ All notable changes to `coven-threads-core` are documented here. `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can classify `missing` / `legacy_v013` / `current_v014` / `unknown`, perform a quiet fail-closed init or upgrade, and recover cleanly with explicit rollback after any init or - migration error. Production dependencies stay unchanged, while - `coven-threads-core` now carries bundled `rusqlite` as a dev-dependency for - executable migration tests. + migration error. The durable contract is explicitly `main.ward_audit`, and + any TEMP shadow/reserved temp object keeps the contract fail-closed as + `unknown`. Production dependencies stay unchanged, while `coven-threads-core` + now carries bundled `rusqlite` as a dev-dependency for executable migration + tests. ## [0.1.3] — prior release diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 6fc18f3..e0432a4 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -21,38 +21,44 @@ //! ## Schema shape and migration gating //! //! `WARD_AUDIT_SCHEMA_SQL` initializes or verifies the exact v0.1.4 -//! `ward_audit` shape in one transaction, without mutating database-wide +//! `main.ward_audit` shape in one transaction, without mutating database-wide //! `PRAGMA user_version`. It is safe for the daemon to execute unconditionally //! on every store open: a pre-install guard permits only `missing` or exact -//! `current_v014`, the DDL runs with `IF NOT EXISTS` compatibility, and a -//! post-install guard requires exact `current_v014` before `COMMIT`. If either -//! guard fails, callers must explicitly `ROLLBACK` before continuing. +//! `current_v014`, the durable DDL runs explicitly in `main` with `IF NOT +//! EXISTS` compatibility, and a post-install guard requires exact +//! `current_v014` before `COMMIT`. If either guard fails, callers must +//! explicitly `ROLLBACK` before continuing. //! `WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable, table-local fingerprint query //! that returns one of four stable tags: -//! - `missing` — `ward_audit` does not exist **and** no reserved main-schema -//! `ward_audit*` table/index/trigger/view names already exist; initialize with +//! - `missing` — `main.ward_audit` does not exist, the reserved durable +//! `ward_audit` / `ward_audit_*` namespace is otherwise empty in `main`, and +//! no temp-schema shadow/reserved object exists; initialize with //! `WARD_AUDIT_SCHEMA_SQL`; -//! - `legacy_v013` — the table exactly matches the v0.1.3 legacy fingerprint; -//! run `WARD_AUDIT_MIGRATION_V014_SQL`; -//! - `current_v014` — the table exactly matches the v0.1.4 current -//! fingerprint; continue without schema work; +//! - `legacy_v013` — `main.ward_audit` exactly matches the v0.1.3 legacy +//! fingerprint and no temp shadow exists; run `WARD_AUDIT_MIGRATION_V014_SQL`; +//! - `current_v014` — `main.ward_audit` exactly matches the v0.1.4 current +//! fingerprint and no temp shadow exists; continue without schema work; //! - `unknown` — every other shape, including any extra or missing declared -//! table constraint, column, index, or trigger, plus absent-table collisions -//! anywhere else in the reserved `ward_audit*` namespace; fail closed. -//! The fingerprint uses exact `sqlite_master.sql` text for the table, explicit -//! indexes, and append-only triggers, plus ordered `pragma_table_info` -//! metadata. It does **not** normalize whitespace: the only accepted -//! `current_v014` table-SQL variants are the fresh `CREATE TABLE ward_audit` -//! form and the quoted `CREATE TABLE "ward_audit"` form SQLite stores after -//! the exact legacy migration path, and the `legacy_v013` fingerprint includes -//! the inline comments preserved from the shipped v0.1.3 DDL. +//! table constraint, column, index, or trigger, any reserved durable-name +//! collision, and any temp-schema table/view/index/trigger named +//! `ward_audit` or `ward_audit_*`; fail closed. +//! The fingerprint uses exact `main.sqlite_master.sql` text for the durable +//! table, explicit durable indexes, and append-only durable triggers, plus +//! ordered `pragma_table_info('ward_audit', 'main')` metadata and +//! `pragma_index_list('ward_audit', 'main')` for explicit index discovery. It +//! does **not** normalize whitespace: the only accepted `current_v014` +//! table-SQL variants are the fresh `CREATE TABLE ward_audit` form and the +//! quoted `CREATE TABLE "ward_audit"` form SQLite stores after the exact legacy +//! migration path, and the `legacy_v013` fingerprint includes the inline +//! comments preserved from the shipped v0.1.3 DDL. //! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` -//! fingerprint. It independently re-checks that fingerprint inside the -//! transaction before any `ALTER`, then adds `detail` and rebuilds `ward_audit` -//! so the current CHECK set is installed and every existing row is preserved. -//! If a later migration step fails after `ALTER TABLE ... ADD COLUMN detail`, -//! callers must explicitly roll back the failed transaction before continuing -//! so SQLite restores the untouched legacy table. +//! fingerprint with no temp shadow. It independently re-checks that fingerprint +//! inside the transaction before any `ALTER`, then adds `detail` and rebuilds +//! `main.ward_audit` so the current CHECK set is installed and every existing +//! row is preserved. If a later migration step fails after +//! `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must explicitly roll +//! back the failed transaction before continuing so SQLite restores the +//! untouched legacy table. //! //! ## Where content hashes ride for applied writes //! @@ -324,28 +330,36 @@ fn reject_tag(reason: &RejectReason) -> &'static str { } } -/// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] when `ward_audit` is -/// absent and the reserved main-schema `ward_audit*` namespace is otherwise -/// empty. +/// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] when `main.ward_audit` +/// is absent, the reserved main-schema `ward_audit` / `ward_audit_*` namespace +/// is otherwise empty, and no temp shadow/reserved object blocks the durable +/// contract. pub const WARD_AUDIT_SCHEMA_STATE_MISSING: &str = "missing"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact -/// v0.1.3 legacy schema fingerprint. +/// `main.ward_audit` v0.1.3 legacy schema fingerprint, with no temp shadow. pub const WARD_AUDIT_SCHEMA_STATE_LEGACY_V013: &str = "legacy_v013"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact -/// v0.1.4 current schema fingerprint. +/// `main.ward_audit` v0.1.4 current schema fingerprint, with no temp shadow. pub const WARD_AUDIT_SCHEMA_STATE_CURRENT_V014: &str = "current_v014"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for every other -/// `ward_audit` shape, plus absent-table reserved-name collisions elsewhere in -/// the main-schema `ward_audit*` namespace. +/// `main.ward_audit` shape, plus reserved-name collisions in the durable +/// namespace or any temp-schema `ward_audit` / `ward_audit_*` shadow object. pub const WARD_AUDIT_SCHEMA_STATE_UNKNOWN: &str = "unknown"; +macro_rules! ward_audit_reserved_name_predicate_sql { + () => { + r#"(lower(name) = 'ward_audit' OR lower(name) GLOB 'ward_audit_*')"# + }; +} + macro_rules! ward_audit_schema_state_ctes_sql { () => { - r#" + concat!( + r#" WITH ward_audit_table AS ( SELECT sql - FROM sqlite_master + FROM main.sqlite_master WHERE type = 'table' AND name = 'ward_audit' ), ward_audit_exists AS ( @@ -369,35 +383,47 @@ WITH ) AS fp FROM ( SELECT cid, name, type, "notnull", dflt_value, pk - FROM pragma_table_info('ward_audit') + FROM pragma_table_info('ward_audit', 'main') ORDER BY cid ) ), ward_audit_index_fingerprint AS ( SELECT COALESCE(group_concat(item, '||'), '') AS fp FROM ( - SELECT printf('%s|%s', name, COALESCE(sql, '')) AS item - FROM sqlite_master - WHERE type = 'index' AND tbl_name = 'ward_audit' AND sql IS NOT NULL - ORDER BY name + SELECT printf('%s|%s', il.name, sm.sql) AS item + FROM pragma_index_list('ward_audit', 'main') AS il + JOIN main.sqlite_master AS sm + ON sm.type = 'index' AND sm.name = il.name + WHERE il.origin = 'c' AND sm.sql IS NOT NULL + ORDER BY il.name ) ), ward_audit_trigger_fingerprint AS ( SELECT COALESCE(group_concat(item, '||'), '') AS fp FROM ( SELECT printf('%s|%s', name, COALESCE(sql, '')) AS item - FROM sqlite_master + FROM main.sqlite_master WHERE type = 'trigger' AND tbl_name = 'ward_audit' ORDER BY name ) ), ward_audit_reserved_namespace_object_count AS ( SELECT COUNT(*) AS count - FROM sqlite_master + FROM main.sqlite_master WHERE type IN ('table', 'index', 'trigger', 'view') - AND lower(name) GLOB 'ward_audit*' + AND "#, + ward_audit_reserved_name_predicate_sql!(), + r#" AND NOT (type = 'table' AND name = 'ward_audit') ), + ward_audit_temp_shadow_object_count AS ( + SELECT COUNT(*) AS count + FROM temp.sqlite_master + WHERE type IN ('table', 'index', 'trigger', 'view') + AND "#, + ward_audit_reserved_name_predicate_sql!(), + r#" + ), ward_audit_shape AS ( SELECT (SELECT ok FROM ward_audit_exists) AS table_exists, @@ -405,9 +431,11 @@ WITH (SELECT fp FROM ward_audit_column_fingerprint) AS column_fp, (SELECT fp FROM ward_audit_index_fingerprint) AS index_fp, (SELECT fp FROM ward_audit_trigger_fingerprint) AS trigger_fp, - (SELECT count FROM ward_audit_reserved_namespace_object_count) AS reserved_namespace_count + (SELECT count FROM ward_audit_reserved_namespace_object_count) AS reserved_namespace_count, + (SELECT count FROM ward_audit_temp_shadow_object_count) AS temp_shadow_count ) "# + ) }; } @@ -516,6 +544,7 @@ macro_rules! ward_audit_exact_legacy_predicate_sql { concat!( r#" table_exists = 1 +AND temp_shadow_count = 0 AND table_sql = "#, ward_audit_exact_legacy_table_sql_sql!(), r#" @@ -536,6 +565,7 @@ macro_rules! ward_audit_exact_current_predicate_sql { concat!( r#" table_exists = 1 +AND temp_shadow_count = 0 AND table_sql IN ("#, ward_audit_exact_current_fresh_table_sql_sql!(), r#", "#, @@ -557,6 +587,7 @@ macro_rules! ward_audit_schema_state_case_sql { () => { concat!( r#"CASE + WHEN temp_shadow_count > 0 THEN 'unknown' WHEN table_exists = 0 AND reserved_namespace_count = 0 THEN 'missing' WHEN "#, ward_audit_exact_legacy_predicate_sql!(), @@ -570,13 +601,13 @@ END"# }; } -/// Table-local schema-state query for `ward.audit` inside `coven.sqlite3` -/// (§3.4). +/// Table-local schema-state query for the durable `main.ward_audit` contract +/// inside `coven.sqlite3` (§3.4). /// /// Callers run this exact query and branch on the stable text result: -/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — `ward_audit` is absent and the -/// reserved main-schema `ward_audit*` namespace is otherwise empty; initialize -/// with +/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — `main.ward_audit` is absent, the +/// reserved main-schema `ward_audit` / `ward_audit_*` namespace is otherwise +/// empty, and no temp shadow/reserved object exists; initialize with /// [`WARD_AUDIT_SCHEMA_SQL`]; /// - [`WARD_AUDIT_SCHEMA_STATE_LEGACY_V013`] — run /// [`WARD_AUDIT_MIGRATION_V014_SQL`]; @@ -584,17 +615,23 @@ END"# /// - [`WARD_AUDIT_SCHEMA_STATE_UNKNOWN`] — fail closed and investigate the /// table manually. /// -/// The fingerprint is strict: the exact `sqlite_master.sql` stored for -/// `ward_audit`, ordered column metadata (including `recorded_at`'s default and -/// primary-key flags), the exact explicit index SQL set, and the exact -/// append-only trigger SQL set must all match. Full stored-table-SQL equality -/// covers every declared table-level constraint (`CHECK`, `UNIQUE`, foreign-key -/// clauses, and the `event_type` list), so any extra or missing column, -/// constraint, index, or trigger returns `unknown`. When `ward_audit` itself is -/// absent, any preexisting main-schema table/index/trigger/view whose name is -/// reserved under the `ward_audit*` namespace also returns `unknown`, so -/// initialization cannot silently skip append-only objects on another table. No -/// whitespace-destroying normalization is applied: the only accepted +/// The fingerprint is strict: the exact `main.sqlite_master.sql` stored for +/// `main.ward_audit`, ordered column metadata from +/// `pragma_table_info('ward_audit', 'main')`, the exact explicit main-index SQL +/// set (discovered with `pragma_index_list('ward_audit', 'main')` and then read +/// from `main.sqlite_master`), and the exact append-only main-trigger SQL set +/// must all match. Full stored-table-SQL equality covers every declared +/// table-level constraint (`CHECK`, `UNIQUE`, foreign-key clauses, and the +/// `event_type` list), so any extra or missing column, constraint, index, or +/// trigger returns `unknown`. Any temp-schema table/view/index/trigger whose +/// name is exactly `ward_audit` or begins with `ward_audit_` also returns +/// `unknown`, even when `main.ward_audit` itself is exact current or legacy, so +/// callers cannot treat a temp shadow as healthy durable state. When +/// `main.ward_audit` itself is absent, any preexisting main-schema +/// table/index/trigger/view whose name is reserved under the same +/// `ward_audit` / `ward_audit_*` namespace likewise returns `unknown`, so +/// initialization cannot silently skip append-only objects on another durable +/// table. No whitespace-destroying normalization is applied: the only accepted /// `current_v014` table SQL variants are the fresh `CREATE TABLE ward_audit /// (...)` form and SQLite's quoted `CREATE TABLE "ward_audit" (...)` form /// produced by the exact legacy migration path, while the `legacy_v013` @@ -610,40 +647,41 @@ FROM ward_audit_shape; "#, ); -/// DDL migration for the exact `legacy_v013` `ward.audit` fingerprint inside -/// `coven.sqlite3` (§3.4). +/// DDL migration for the exact `main.ward_audit` `legacy_v013` fingerprint +/// inside `coven.sqlite3` (§3.4). /// /// Append-only is enforced *in the store*: UPDATE and DELETE abort via /// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). /// SQLite cannot `ALTER` a CHECK constraint on an existing table, so this /// transaction: /// 1. re-checks the exact legacy fingerprint in SQL before any mutation, -/// including exact stored `sqlite_master.sql` equality plus column/index/ -/// trigger fingerprints; -/// 2. adds the legacy `detail` column so the old table matches the copy shape; -/// 3. creates `ward_audit_new` with the updated CHECK; +/// including exact stored `main.sqlite_master.sql` equality plus main +/// column/index/trigger fingerprints and temp-shadow rejection; +/// 2. adds the legacy `detail` column on `main.ward_audit` so the old table +/// matches the copy shape; +/// 3. creates `main.ward_audit_new` with the updated CHECK; /// 4. copies every existing row, preserving `detail`; /// 5. swaps the tables; and -/// 6. re-creates the exact explicit indexes and append-only triggers. +/// 6. re-creates the exact explicit main indexes and append-only main triggers. /// /// Callers should still branch on [`WARD_AUDIT_SCHEMA_STATE_SQL`] first: /// initialize when the state is `missing`, migrate only `legacy_v013`, /// continue on `current_v014`, and fail closed on `unknown`. This migration -/// independently guards the same `legacy_v013` fingerprint so callers cannot -/// mutate a partial or already-current schema by skipping classification. If a -/// later step fails after `ALTER TABLE ... ADD COLUMN detail`, callers must -/// `ROLLBACK` the failed transaction before continuing so SQLite restores the -/// untouched legacy table. This SQL does not read or write database-wide -/// `PRAGMA user_version`. +/// independently guards the same `legacy_v013` fingerprint and temp-shadow +/// rejection so callers cannot mutate a partial, already-current, or +/// temp-shadowed schema by skipping classification. If a later step fails after +/// `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must `ROLLBACK` the +/// failed transaction before continuing so SQLite restores the untouched legacy +/// table. This SQL does not read or write database-wide `PRAGMA user_version`. pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = concat!( r#" BEGIN; -CREATE TEMP TABLE ward_audit_migration_guard ( +CREATE TEMP TABLE coven_threads_ward_audit_migration_guard ( ok INTEGER NOT NULL CHECK (ok = 1) ); -INSERT INTO ward_audit_migration_guard (ok) +INSERT INTO coven_threads_ward_audit_migration_guard (ok) "#, ward_audit_schema_state_ctes_sql!(), r#" @@ -655,11 +693,11 @@ SELECT CASE END FROM ward_audit_shape; -DROP TABLE ward_audit_migration_guard; +DROP TABLE coven_threads_ward_audit_migration_guard; -ALTER TABLE ward_audit ADD COLUMN detail TEXT; +ALTER TABLE main.ward_audit ADD COLUMN detail TEXT; -CREATE TABLE ward_audit_new ( +CREATE TABLE main.ward_audit_new ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_type TEXT NOT NULL CHECK (event_type IN ( 'proposal_submitted','proposal_approved','proposal_rejected', @@ -682,7 +720,7 @@ CREATE TABLE ward_audit_new ( recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); -INSERT INTO ward_audit_new ( +INSERT INTO main.ward_audit_new ( id, event_type, proposal_id, familiar_id, ward_version, ward_hash, tier, decision, approver, diff_hash, detail, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at @@ -691,21 +729,21 @@ SELECT id, event_type, proposal_id, familiar_id, ward_version, ward_hash, tier, decision, approver, diff_hash, detail, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at -FROM ward_audit; +FROM main.ward_audit; -DROP TABLE ward_audit; -ALTER TABLE ward_audit_new RENAME TO ward_audit; +DROP TABLE main.ward_audit; +ALTER TABLE main.ward_audit_new RENAME TO ward_audit; -CREATE INDEX IF NOT EXISTS ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); -CREATE INDEX IF NOT EXISTS ward_audit_event_idx ON ward_audit (event_type, recorded_at); +CREATE INDEX IF NOT EXISTS main.ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); +CREATE INDEX IF NOT EXISTS main.ward_audit_event_idx ON ward_audit (event_type, recorded_at); -CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_update +CREATE TRIGGER IF NOT EXISTS main.ward_audit_append_only_update BEFORE UPDATE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; -CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_delete +CREATE TRIGGER IF NOT EXISTS main.ward_audit_append_only_delete BEFORE DELETE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); @@ -715,12 +753,12 @@ COMMIT; "#, ); -// Shared raw current-v0.1.4 DDL body used by the guarded init SQL and drift -// tests. +// Shared durable-main current-v0.1.4 DDL body used by the guarded init SQL and +// drift tests. macro_rules! ward_audit_current_objects_sql { () => { r#" -CREATE TABLE IF NOT EXISTS ward_audit ( +CREATE TABLE IF NOT EXISTS main.ward_audit ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_type TEXT NOT NULL CHECK (event_type IN ( 'proposal_submitted','proposal_approved','proposal_rejected', @@ -743,16 +781,16 @@ CREATE TABLE IF NOT EXISTS ward_audit ( recorded_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); -CREATE INDEX IF NOT EXISTS ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); -CREATE INDEX IF NOT EXISTS ward_audit_event_idx ON ward_audit (event_type, recorded_at); +CREATE INDEX IF NOT EXISTS main.ward_audit_familiar_idx ON ward_audit (familiar_id, recorded_at); +CREATE INDEX IF NOT EXISTS main.ward_audit_event_idx ON ward_audit (event_type, recorded_at); -CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_update +CREATE TRIGGER IF NOT EXISTS main.ward_audit_append_only_update BEFORE UPDATE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; -CREATE TRIGGER IF NOT EXISTS ward_audit_append_only_delete +CREATE TRIGGER IF NOT EXISTS main.ward_audit_append_only_delete BEFORE DELETE ON ward_audit BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); @@ -761,25 +799,27 @@ END; }; } -/// DDL for the `ward.audit` table inside `coven.sqlite3` (§3.4). +/// DDL for the durable `main.ward_audit` table inside `coven.sqlite3` (§3.4). /// /// See module docs for the full schema-state contract. This transaction is /// safe to run unconditionally on every store open: it permits only exact /// `missing` or `current_v014` before any mutation, uses idempotent `IF NOT /// EXISTS` DDL for daemon compatibility, then requires exact `current_v014` -/// before `COMMIT`. Exact `legacy_v013` and every `unknown` shape fail closed. -/// If this SQL errors, callers must explicitly `ROLLBACK` before continuing so -/// SQLite discards any uncommitted work. This DDL never mutates database-wide -/// `PRAGMA user_version`. +/// before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, and +/// every temp shadow/reserved temp object fail closed. Durable schema objects +/// are explicitly created in `main`; the temp guard tables live in `temp` +/// under unique non-reserved names. If this SQL errors, callers must +/// explicitly `ROLLBACK` before continuing so SQLite discards any uncommitted +/// work. This DDL never mutates database-wide `PRAGMA user_version`. pub const WARD_AUDIT_SCHEMA_SQL: &str = concat!( r#" BEGIN; -CREATE TEMP TABLE ward_audit_schema_pre_guard ( +CREATE TEMP TABLE coven_threads_ward_audit_schema_pre_guard ( ok INTEGER NOT NULL CHECK (ok = 1) ); -INSERT INTO ward_audit_schema_pre_guard (ok) +INSERT INTO coven_threads_ward_audit_schema_pre_guard (ok) "#, ward_audit_schema_state_ctes_sql!(), r#" @@ -791,15 +831,15 @@ SELECT CASE END FROM ward_audit_shape; -DROP TABLE ward_audit_schema_pre_guard; +DROP TABLE coven_threads_ward_audit_schema_pre_guard; "#, ward_audit_current_objects_sql!(), r#" -CREATE TEMP TABLE ward_audit_schema_post_guard ( +CREATE TEMP TABLE coven_threads_ward_audit_schema_post_guard ( ok INTEGER NOT NULL CHECK (ok = 1) ); -INSERT INTO ward_audit_schema_post_guard (ok) +INSERT INTO coven_threads_ward_audit_schema_post_guard (ok) "#, ward_audit_schema_state_ctes_sql!(), r#" @@ -811,7 +851,7 @@ SELECT CASE END FROM ward_audit_shape; -DROP TABLE ward_audit_schema_post_guard; +DROP TABLE coven_threads_ward_audit_schema_post_guard; COMMIT; "#, @@ -925,24 +965,51 @@ END; .unwrap() } - fn stored_table_sql(conn: &Connection) -> String { + fn schema_master_table(schema: &str) -> &'static str { + match schema { + "main" => "main.sqlite_master", + "temp" => "temp.sqlite_master", + _ => panic!("unexpected schema: {schema}"), + } + } + + fn ward_audit_column_names(conn: &Connection, schema: &str) -> Vec { + let sql = + format!("SELECT name FROM pragma_table_info('ward_audit', '{schema}') ORDER BY cid;"); + let mut stmt = conn.prepare(&sql).unwrap(); + stmt.query_map([], |row| row.get::<_, String>(0)) + .unwrap() + .collect::>() + .unwrap() + } + + fn stored_table_sql_in_schema(conn: &Connection, schema: &str) -> String { conn.query_row( - "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'ward_audit';", + &format!( + "SELECT sql FROM {} WHERE type = 'table' AND name = 'ward_audit';", + schema_master_table(schema) + ), [], |row| row.get(0), ) .unwrap() } + fn stored_table_sql(conn: &Connection) -> String { + stored_table_sql_in_schema(conn, "main") + } + fn explicit_index_sql_fingerprint(conn: &Connection) -> String { conn.query_row( r#" SELECT COALESCE(group_concat(item, '||'), '') FROM ( - SELECT printf('%s|%s', name, sql) AS item - FROM sqlite_master - WHERE type = 'index' AND tbl_name = 'ward_audit' AND sql IS NOT NULL - ORDER BY name + SELECT printf('%s|%s', il.name, sm.sql) AS item + FROM pragma_index_list('ward_audit', 'main') AS il + JOIN main.sqlite_master AS sm + ON sm.type = 'index' AND sm.name = il.name + WHERE il.origin = 'c' AND sm.sql IS NOT NULL + ORDER BY il.name ); "#, [], @@ -957,7 +1024,7 @@ END; SELECT COALESCE(group_concat(item, '||'), '') FROM ( SELECT printf('%s|%s', name, sql) AS item - FROM sqlite_master + FROM main.sqlite_master WHERE type = 'trigger' AND tbl_name = 'ward_audit' ORDER BY name ); @@ -982,6 +1049,39 @@ END; schema_sql } + fn table_only_sql(base_schema_sql: &str) -> String { + let marker = "\n);\n\nCREATE INDEX"; + let (table_sql, _) = base_schema_sql + .split_once(marker) + .expect("ward_audit schema SQL must contain the table/index boundary marker"); + format!("{table_sql}\n);") + } + + fn temp_shadow_table_sql(base_schema_sql: &str) -> String { + let table_sql = table_only_sql(base_schema_sql); + let shadow_sql = table_sql + .replacen( + "CREATE TABLE IF NOT EXISTS main.ward_audit (", + "CREATE TEMP TABLE ward_audit (", + 1, + ) + .replacen( + "CREATE TABLE IF NOT EXISTS ward_audit (", + "CREATE TEMP TABLE ward_audit (", + 1, + ); + assert_ne!( + shadow_sql, table_sql, + "expected to rewrite ward_audit to TEMP" + ); + shadow_sql + } + + fn create_temp_shadow_table(conn: &Connection, base_schema_sql: &str) { + conn.execute_batch(&temp_shadow_table_sql(base_schema_sql)) + .unwrap(); + } + fn legacy_schema_with_extra_table_constraint(constraint_sql: &str) -> String { schema_sql_with_extra_table_constraint(LEGACY_WARD_AUDIT_SCHEMA_SQL, constraint_sql) } @@ -1019,23 +1119,27 @@ END; .collect() } - fn explicit_index_names(conn: &Connection) -> BTreeSet { - let mut stmt = conn - .prepare( - "SELECT name FROM pragma_index_list('ward_audit') WHERE origin = 'c' AND name NOT LIKE 'sqlite_autoindex_%' ORDER BY name;", - ) - .unwrap(); + fn explicit_index_names_in_schema(conn: &Connection, schema: &str) -> BTreeSet { + let sql = format!( + "SELECT name FROM pragma_index_list('ward_audit', '{schema}') WHERE origin = 'c' AND name NOT LIKE 'sqlite_autoindex_%' ORDER BY name;" + ); + let mut stmt = conn.prepare(&sql).unwrap(); stmt.query_map([], |row| row.get::<_, String>(0)) .unwrap() .collect::>() .unwrap() } - fn trigger_names(conn: &Connection) -> BTreeSet { + fn explicit_index_names(conn: &Connection) -> BTreeSet { + explicit_index_names_in_schema(conn, "main") + } + + fn trigger_names_in_schema(conn: &Connection, schema: &str) -> BTreeSet { let mut stmt = conn - .prepare( - "SELECT name FROM sqlite_master WHERE type = 'trigger' AND tbl_name = 'ward_audit' ORDER BY name;", - ) + .prepare(&format!( + "SELECT name FROM {} WHERE type = 'trigger' AND tbl_name = 'ward_audit' ORDER BY name;", + schema_master_table(schema) + )) .unwrap(); stmt.query_map([], |row| row.get::<_, String>(0)) .unwrap() @@ -1043,9 +1147,15 @@ END; .unwrap() } - fn has_column(conn: &Connection, name: &str) -> bool { + fn trigger_names(conn: &Connection) -> BTreeSet { + trigger_names_in_schema(conn, "main") + } + + fn has_column_in_schema(conn: &Connection, schema: &str, name: &str) -> bool { conn.query_row( - "SELECT EXISTS(SELECT 1 FROM pragma_table_info('ward_audit') WHERE name = ?1);", + &format!( + "SELECT EXISTS(SELECT 1 FROM pragma_table_info('ward_audit', '{schema}') WHERE name = ?1);" + ), params![name], |row| row.get::<_, i64>(0), ) @@ -1053,9 +1163,21 @@ END; == 1 } - fn main_schema_object_exists(conn: &Connection, object_type: &str, name: &str) -> bool { + fn has_column(conn: &Connection, name: &str) -> bool { + has_column_in_schema(conn, "main", name) + } + + fn schema_object_exists( + conn: &Connection, + schema: &str, + object_type: &str, + name: &str, + ) -> bool { conn.query_row( - "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type = ?1 AND name = ?2);", + &format!( + "SELECT EXISTS(SELECT 1 FROM {} WHERE type = ?1 AND name = ?2);", + schema_master_table(schema) + ), params![object_type, name], |row| row.get::<_, i64>(0), ) @@ -1063,6 +1185,23 @@ END; == 1 } + fn main_schema_object_exists(conn: &Connection, object_type: &str, name: &str) -> bool { + schema_object_exists(conn, "main", object_type, name) + } + + fn temp_schema_object_exists(conn: &Connection, object_type: &str, name: &str) -> bool { + schema_object_exists(conn, "temp", object_type, name) + } + + fn ward_audit_row_count(conn: &Connection, schema: &str) -> i64 { + conn.query_row( + &format!("SELECT COUNT(*) FROM {schema}.ward_audit;"), + [], + |row| row.get(0), + ) + .unwrap() + } + fn assert_fresh_schema_preserves_user_version(initial_version: i64) { let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, initial_version); @@ -1117,15 +1256,17 @@ END; assert_legacy_schema_guard_rollback_preserves_state(&schema_sql, after_rollback); } - fn load_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { + fn load_audit_row_from_schema(conn: &Connection, schema: &str, id: i64) -> StoredAuditRow { conn.query_row( - r#" + &format!( + r#" SELECT id, event_type, proposal_id, familiar_id, ward_version, ward_hash, tier, decision, approver, diff_hash, detail, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at - FROM ward_audit + FROM {schema}.ward_audit WHERE id = ?1 - "#, + "# + ), params![id], |row| { Ok(StoredAuditRow { @@ -1152,15 +1293,25 @@ END; .unwrap() } - fn load_legacy_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { + fn load_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { + load_audit_row_from_schema(conn, "main", id) + } + + fn load_legacy_audit_row_from_schema( + conn: &Connection, + schema: &str, + id: i64, + ) -> StoredAuditRow { conn.query_row( - r#" + &format!( + r#" SELECT id, event_type, proposal_id, familiar_id, ward_version, ward_hash, tier, decision, approver, diff_hash, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at - FROM ward_audit + FROM {schema}.ward_audit WHERE id = ?1 - "#, + "# + ), params![id], |row| { Ok(StoredAuditRow { @@ -1187,9 +1338,13 @@ END; .unwrap() } + fn load_legacy_audit_row(conn: &Connection, id: i64) -> StoredAuditRow { + load_legacy_audit_row_from_schema(conn, "main", id) + } + fn load_legacy_extra_value(conn: &Connection, id: i64) -> Option { conn.query_row( - "SELECT legacy_extra FROM ward_audit WHERE id = ?1;", + "SELECT legacy_extra FROM main.ward_audit WHERE id = ?1;", params![id], |row| row.get(0), ) @@ -1205,7 +1360,49 @@ END; .unwrap() } + fn insert_current_apply_audit_row_into_schema(conn: &Connection, schema: &str) -> i64 { + conn.execute( + &format!( + r#" + INSERT INTO {schema}.ward_audit ( + event_type, proposal_id, familiar_id, ward_version, ward_hash, + tier, decision, approver, diff_hash, detail, files_touched, + channel, thread_id, submitted_at, decided_at, recorded_at + ) VALUES ( + ?1, ?2, ?3, ?4, ?5, + ?6, ?7, ?8, ?9, ?10, ?11, + ?12, ?13, ?14, ?15, ?16 + ) + "# + ), + params![ + "apply_audit", + Some("proposal-current"), + "familiar-current", + Some("0.1.4"), + FIXED_WARD_HASH.as_ref(), + Some("tier_2"), + "applied", + Option::::None, + Some(FIXED_DIFF_HASH.as_ref()), + FIXED_PREV_DETAIL, + FIXED_FILES_TOUCHED, + Some("mutation"), + Option::::None, + FIXED_SUBMITTED_AT, + FIXED_DECIDED_AT, + FIXED_RECORDED_AT, + ], + ) + .unwrap(); + conn.last_insert_rowid() + } + fn insert_current_apply_audit_row(conn: &Connection) -> i64 { + insert_current_apply_audit_row_into_schema(conn, "main") + } + + fn insert_current_apply_audit_row_unqualified(conn: &Connection) -> i64 { conn.execute( r#" INSERT INTO ward_audit ( @@ -1241,14 +1438,16 @@ END; conn.last_insert_rowid() } - fn try_insert_legacy_ward_updated_row( + fn try_insert_legacy_ward_updated_row_into_schema( conn: &Connection, + schema: &str, decision: &str, recorded_at: &str, ) -> rusqlite::Result { conn.execute( - r#" - INSERT INTO ward_audit ( + &format!( + r#" + INSERT INTO {schema}.ward_audit ( event_type, proposal_id, familiar_id, ward_version, ward_hash, tier, decision, approver, diff_hash, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at @@ -1257,7 +1456,8 @@ END; ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15 ) - "#, + "# + ), params![ "ward_updated", Some("proposal-legacy"), @@ -1279,6 +1479,14 @@ END; Ok(conn.last_insert_rowid()) } + fn try_insert_legacy_ward_updated_row( + conn: &Connection, + decision: &str, + recorded_at: &str, + ) -> rusqlite::Result { + try_insert_legacy_ward_updated_row_into_schema(conn, "main", decision, recorded_at) + } + fn insert_legacy_ward_updated_row(conn: &Connection) -> i64 { try_insert_legacy_ward_updated_row(conn, "updated", FIXED_RECORDED_AT).unwrap() } @@ -1286,7 +1494,7 @@ END; fn insert_legacy_ward_updated_row_with_extra(conn: &Connection, extra: &str) -> i64 { conn.execute( r#" - INSERT INTO ward_audit ( + INSERT INTO main.ward_audit ( event_type, proposal_id, familiar_id, ward_version, ward_hash, tier, decision, approver, diff_hash, files_touched, channel, thread_id, submitted_at, decided_at, recorded_at, legacy_extra @@ -1454,6 +1662,204 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_MISSING); } + #[test] + fn schema_qualified_table_valued_pragmas_resolve_main_and_temp_separately() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + conn.execute_batch( + "CREATE TEMP TABLE ward_audit ( + id INTEGER PRIMARY KEY, + shadow TEXT NOT NULL + ); + CREATE INDEX temp.temp_shadow_idx ON ward_audit (shadow);", + ) + .unwrap(); + + assert_eq!( + ward_audit_column_names(&conn, "main"), + vec![ + "id".to_string(), + "event_type".to_string(), + "proposal_id".to_string(), + "familiar_id".to_string(), + "ward_version".to_string(), + "ward_hash".to_string(), + "tier".to_string(), + "decision".to_string(), + "approver".to_string(), + "diff_hash".to_string(), + "detail".to_string(), + "files_touched".to_string(), + "channel".to_string(), + "thread_id".to_string(), + "submitted_at".to_string(), + "decided_at".to_string(), + "recorded_at".to_string(), + ] + ); + assert_eq!( + ward_audit_column_names(&conn, "temp"), + vec!["id".to_string(), "shadow".to_string()] + ); + assert_eq!( + explicit_index_names_in_schema(&conn, "main"), + expected_explicit_index_names() + ); + assert_eq!( + explicit_index_names_in_schema(&conn, "temp"), + BTreeSet::from(["temp_shadow_idx".to_string()]) + ); + } + + #[test] + fn current_main_with_temp_shadow_is_unknown_and_guards_preserve_main_and_temp_rows() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + let main_row_id = insert_current_apply_audit_row(&conn); + let main_before = load_audit_row(&conn, main_row_id); + + create_temp_shadow_table(&conn, ward_audit_current_objects_sql!()); + let temp_row_id = insert_current_apply_audit_row_into_schema(&conn, "temp"); + let temp_before = load_audit_row_from_schema(&conn, "temp", temp_row_id); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let init_err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("temp shadow must make schema init fail closed"); + assert!( + init_err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error: {init_err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_audit_row(&conn, main_row_id), main_before); + assert_eq!( + load_audit_row_from_schema(&conn, "temp", temp_row_id), + temp_before + ); + assert_eq!(ward_audit_row_count(&conn, "main"), 1); + assert_eq!(ward_audit_row_count(&conn, "temp"), 1); + assert!(temp_schema_object_exists(&conn, "table", "ward_audit")); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let migration_err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("temp shadow must make migration fail closed"); + assert!( + migration_err + .to_string() + .contains("CHECK constraint failed"), + "unexpected migration error: {migration_err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_audit_row(&conn, main_row_id), main_before); + assert_eq!( + load_audit_row_from_schema(&conn, "temp", temp_row_id), + temp_before + ); + assert_eq!(ward_audit_row_count(&conn, "main"), 1); + assert_eq!(ward_audit_row_count(&conn, "temp"), 1); + assert!(temp_schema_object_exists(&conn, "table", "ward_audit")); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + conn.execute_batch("DROP TABLE temp.ward_audit;").unwrap(); + assert!(!temp_schema_object_exists(&conn, "table", "ward_audit")); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!(load_audit_row(&conn, main_row_id), main_before); + } + + #[test] + fn missing_main_with_temp_ward_audit_shadow_is_unknown_and_schema_sql_rejects_without_creating_main( + ) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 23); + create_temp_shadow_table(&conn, ward_audit_current_objects_sql!()); + let temp_row_id = insert_current_apply_audit_row_into_schema(&conn, "temp"); + let temp_before = load_audit_row_from_schema(&conn, "temp", temp_row_id); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("temp ward_audit shadow must block schema init before main creation"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error: {err}" + ); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit")); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert!(!main_schema_object_exists(&conn, "table", "ward_audit")); + assert_eq!( + load_audit_row_from_schema(&conn, "temp", temp_row_id), + temp_before + ); + assert_eq!(ward_audit_row_count(&conn, "temp"), 1); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert_eq!(user_version(&conn), 23); + + conn.execute_batch("DROP TABLE temp.ward_audit;").unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_MISSING); + } + + #[test] + fn missing_main_with_reserved_temp_objects_is_unknown_and_schema_sql_preserves_temp_objects() { + for (label, setup_sql, object_type, object_name) in [ + ( + "view", + "CREATE TEMP VIEW ward_audit_shadow_view AS SELECT 1 AS sentinel;", + "view", + "ward_audit_shadow_view", + ), + ( + "index", + "CREATE TEMP TABLE other (id INTEGER PRIMARY KEY, note TEXT NOT NULL); + CREATE INDEX temp.ward_audit_shadow_idx ON other (note);", + "index", + "ward_audit_shadow_idx", + ), + ( + "trigger", + "CREATE TEMP TABLE other (id INTEGER PRIMARY KEY, note TEXT NOT NULL); + CREATE TRIGGER temp.ward_audit_shadow_trigger + BEFORE UPDATE ON other + BEGIN + SELECT RAISE(ABORT, 'other is append-only'); + END;", + "trigger", + "ward_audit_shadow_trigger", + ), + ] { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(setup_sql).unwrap(); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("reserved temp object must block schema init"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error for {label}: {err}" + ); + assert!( + !main_schema_object_exists(&conn, "table", "ward_audit"), + "schema init for {label} must not create main.ward_audit before rollback" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert!( + temp_schema_object_exists(&conn, object_type, object_name), + "{label} temp object should survive rollback" + ); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit")); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + } + #[test] fn fresh_schema_sql_initializes_current_schema_atomically_and_enforces_append_only() { let conn = Connection::open_in_memory().unwrap(); @@ -1602,6 +2008,53 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); } + #[test] + fn legacy_main_with_temp_shadow_is_unknown_and_migration_rejects_before_mutating_either_schema() + { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + let main_row_id = insert_legacy_ward_updated_row(&conn); + let main_before = load_legacy_audit_row(&conn, main_row_id); + + create_temp_shadow_table(&conn, LEGACY_WARD_AUDIT_SCHEMA_SQL); + let temp_row_id = try_insert_legacy_ward_updated_row_into_schema( + &conn, + "temp", + "updated", + FIXED_RECORDED_AT, + ) + .unwrap(); + let temp_before = load_legacy_audit_row_from_schema(&conn, "temp", temp_row_id); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("temp shadow must make legacy migration fail before mutation"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected migration error: {err}" + ); + assert!(!has_column_in_schema(&conn, "main", "detail")); + assert!(!has_column_in_schema(&conn, "temp", "detail")); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit_new")); + assert!(temp_schema_object_exists(&conn, "table", "ward_audit")); + + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_legacy_audit_row(&conn, main_row_id), main_before); + assert_eq!( + load_legacy_audit_row_from_schema(&conn, "temp", temp_row_id), + temp_before + ); + assert_eq!(user_version(&conn), 37); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + conn.execute_batch("DROP TABLE temp.ward_audit;").unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + } + #[test] fn legacy_schema_with_spaced_event_type_literal_is_unknown_and_guard_preserves_state() { let schema_sql = drift_event_type_literal( @@ -2094,6 +2547,31 @@ END; assert_eq!(user_version(&conn), 37); } + #[test] + fn unqualified_insert_targets_temp_shadow_while_schema_state_stays_unknown() { + let conn = Connection::open_in_memory().unwrap(); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + let main_row_id = insert_current_apply_audit_row(&conn); + let main_before = load_audit_row(&conn, main_row_id); + + create_temp_shadow_table(&conn, ward_audit_current_objects_sql!()); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert_eq!(ward_audit_row_count(&conn, "main"), 1); + assert_eq!(ward_audit_row_count(&conn, "temp"), 0); + + let temp_row_id = insert_current_apply_audit_row_unqualified(&conn); + + assert_eq!(load_audit_row(&conn, main_row_id), main_before); + assert_eq!(ward_audit_row_count(&conn, "main"), 1); + assert_eq!(ward_audit_row_count(&conn, "temp"), 1); + assert_eq!( + load_audit_row_from_schema(&conn, "temp", temp_row_id).event_type, + "apply_audit" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + #[test] fn post_alter_failure_rollback_restores_legacy_schema_after_create_conflict() { let conn = Connection::open_in_memory().unwrap(); diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 0111b32..d233792 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -1,20 +1,24 @@ # ApplyAudit Migration Repair Implementation Plan **Goal:** Replace the old two-boolean `ward_audit` migration gate with a full -schema fingerprint/state contract built around exact stored -`sqlite_master.sql` fingerprints for the table/indexes/triggers plus ordered -column metadata, reserve the `ward_audit*` namespace on fresh installs, make -schema initialization fail closed and atomic, preserve evidence and unrelated -`user_version`, and prove rollback behavior with executable rusqlite tests. +schema fingerprint/state contract built around exact durable +`main.sqlite_master.sql` fingerprints for the table/indexes/triggers plus +ordered column metadata, reserve the durable `ward_audit` / +`ward_audit_*` namespace, fail closed on any TEMP shadow/reserved temp object, +make schema initialization fail closed and atomic, preserve evidence and +unrelated `user_version`, and prove rollback behavior with executable rusqlite +tests. **Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on `missing` / `legacy_v013` / `current_v014` / `unknown`, reserve `missing` for -an absent table with no preexisting main-schema `ward_audit*` objects, wrap -`WARD_AUDIT_SCHEMA_SQL` in atomic pre/post guards that allow only -`missing`/`current_v014`, and embed the same exact `legacy_v013` predicate -inside `WARD_AUDIT_MIGRATION_V014_SQL` before any `ALTER TABLE`. The init and -migration paths remain table-local and never write database-wide `user_version`. +an absent `main.ward_audit` with no preexisting durable reserved object and no +temp shadow/reserved temp object, wrap `WARD_AUDIT_SCHEMA_SQL` in atomic +pre/post guards that allow only `missing`/`current_v014`, and embed the same +exact `legacy_v013` + no-temp-shadow predicate inside +`WARD_AUDIT_MIGRATION_V014_SQL` before any `ALTER TABLE`. Durable DDL/DML stays +explicitly qualified to `main` wherever SQLite permits it, and the init/ +migration paths never write database-wide `user_version`. **Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` for in-memory tests. @@ -40,7 +44,13 @@ files: - exact legacy fixture → `legacy_v013`; - exact current schema → `current_v014`; - absent-table reserved-name collisions (`ward_audit_event_idx`, - `ward_audit_append_only_update`) → `unknown`. + `ward_audit_append_only_update`) → `unknown`; + - current/legacy/missing durable states plus a TEMP `ward_audit` shadow or + reserved temp object → `unknown`. +- Add syntax-verification tests for bundled SQLite: + - `pragma_table_info('ward_audit', 'main')` sees durable columns even when a + TEMP `ward_audit` exists; and + - `pragma_index_list('ward_audit', 'main')` sees durable explicit indexes. - Add init-safety tests for: - clean empty DB → `WARD_AUDIT_SCHEMA_SQL` succeeds atomically, lands on `current_v014`, and append-only UPDATE/DELETE still abort; @@ -48,6 +58,11 @@ files: preserves rows/objects; - exact legacy schema → `WARD_AUDIT_SCHEMA_SQL` rejects, requires explicit `ROLLBACK`, and preserves state/data; + - exact current `main.ward_audit` + TEMP shadow clone → init rejects, + explicit rollback preserves both main and temp rows, and dropping the TEMP + shadow restores `current_v014`; + - missing main + TEMP `ward_audit` / reserved `ward_audit_*` temp object → + init rejects and never creates `main.ward_audit`; - unknown partial current schema → `WARD_AUDIT_SCHEMA_SQL` rejects and preserves the drifted state. - Add drift tests for: @@ -70,6 +85,10 @@ files: - current with altered append-only trigger error literal/body → `unknown`. - Update existing migration failure tests so exact current/rerun cases fail at the new guard and always `ROLLBACK` with `unwrap()`. +- Add legacy TEMP-shadow migration coverage proving the guard rejects before any + main/temp mutation and rollback preserves the exact legacy durable table. +- Add reason-demo coverage proving an unqualified `INSERT INTO ward_audit ...` + lands in TEMP while the durable contract remains `unknown`. - Add post-`ALTER` rollback coverage by precreating conflicting `ward_audit_new`, forcing `CREATE TABLE ward_audit_new` to fail after the guard and `ALTER TABLE` succeed, then rolling back and asserting the legacy @@ -83,13 +102,18 @@ files: - Add stable state tag constants. - Add public `WARD_AUDIT_SCHEMA_STATE_SQL` that fingerprints: - - exact stored `sqlite_master.sql` for `ward_audit`, covering all table-level - constraints; - - ordered column metadata (including `recorded_at` default and PK metadata); - - exact explicit index SQL (ordered by name, excluding SQLite autoindexes); - - exact append-only trigger SQL (ordered by name); + - exact stored `main.sqlite_master.sql` for `ward_audit`, covering all + table-level constraints; + - ordered durable column metadata from `pragma_table_info('ward_audit', + 'main')` (including `recorded_at` default and PK metadata); + - explicit durable index discovery from `pragma_index_list('ward_audit', + 'main')`, with exact index SQL then read from `main.sqlite_master`; + - exact append-only trigger SQL from `main.sqlite_master`; - when `ward_audit` is absent, an empty reserved main-schema - `ward_audit*` namespace across tables/indexes/triggers/views; and + `ward_audit` / `ward_audit_*` namespace across tables/indexes/triggers/ + views; + - at every durable state, rejection when any temp-schema table/view/index/ + trigger is named `ward_audit` or begins with `ward_audit_`; and - only the controlled fresh/migrated current table-SQL variants plus the shipped legacy table-SQL variant, including the presence of `apply_audit` and any preserved inline comments. @@ -97,33 +121,36 @@ files: for root-API callers. - Update `WARD_AUDIT_SCHEMA_SQL` so it: - starts a transaction; - - creates a TEMP pre-install guard with `CHECK (ok = 1)`; + - creates a uniquely named TEMP pre-install guard with `CHECK (ok = 1)`; - inserts `1` only when the shared schema-state expression returns `missing` or `current_v014`, otherwise inserts `0` and aborts before any mutation; - keeps `CREATE TABLE/INDEX/TRIGGER IF NOT EXISTS` for current daemon - compatibility; - - creates a TEMP post-install guard that requires exact `current_v014`, - aborting on any silent no-op/collision or malformed result; + compatibility while targeting `main` unambiguously wherever SQLite syntax + allows it; + - creates a uniquely named TEMP post-install guard that requires exact + `current_v014`, aborting on any silent no-op/collision or malformed result; - commits on success; and - requires callers to `ROLLBACK` after any init error. - Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: - starts a transaction; - - creates a TEMP guard table with `CHECK (ok = 1)`; - - inserts `1` only when the full exact legacy predicate holds, otherwise - inserts `0` and aborts; + - creates a uniquely named TEMP guard table with `CHECK (ok = 1)`; + - inserts `1` only when the full exact legacy durable predicate holds and no + temp shadow/reserved temp object exists, otherwise inserts `0` and aborts; - drops the guard on the success path; - keeps `ALTER ADD detail`, strict replacement-table creation, detail copy, - and explicit index/trigger recreation; + and explicit main index/trigger recreation, all qualified to `main` + wherever SQLite syntax permits; - never writes `PRAGMA user_version`. ## Task 3: Update the written contract and validate - Update `audit.rs` docs, the design doc, the plan doc, and `CHANGELOG.md` to - say that exact stored `sqlite_master.sql` equality covers every declared - table-level constraint, alongside column/index/trigger fingerprints, that the - main-schema `ward_audit*` namespace is reserved on fresh installs, and that - no whitespace-destroying normalization is allowed. + say that the durable audit contract is `main.ward_audit`, exact stored + `main.sqlite_master.sql` equality covers every declared table-level + constraint, schema-qualified PRAGMAs are required for durable column/index + inspection, temp shadows/reserved temp objects fail closed, and no + whitespace-destroying normalization is allowed. - Document the caller contract explicitly: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - initialize on `missing` only; @@ -144,7 +171,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): make schema initialization fail closed +fix(audit): reject temporary schema shadows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index 1a7d7c6..5ab2234 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -11,32 +11,40 @@ schema contract. ## Schema-state contract `crates/coven-threads-core::WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable query -callers run before touching `ward_audit`. It returns one stable tag: +callers run before touching the durable `main.ward_audit` table. It returns one +stable tag: -- `missing` — `ward_audit` does not exist and no reserved main-schema - `ward_audit*` table/index/trigger/view names already exist; initialize with - `WARD_AUDIT_SCHEMA_SQL`. +- `missing` — `main.ward_audit` does not exist, no reserved durable + `ward_audit` / `ward_audit_*` main-schema object already exists, and no temp + shadow/reserved temp object exists; initialize with `WARD_AUDIT_SCHEMA_SQL`. - `legacy_v013` — exact v0.1.3 legacy fingerprint; run `WARD_AUDIT_MIGRATION_V014_SQL`. - `current_v014` — exact v0.1.4 current fingerprint; continue without schema work. - `unknown` — every other shape; fail closed and investigate manually. -The fingerprint is exact and table-local: +The fingerprint is exact and table-local to `main.ward_audit`: -- exact stored `sqlite_master.sql` for the `ward_audit` table; -- ordered column metadata from `pragma_table_info('ward_audit')`, including - name, type, `NOT NULL`, `DEFAULT`, and primary-key metadata; -- exact stored `sqlite_master.sql` for each explicit index (ordered by name and - fingerprinted as `name|sql`, excluding SQLite internal autoindexes); and -- exact stored `sqlite_master.sql` for each append-only trigger (ordered by - name and fingerprinted as `name|sql`). +- exact stored `main.sqlite_master.sql` for the `ward_audit` table; +- ordered durable-column metadata from `pragma_table_info('ward_audit', + 'main')`, including name, type, `NOT NULL`, `DEFAULT`, and primary-key + metadata; +- durable explicit index discovery from `pragma_index_list('ward_audit', + 'main')`, with exact index SQL then read from `main.sqlite_master` and + fingerprinted as `name|sql` (excluding SQLite internal autoindexes); and +- exact stored `main.sqlite_master.sql` for each append-only trigger (ordered + by name and fingerprinted as `name|sql`). The full stored table definition covers every declared table-level constraint — the `event_type` CHECK list, extra `CHECK` clauses, `UNIQUE` clauses, and foreign-key clauses — so any extra or missing constraint, column, index, or -trigger classifies as `unknown`. When `ward_audit` itself is absent, any -preexisting main-schema object whose name is reserved under `ward_audit*` also +trigger classifies as `unknown`. Any temp-schema table/view/index/trigger whose +name is exactly `ward_audit` or begins with `ward_audit_` also classifies as +`unknown`, even when `main.ward_audit` is otherwise exact current or legacy; +that fail-closed rule stops TEMP shadows from being mistaken for healthy +durable state and blocks unqualified daemon writes from proceeding under a false +`current_v014`. When `main.ward_audit` itself is absent, any preexisting +main-schema object whose name is reserved under the same durable namespace also classifies as `unknown`; that prevents schema init from silently skipping append-only objects already attached to other tables. No whitespace-destroying normalization is applied: only the exact stored SQL variants SQLite emits for @@ -53,37 +61,38 @@ the daemon executes it unconditionally on every store open. The safe contract is therefore: 1. `BEGIN;` -2. run a TEMP pre-install guard that reuses the exact schema-state CTEs and - permits only `missing` or exact `current_v014`; -3. create the `ward_audit` table, explicit indexes, and append-only triggers - with `IF NOT EXISTS`; -4. run a TEMP post-install guard that reuses the same schema-state expression - and requires exact `current_v014`; and +2. run a uniquely named TEMP pre-install guard that reuses the exact + schema-state CTEs and permits only `missing` or exact `current_v014`; +3. create the durable `main.ward_audit` table, explicit main indexes, and + append-only main triggers with `IF NOT EXISTS`; +4. run a uniquely named TEMP post-install guard that reuses the same + schema-state expression and requires exact `current_v014`; and 5. `COMMIT;` This makes fresh installs atomic, makes exact current reruns idempotent, and -fails closed for `legacy_v013`, `unknown`, or any malformed/silently skipped -install result. If either guard errors, callers must explicitly `ROLLBACK` -before continuing so SQLite removes any uncommitted `ward_audit` table created -on the failed path while preserving unrelated objects that caused the -collision. +fails closed for `legacy_v013`, `unknown`, any temp shadow/reserved temp +object, or any malformed/silently skipped install result. If either guard +errors, callers must explicitly `ROLLBACK` before continuing so SQLite removes +any uncommitted `main.ward_audit` table created on the failed path while +preserving unrelated durable/temp objects that caused the collision. ## Migration design The v0.1.4 repair remains a table-local migration. The SQL still: -1. `ALTER TABLE ward_audit ADD COLUMN detail TEXT`; -2. creates a strict replacement `ward_audit_new`; +1. `ALTER TABLE main.ward_audit ADD COLUMN detail TEXT`; +2. creates a strict replacement `main.ward_audit_new`; 3. copies every row, preserving `detail`; 4. swaps tables; and -5. re-creates the required explicit indexes and append-only triggers. +5. re-creates the required explicit main indexes and append-only main triggers. The change is the guardrail: immediately after `BEGIN;`, the migration creates -a TEMP guard table with `CHECK (ok = 1)` and inserts `1` only if the exact -`legacy_v013` fingerprint holds. Any missing/current/partial schema inserts `0` -instead, aborting before `ALTER TABLE`. This makes the migration fail closed -even if a caller skips the classification query, and keeps initialization and -migration aligned on the same exact state contract. +a uniquely named TEMP guard table with `CHECK (ok = 1)` and inserts `1` only if +the exact `legacy_v013` durable fingerprint holds **and** no temp +shadow/reserved temp object exists. Any missing/current/partial/temp-shadowed +schema inserts `0` instead, aborting before `ALTER TABLE`. This makes the +migration fail closed even if a caller skips the classification query, and keeps +initialization and migration aligned on the same exact state contract. If a later step fails after `ALTER TABLE` (for example, `ward_audit_new` already exists), callers must explicitly `ROLLBACK` the failed transaction @@ -96,8 +105,9 @@ version state is preserved. Executable rusqlite tests cover: 1. schema-state classification for `missing`, exact `legacy_v013`, and exact - `current_v014`, including absent-table reserved-namespace collisions - classifying as `unknown`; + `current_v014`, including absent-table reserved durable-namespace collisions, + reserved temp objects, and TEMP `ward_audit` shadows classifying as + `unknown`; 2. fresh empty-store initialization classifying `missing`, running `WARD_AUDIT_SCHEMA_SQL` atomically to `current_v014`, and enforcing append-only UPDATE/DELETE rejection; @@ -123,12 +133,16 @@ Executable rusqlite tests cover: 10. successful exact-legacy upgrade landing in `current_v014`, with fresh and migrated current schemas both classifying `current_v014` while matching only their controlled exact stored SQL variants; and -11. post-`ALTER` failure rollback restoring the full legacy table state. +11. post-`ALTER` failure rollback restoring the full legacy table state; and +12. schema-qualified PRAGMA syntax on bundled SQLite plus the reason for the + fail-closed contract: unqualified inserts hit TEMP first while the contract + remains `unknown`. ## Scope This repair does not add daemon migration orchestration or change the `WardAuditRecord` wire shape. The daemon remains responsible for calling `WARD_AUDIT_SCHEMA_STATE_SQL`, choosing fresh initialization vs legacy -migration, running `WARD_AUDIT_SCHEMA_SQL` only through the allowed +migration, treating `main.ward_audit` as the only durable audit contract, +running `WARD_AUDIT_SCHEMA_SQL` only through the allowed `missing`/`current_v014` contract, and failing closed on `unknown`. From 8c30603594e7ec36ca78efe3535af5466223e245 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 01:09:02 -0500 Subject: [PATCH 21/25] docs(audit): satisfy clippy list formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/coven-threads-core/src/audit.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index e0432a4..6e4fc49 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -42,6 +42,7 @@ //! table constraint, column, index, or trigger, any reserved durable-name //! collision, and any temp-schema table/view/index/trigger named //! `ward_audit` or `ward_audit_*`; fail closed. +//! //! The fingerprint uses exact `main.sqlite_master.sql` text for the durable //! table, explicit durable indexes, and append-only durable triggers, plus //! ordered `pragma_table_info('ward_audit', 'main')` metadata and From 8325fcade237e915f0fa24d4a503a54d78cf906b Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 01:41:08 -0500 Subject: [PATCH 22/25] fix(audit): reject durable namespace drift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 44 ++- crates/coven-threads-core/src/audit.rs | 336 ++++++++++++++---- ...2026-07-19-apply-audit-migration-repair.md | 52 +-- ...-19-apply-audit-migration-repair-design.md | 93 +++-- 4 files changed, 386 insertions(+), 139 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 40fd569..e133d9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,29 +19,35 @@ All notable changes to `coven-threads-core` are documented here. `main.sqlite_master.sql` fingerprints for the durable `main.ward_audit` table, explicit main indexes, and append-only main triggers, plus ordered durable column metadata discovered through schema-qualified PRAGMAs. - `missing` now means `main.ward_audit` is absent, the reserved main-schema - `ward_audit` / `ward_audit_*` namespace is otherwise empty, **and** no temp - shadow/reserved temp object exists; any temp-schema table/view/index/trigger - named `ward_audit` or `ward_audit_*` fail-closes as `unknown`. Only the - controlled fresh/migrated v0.1.4 table SQL variants and the shipped v0.1.3 - table SQL are accepted; no whitespace-destroying normalization is applied. + Across all durable states, the reserved main-schema `ward_audit` / + `ward_audit_*` namespace is whitelisted to exactly `main.ward_audit`, its two + explicit indexes, and its two append-only triggers attached to + `main.ward_audit`; every other reserved main object (including + `ward_audit_new`, views, backup/shadow tables, or reserved-name + indexes/triggers attached elsewhere) fail-closes as `unknown`. `missing` now + means `main.ward_audit` is absent, no unexpected durable reserved object + exists, **and** no temp shadow/reserved temp object exists; any temp-schema + table/view/index/trigger named `ward_audit` or `ward_audit_*` also + fail-closes as `unknown`. Only the controlled fresh/migrated v0.1.4 table SQL + variants and the shipped v0.1.3 table SQL are accepted; no + whitespace-destroying normalization is applied. - `WARD_AUDIT_SCHEMA_SQL` — atomic, self-guarding schema initialization for the exact `current_v014` fingerprint. It begins a transaction, permits only `missing` or exact `current_v014` before any mutation, uses idempotent `IF NOT EXISTS` DDL for daemon compatibility, targets durable schema objects in `main` wherever SQLite syntax permits, then requires exact `current_v014` - before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, and any - temp shadow/reserved temp object fail closed; callers must `ROLLBACK` after - any init error. + before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, any + unexpected durable reserved object, and any temp shadow/reserved temp object + fail closed; callers must `ROLLBACK` after any init error. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving rebuild of `main.ward_audit` guarded by the exact `legacy_v013` fingerprint - plus temp-shadow rejection. The daemon should query + plus durable-whitelist and temp-shadow rejection. The daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, migrate only `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The migration independently re-checks `legacy_v013` before any `ALTER`, using the - same exact stored-table + column/index/trigger predicate plus temp-shadow - rejection, then copies rows into the replacement table without discarding - evidence. + same exact stored-table + column/index/trigger predicate plus durable + namespace whitelist and temp-shadow rejection, then copies rows into the + replacement table without discarding evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, @@ -65,15 +71,17 @@ All notable changes to `coven-threads-core` are documented here. `current_schema_with_extra_index_is_unknown`, `current_schema_with_desc_index_is_unknown`, `current_schema_with_collated_index_is_unknown`, + `current_schema_with_extra_reserved_main_view_or_table_is_unknown_and_schema_sql_preserves_state`, `current_schema_with_altered_trigger_error_literal_is_unknown`, `current_schema_with_altered_trigger_body_is_unknown`, `absent_ward_audit_with_reserved_index_collision_is_unknown_and_schema_sql_preserves_other_objects`, `absent_ward_audit_with_reserved_trigger_collision_is_unknown_and_schema_sql_preserves_other_objects`, `unknown_partial_current_schema_rejects_schema_sql_and_preserves_state`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, - `legacy_schema_upgrades_and_preserves_append_only_behavior`, and - `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, and - `post_alter_failure_rollback_restores_legacy_schema_after_create_conflict`, and + `legacy_schema_with_preexisting_main_ward_audit_new_is_unknown_and_guard_rejects_before_alter`, + `legacy_schema_upgrades_and_preserves_append_only_behavior`, + `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, + `sqlite_post_alter_failure_rollback_restores_legacy_schema_for_production_migration_contract`, `schema_qualified_table_valued_pragmas_resolve_main_and_temp_separately`, `current_main_with_temp_shadow_is_unknown_and_guards_preserve_main_and_temp_rows`, `missing_main_with_temp_ward_audit_shadow_is_unknown_and_schema_sql_rejects_without_creating_main`, @@ -90,7 +98,9 @@ All notable changes to `coven-threads-core` are documented here. `legacy_v013` / `current_v014` / `unknown`, perform a quiet fail-closed init or upgrade, and recover cleanly with explicit rollback after any init or migration error. The durable contract is explicitly `main.ward_audit`, and - any TEMP shadow/reserved temp object keeps the contract fail-closed as + the reserved durable namespace is valid only for that table plus its two + explicit indexes and two append-only triggers; any extra durable reserved + object or TEMP shadow/reserved temp object keeps the contract fail-closed as `unknown`. Production dependencies stay unchanged, while `coven-threads-core` now carries bundled `rusqlite` as a dev-dependency for executable migration tests. diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 6e4fc49..ef139fe 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -30,18 +30,23 @@ //! explicitly `ROLLBACK` before continuing. //! `WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable, table-local fingerprint query //! that returns one of four stable tags: -//! - `missing` — `main.ward_audit` does not exist, the reserved durable -//! `ward_audit` / `ward_audit_*` namespace is otherwise empty in `main`, and -//! no temp-schema shadow/reserved object exists; initialize with +//! - `missing` — `main.ward_audit` does not exist, no unexpected durable +//! main-schema object named `ward_audit` or `ward_audit_*` exists, and no +//! temp-schema shadow/reserved object exists; initialize with //! `WARD_AUDIT_SCHEMA_SQL`; //! - `legacy_v013` — `main.ward_audit` exactly matches the v0.1.3 legacy -//! fingerprint and no temp shadow exists; run `WARD_AUDIT_MIGRATION_V014_SQL`; +//! fingerprint, the durable reserved namespace contains only the expected +//! table/index/trigger objects attached to `main.ward_audit`, and no temp +//! shadow exists; run `WARD_AUDIT_MIGRATION_V014_SQL`; //! - `current_v014` — `main.ward_audit` exactly matches the v0.1.4 current -//! fingerprint and no temp shadow exists; continue without schema work; +//! fingerprint, the durable reserved namespace contains only the expected +//! table/index/trigger objects attached to `main.ward_audit`, and no temp +//! shadow exists; continue without schema work; //! - `unknown` — every other shape, including any extra or missing declared -//! table constraint, column, index, or trigger, any reserved durable-name -//! collision, and any temp-schema table/view/index/trigger named -//! `ward_audit` or `ward_audit_*`; fail closed. +//! table constraint, column, index, or trigger, any unexpected durable +//! main-schema object named `ward_audit` or `ward_audit_*`, and any +//! temp-schema table/view/index/trigger named `ward_audit` or +//! `ward_audit_*`; fail closed. //! //! The fingerprint uses exact `main.sqlite_master.sql` text for the durable //! table, explicit durable indexes, and append-only durable triggers, plus @@ -53,10 +58,11 @@ //! migration path, and the `legacy_v013` fingerprint includes the inline //! comments preserved from the shipped v0.1.3 DDL. //! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` -//! fingerprint with no temp shadow. It independently re-checks that fingerprint -//! inside the transaction before any `ALTER`, then adds `detail` and rebuilds -//! `main.ward_audit` so the current CHECK set is installed and every existing -//! row is preserved. If a later migration step fails after +//! fingerprint with no unexpected durable reserved-namespace object and no temp +//! shadow. It independently re-checks that fingerprint inside the transaction +//! before any `ALTER`, then adds `detail` and rebuilds `main.ward_audit` so the +//! current CHECK set is installed and every existing row is preserved. If a +//! later migration step fails after //! `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must explicitly roll //! back the failed transaction before continuing so SQLite restores the //! untouched legacy table. @@ -332,19 +338,21 @@ fn reject_tag(reason: &RejectReason) -> &'static str { } /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] when `main.ward_audit` -/// is absent, the reserved main-schema `ward_audit` / `ward_audit_*` namespace -/// is otherwise empty, and no temp shadow/reserved object blocks the durable -/// contract. +/// is absent, no unexpected durable main-schema `ward_audit` / +/// `ward_audit_*` object exists, and no temp shadow/reserved object blocks the +/// durable contract. pub const WARD_AUDIT_SCHEMA_STATE_MISSING: &str = "missing"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact -/// `main.ward_audit` v0.1.3 legacy schema fingerprint, with no temp shadow. +/// `main.ward_audit` v0.1.3 legacy schema fingerprint, with no unexpected +/// durable reserved-namespace object and no temp shadow. pub const WARD_AUDIT_SCHEMA_STATE_LEGACY_V013: &str = "legacy_v013"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact -/// `main.ward_audit` v0.1.4 current schema fingerprint, with no temp shadow. +/// `main.ward_audit` v0.1.4 current schema fingerprint, with no unexpected +/// durable reserved-namespace object and no temp shadow. pub const WARD_AUDIT_SCHEMA_STATE_CURRENT_V014: &str = "current_v014"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for every other -/// `main.ward_audit` shape, plus reserved-name collisions in the durable -/// namespace or any temp-schema `ward_audit` / `ward_audit_*` shadow object. +/// `main.ward_audit` shape, plus any unexpected durable reserved-namespace +/// object or any temp-schema `ward_audit` / `ward_audit_*` shadow object. pub const WARD_AUDIT_SCHEMA_STATE_UNKNOWN: &str = "unknown"; macro_rules! ward_audit_reserved_name_predicate_sql { @@ -353,6 +361,18 @@ macro_rules! ward_audit_reserved_name_predicate_sql { }; } +macro_rules! ward_audit_expected_durable_object_predicate_sql { + () => { + r#"( + (type = 'table' AND name = 'ward_audit') + OR (type = 'index' AND name = 'ward_audit_event_idx' AND tbl_name = 'ward_audit') + OR (type = 'index' AND name = 'ward_audit_familiar_idx' AND tbl_name = 'ward_audit') + OR (type = 'trigger' AND name = 'ward_audit_append_only_update' AND tbl_name = 'ward_audit') + OR (type = 'trigger' AND name = 'ward_audit_append_only_delete' AND tbl_name = 'ward_audit') + )"# + }; +} + macro_rules! ward_audit_schema_state_ctes_sql { () => { concat!( @@ -408,14 +428,16 @@ WITH ORDER BY name ) ), - ward_audit_reserved_namespace_object_count AS ( + ward_audit_unexpected_durable_namespace_object_count AS ( SELECT COUNT(*) AS count FROM main.sqlite_master WHERE type IN ('table', 'index', 'trigger', 'view') AND "#, ward_audit_reserved_name_predicate_sql!(), r#" - AND NOT (type = 'table' AND name = 'ward_audit') + AND NOT "#, + ward_audit_expected_durable_object_predicate_sql!(), + r#" ), ward_audit_temp_shadow_object_count AS ( SELECT COUNT(*) AS count @@ -432,7 +454,8 @@ WITH (SELECT fp FROM ward_audit_column_fingerprint) AS column_fp, (SELECT fp FROM ward_audit_index_fingerprint) AS index_fp, (SELECT fp FROM ward_audit_trigger_fingerprint) AS trigger_fp, - (SELECT count FROM ward_audit_reserved_namespace_object_count) AS reserved_namespace_count, + (SELECT count FROM ward_audit_unexpected_durable_namespace_object_count) + AS unexpected_durable_namespace_object_count, (SELECT count FROM ward_audit_temp_shadow_object_count) AS temp_shadow_count ) "# @@ -545,6 +568,7 @@ macro_rules! ward_audit_exact_legacy_predicate_sql { concat!( r#" table_exists = 1 +AND unexpected_durable_namespace_object_count = 0 AND temp_shadow_count = 0 AND table_sql = "#, ward_audit_exact_legacy_table_sql_sql!(), @@ -566,6 +590,7 @@ macro_rules! ward_audit_exact_current_predicate_sql { concat!( r#" table_exists = 1 +AND unexpected_durable_namespace_object_count = 0 AND temp_shadow_count = 0 AND table_sql IN ("#, ward_audit_exact_current_fresh_table_sql_sql!(), @@ -589,7 +614,7 @@ macro_rules! ward_audit_schema_state_case_sql { concat!( r#"CASE WHEN temp_shadow_count > 0 THEN 'unknown' - WHEN table_exists = 0 AND reserved_namespace_count = 0 THEN 'missing' + WHEN table_exists = 0 AND unexpected_durable_namespace_object_count = 0 THEN 'missing' WHEN "#, ward_audit_exact_legacy_predicate_sql!(), r#" THEN 'legacy_v013' @@ -606,10 +631,10 @@ END"# /// inside `coven.sqlite3` (§3.4). /// /// Callers run this exact query and branch on the stable text result: -/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — `main.ward_audit` is absent, the -/// reserved main-schema `ward_audit` / `ward_audit_*` namespace is otherwise -/// empty, and no temp shadow/reserved object exists; initialize with -/// [`WARD_AUDIT_SCHEMA_SQL`]; +/// - [`WARD_AUDIT_SCHEMA_STATE_MISSING`] — `main.ward_audit` is absent, no +/// unexpected durable main-schema object named `ward_audit` or +/// `ward_audit_*` exists, and no temp shadow/reserved object exists; +/// initialize with [`WARD_AUDIT_SCHEMA_SQL`]; /// - [`WARD_AUDIT_SCHEMA_STATE_LEGACY_V013`] — run /// [`WARD_AUDIT_MIGRATION_V014_SQL`]; /// - [`WARD_AUDIT_SCHEMA_STATE_CURRENT_V014`] — continue without schema work; @@ -624,20 +649,23 @@ END"# /// must all match. Full stored-table-SQL equality covers every declared /// table-level constraint (`CHECK`, `UNIQUE`, foreign-key clauses, and the /// `event_type` list), so any extra or missing column, constraint, index, or -/// trigger returns `unknown`. Any temp-schema table/view/index/trigger whose -/// name is exactly `ward_audit` or begins with `ward_audit_` also returns -/// `unknown`, even when `main.ward_audit` itself is exact current or legacy, so -/// callers cannot treat a temp shadow as healthy durable state. When -/// `main.ward_audit` itself is absent, any preexisting main-schema -/// table/index/trigger/view whose name is reserved under the same -/// `ward_audit` / `ward_audit_*` namespace likewise returns `unknown`, so -/// initialization cannot silently skip append-only objects on another durable -/// table. No whitespace-destroying normalization is applied: the only accepted -/// `current_v014` table SQL variants are the fresh `CREATE TABLE ward_audit -/// (...)` form and SQLite's quoted `CREATE TABLE "ward_audit" (...)` form -/// produced by the exact legacy migration path, while the `legacy_v013` -/// fingerprint intentionally includes the inline comments preserved from the -/// shipped v0.1.3 DDL. +/// trigger returns `unknown`. Across **all** durable states, the reserved +/// main-schema namespace is whitelisted to exactly these objects attached to +/// `main.ward_audit`: the `ward_audit` table, `ward_audit_event_idx`, +/// `ward_audit_familiar_idx`, `ward_audit_append_only_update`, and +/// `ward_audit_append_only_delete`. Any other main-schema table/view/index/ +/// trigger whose name is exactly `ward_audit` or begins with `ward_audit_` +/// returns `unknown`, including `ward_audit_new`, backup/shadow tables, or +/// reserved-name indexes/triggers attached elsewhere. Any temp-schema +/// table/view/index/trigger whose name is exactly `ward_audit` or begins with +/// `ward_audit_` also returns `unknown`, even when `main.ward_audit` itself is +/// exact current or legacy, so callers cannot treat a shadowed namespace as +/// healthy durable state. No whitespace-destroying normalization is applied: +/// the only accepted `current_v014` table SQL variants are the fresh +/// `CREATE TABLE ward_audit (...)` form and SQLite's quoted +/// `CREATE TABLE "ward_audit" (...)` form produced by the exact legacy +/// migration path, while the `legacy_v013` fingerprint intentionally includes +/// the inline comments preserved from the shipped v0.1.3 DDL. pub const WARD_AUDIT_SCHEMA_STATE_SQL: &str = concat!( ward_audit_schema_state_ctes_sql!(), r#" @@ -657,7 +685,8 @@ FROM ward_audit_shape; /// transaction: /// 1. re-checks the exact legacy fingerprint in SQL before any mutation, /// including exact stored `main.sqlite_master.sql` equality plus main -/// column/index/trigger fingerprints and temp-shadow rejection; +/// column/index/trigger fingerprints, the durable reserved-namespace +/// whitelist, and temp-shadow rejection; /// 2. adds the legacy `detail` column on `main.ward_audit` so the old table /// matches the copy shape; /// 3. creates `main.ward_audit_new` with the updated CHECK; @@ -668,12 +697,13 @@ FROM ward_audit_shape; /// Callers should still branch on [`WARD_AUDIT_SCHEMA_STATE_SQL`] first: /// initialize when the state is `missing`, migrate only `legacy_v013`, /// continue on `current_v014`, and fail closed on `unknown`. This migration -/// independently guards the same `legacy_v013` fingerprint and temp-shadow -/// rejection so callers cannot mutate a partial, already-current, or -/// temp-shadowed schema by skipping classification. If a later step fails after -/// `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must `ROLLBACK` the -/// failed transaction before continuing so SQLite restores the untouched legacy -/// table. This SQL does not read or write database-wide `PRAGMA user_version`. +/// independently guards the same `legacy_v013` fingerprint, durable +/// reserved-namespace whitelist, and temp-shadow rejection so callers cannot +/// mutate a partial, already-current, or shadowed schema by skipping +/// classification. If a later step fails after `ALTER TABLE main.ward_audit ADD +/// COLUMN detail`, callers must `ROLLBACK` the failed transaction before +/// continuing so SQLite restores the untouched legacy table. This SQL does not +/// read or write database-wide `PRAGMA user_version`. pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = concat!( r#" BEGIN; @@ -806,12 +836,13 @@ END; /// safe to run unconditionally on every store open: it permits only exact /// `missing` or `current_v014` before any mutation, uses idempotent `IF NOT /// EXISTS` DDL for daemon compatibility, then requires exact `current_v014` -/// before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, and -/// every temp shadow/reserved temp object fail closed. Durable schema objects -/// are explicitly created in `main`; the temp guard tables live in `temp` -/// under unique non-reserved names. If this SQL errors, callers must -/// explicitly `ROLLBACK` before continuing so SQLite discards any uncommitted -/// work. This DDL never mutates database-wide `PRAGMA user_version`. +/// before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, every +/// unexpected durable reserved-namespace object, and every temp +/// shadow/reserved temp object fail closed. Durable schema objects are +/// explicitly created in `main`; the temp guard tables live in `temp` under +/// unique non-reserved names. If this SQL errors, callers must explicitly +/// `ROLLBACK` before continuing so SQLite discards any uncommitted work. This +/// DDL never mutates database-wide `PRAGMA user_version`. pub const WARD_AUDIT_SCHEMA_SQL: &str = concat!( r#" BEGIN; @@ -1194,6 +1225,46 @@ END; schema_object_exists(conn, "temp", object_type, name) } + fn reserved_main_namespace_object_key(object_type: &str, name: &str, tbl_name: &str) -> String { + format!("{object_type}|{name}|{tbl_name}") + } + + fn reserved_main_namespace_objects(conn: &Connection) -> BTreeSet { + let mut stmt = conn + .prepare( + r#" + SELECT printf('%s|%s|%s', type, name, COALESCE(tbl_name, '')) + FROM main.sqlite_master + WHERE type IN ('table', 'index', 'trigger', 'view') + AND (lower(name) = 'ward_audit' OR lower(name) GLOB 'ward_audit_*') + ORDER BY type, name; + "#, + ) + .unwrap(); + stmt.query_map([], |row| row.get::<_, String>(0)) + .unwrap() + .collect::>() + .unwrap() + } + + fn expected_reserved_main_namespace_objects() -> BTreeSet { + BTreeSet::from([ + reserved_main_namespace_object_key("index", "ward_audit_event_idx", "ward_audit"), + reserved_main_namespace_object_key("index", "ward_audit_familiar_idx", "ward_audit"), + reserved_main_namespace_object_key("table", "ward_audit", "ward_audit"), + reserved_main_namespace_object_key( + "trigger", + "ward_audit_append_only_delete", + "ward_audit", + ), + reserved_main_namespace_object_key( + "trigger", + "ward_audit_append_only_update", + "ward_audit", + ), + ]) + } + fn ward_audit_row_count(conn: &Connection, schema: &str) -> i64 { conn.query_row( &format!("SELECT COUNT(*) FROM {schema}.ward_audit;"), @@ -1894,6 +1965,10 @@ END; let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + assert_eq!( + reserved_main_namespace_objects(&conn), + expected_reserved_main_namespace_objects() + ); assert_eq!( stored_table_sql(&conn), sql_literal_value(&conn, ward_audit_exact_legacy_table_sql_sql!()) @@ -1913,6 +1988,10 @@ END; let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!( + reserved_main_namespace_objects(&conn), + expected_reserved_main_namespace_objects() + ); assert_eq!( stored_table_sql(&conn), sql_literal_value(&conn, ward_audit_exact_current_fresh_table_sql_sql!()) @@ -2271,6 +2350,78 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); } + #[test] + fn current_schema_with_extra_reserved_main_view_or_table_is_unknown_and_schema_sql_preserves_state( + ) { + for (label, setup_sql, object_type, object_name, object_row_value) in [ + ( + "view", + "CREATE VIEW ward_audit_history AS SELECT id, decision FROM main.ward_audit;", + "view", + "ward_audit_history", + None, + ), + ( + "table", + "CREATE TABLE ward_audit_backup (id INTEGER PRIMARY KEY, note TEXT NOT NULL); + INSERT INTO ward_audit_backup (id, note) VALUES (1, 'sentinel');", + "table", + "ward_audit_backup", + Some("sentinel"), + ), + ] { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 41); + conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_current_apply_audit_row(&conn); + let before = load_audit_row(&conn, row_id); + + conn.execute_batch(setup_sql).unwrap(); + + let reserved_objects = reserved_main_namespace_objects(&conn); + assert!( + expected_reserved_main_namespace_objects().is_subset(&reserved_objects), + "{label} drift must preserve the expected durable whitelist objects" + ); + assert!( + reserved_objects.contains(&reserved_main_namespace_object_key( + object_type, + object_name, + object_name + )), + "{label} drift should register the extra durable namespace object" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_SCHEMA_SQL) + .expect_err("extra reserved durable object must make schema init fail closed"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected schema init error for {label}: {err}" + ); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_audit_row(&conn, row_id), before); + assert_eq!(user_version(&conn), 41); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + assert!(main_schema_object_exists(&conn, object_type, object_name)); + if let Some(expected_note) = object_row_value { + assert_eq!( + conn.query_row( + "SELECT note FROM ward_audit_backup WHERE id = 1;", + [], + |row| row.get::<_, String>(0) + ) + .unwrap(), + expected_note + ); + } + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + } + } + #[test] fn current_schema_with_altered_trigger_error_literal_is_unknown() { let conn = Connection::open_in_memory().unwrap(); @@ -2463,6 +2614,54 @@ END; assert_eq!(user_version(&conn), before_version); } + #[test] + fn legacy_schema_with_preexisting_main_ward_audit_new_is_unknown_and_guard_rejects_before_alter( + ) { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_legacy_ward_updated_row(&conn); + let before = load_legacy_audit_row(&conn, row_id); + conn.execute_batch( + "CREATE TABLE main.ward_audit_new (conflict TEXT NOT NULL); + INSERT INTO main.ward_audit_new (conflict) VALUES ('sentinel');", + ) + .unwrap(); + + let reserved_objects = reserved_main_namespace_objects(&conn); + assert!( + expected_reserved_main_namespace_objects().is_subset(&reserved_objects), + "legacy fixture should still include the expected durable whitelist objects" + ); + assert!( + reserved_objects.contains(&reserved_main_namespace_object_key( + "table", + "ward_audit_new", + "ward_audit_new" + )), + "preexisting ward_audit_new should register as an unexpected durable object" + ); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + let err = conn + .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .expect_err("preexisting ward_audit_new must make migration fail before ALTER"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected migration error: {err}" + ); + assert!(!has_column(&conn, "detail")); + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_legacy_audit_row(&conn, row_id), before); + assert_eq!(user_version(&conn), 37); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + assert!(!has_column(&conn, "detail")); + assert_eq!(ward_audit_new_conflict_value(&conn), "sentinel"); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + } + #[test] fn legacy_schema_upgrades_and_preserves_append_only_behavior() { let conn = Connection::open_in_memory().unwrap(); @@ -2574,22 +2773,29 @@ END; } #[test] - fn post_alter_failure_rollback_restores_legacy_schema_after_create_conflict() { + fn sqlite_post_alter_failure_rollback_restores_legacy_schema_for_production_migration_contract() + { let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, 37); conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); let row_id = insert_legacy_ward_updated_row(&conn); let before = load_legacy_audit_row(&conn, row_id); - conn.execute_batch( - "CREATE TABLE ward_audit_new (conflict TEXT NOT NULL);\nINSERT INTO ward_audit_new (conflict) VALUES ('sentinel');", - ) - .unwrap(); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + // Validate SQLite rollback semantics for the production migration's + // post-ALTER failure contract without relying on a preexisting reserved + // durable namespace object. let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) - .expect_err("conflicting replacement table must fail after ALTER TABLE succeeds"); + .execute_batch( + r#" + BEGIN; + ALTER TABLE main.ward_audit ADD COLUMN detail TEXT; + CREATE TABLE main.ward_audit_new (conflict TEXT NOT NULL); + CREATE TABLE main.ward_audit_new (conflict TEXT NOT NULL); + "#, + ) + .expect_err("controlled post-ALTER SQL failure must abort the transaction"); assert!( err.to_string() .contains("table ward_audit_new already exists"), @@ -2601,6 +2807,12 @@ END; assert_eq!(user_version(&conn), 37); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); assert!(!has_column(&conn, "detail")); - assert_eq!(ward_audit_new_conflict_value(&conn), "sentinel"); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit_new")); + assert_eq!( + stored_table_sql(&conn), + sql_literal_value(&conn, ward_audit_exact_legacy_table_sql_sql!()) + ); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); } } diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index d233792..75d6270 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -4,21 +4,22 @@ schema fingerprint/state contract built around exact durable `main.sqlite_master.sql` fingerprints for the table/indexes/triggers plus ordered column metadata, reserve the durable `ward_audit` / -`ward_audit_*` namespace, fail closed on any TEMP shadow/reserved temp object, -make schema initialization fail closed and atomic, preserve evidence and -unrelated `user_version`, and prove rollback behavior with executable rusqlite -tests. +`ward_audit_*` namespace to an exact five-object whitelist in every state, fail +closed on any TEMP shadow/reserved temp object, make schema initialization fail +closed and atomic, preserve evidence and unrelated `user_version`, and prove +rollback behavior with executable rusqlite tests. **Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on `missing` / `legacy_v013` / `current_v014` / `unknown`, reserve `missing` for -an absent `main.ward_audit` with no preexisting durable reserved object and no -temp shadow/reserved temp object, wrap `WARD_AUDIT_SCHEMA_SQL` in atomic +an absent `main.ward_audit` with no unexpected durable reserved-name object and +no temp shadow/reserved temp object, require the same exact durable whitelist +for `legacy_v013` and `current_v014`, wrap `WARD_AUDIT_SCHEMA_SQL` in atomic pre/post guards that allow only `missing`/`current_v014`, and embed the same -exact `legacy_v013` + no-temp-shadow predicate inside -`WARD_AUDIT_MIGRATION_V014_SQL` before any `ALTER TABLE`. Durable DDL/DML stays -explicitly qualified to `main` wherever SQLite permits it, and the init/ -migration paths never write database-wide `user_version`. +exact `legacy_v013` + no-unexpected-durable-object + no-temp-shadow predicate +inside `WARD_AUDIT_MIGRATION_V014_SQL` before any `ALTER TABLE`. Durable +DDL/DML stays explicitly qualified to `main` wherever SQLite permits it, and +the init/migration paths never write database-wide `user_version`. **Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` for in-memory tests. @@ -43,8 +44,12 @@ files: - empty DB → `missing`; - exact legacy fixture → `legacy_v013`; - exact current schema → `current_v014`; + - exact legacy/current with only the durable whitelist objects still + classifying correctly; - absent-table reserved-name collisions (`ward_audit_event_idx`, `ward_audit_append_only_update`) → `unknown`; + - exact current + extra durable reserved-name view/table → `unknown`; + - exact legacy + preexisting `main.ward_audit_new` → `unknown`; - current/legacy/missing durable states plus a TEMP `ward_audit` shadow or reserved temp object → `unknown`. - Add syntax-verification tests for bundled SQLite: @@ -89,10 +94,11 @@ files: main/temp mutation and rollback preserves the exact legacy durable table. - Add reason-demo coverage proving an unqualified `INSERT INTO ward_audit ...` lands in TEMP while the durable contract remains `unknown`. -- Add post-`ALTER` rollback coverage by precreating conflicting - `ward_audit_new`, forcing `CREATE TABLE ward_audit_new` to fail after the - guard and `ALTER TABLE` succeed, then rolling back and asserting the legacy - row/schema are restored. +- Add a controlled post-`ALTER` rollback coverage path that starts from exact + legacy, performs the same `ALTER TABLE main.ward_audit ADD COLUMN detail + TEXT`, then forces a later SQL error by creating `main.ward_audit_new` twice + inside the same transaction; require the error, explicit rollback success, + full row/schema/user_version restoration, and `legacy_v013` classification. - Add exact-stored-SQL tests proving fresh current and successfully migrated current both classify `current_v014` while matching only their controlled SQLite-emitted table-SQL variants, and that the shipped legacy fixture @@ -109,9 +115,11 @@ files: - explicit durable index discovery from `pragma_index_list('ward_audit', 'main')`, with exact index SQL then read from `main.sqlite_master`; - exact append-only trigger SQL from `main.sqlite_master`; - - when `ward_audit` is absent, an empty reserved main-schema - `ward_audit` / `ward_audit_*` namespace across tables/indexes/triggers/ - views; + - at every durable state, an exact reserved main-schema whitelist consisting + only of table `main.ward_audit`, indexes `ward_audit_event_idx` and + `ward_audit_familiar_idx` attached to it, and triggers + `ward_audit_append_only_update` / `ward_audit_append_only_delete` attached + to it, with every other `ward_audit` / `ward_audit_*` main object rejected; - at every durable state, rejection when any temp-schema table/view/index/ trigger is named `ward_audit` or begins with `ward_audit_`; and - only the controlled fresh/migrated current table-SQL variants plus the @@ -136,7 +144,8 @@ files: - starts a transaction; - creates a uniquely named TEMP guard table with `CHECK (ok = 1)`; - inserts `1` only when the full exact legacy durable predicate holds and no - temp shadow/reserved temp object exists, otherwise inserts `0` and aborts; + unexpected durable reserved-name object or temp shadow/reserved temp object + exists, otherwise inserts `0` and aborts; - drops the guard on the success path; - keeps `ALTER ADD detail`, strict replacement-table creation, detail copy, and explicit main index/trigger recreation, all qualified to `main` @@ -149,8 +158,9 @@ files: say that the durable audit contract is `main.ward_audit`, exact stored `main.sqlite_master.sql` equality covers every declared table-level constraint, schema-qualified PRAGMAs are required for durable column/index - inspection, temp shadows/reserved temp objects fail closed, and no - whitespace-destroying normalization is allowed. + inspection, the exact durable namespace whitelist applies in every state, + temp shadows/reserved temp objects fail closed, and no whitespace-destroying + normalization is allowed. - Document the caller contract explicitly: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - initialize on `missing` only; @@ -171,7 +181,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): reject temporary schema shadows +fix(audit): reject durable namespace drift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index 5ab2234..ec5905a 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -14,13 +14,14 @@ schema contract. callers run before touching the durable `main.ward_audit` table. It returns one stable tag: -- `missing` — `main.ward_audit` does not exist, no reserved durable - `ward_audit` / `ward_audit_*` main-schema object already exists, and no temp +- `missing` — `main.ward_audit` does not exist, no unexpected durable + main-schema object named `ward_audit` or `ward_audit_*` exists, and no temp shadow/reserved temp object exists; initialize with `WARD_AUDIT_SCHEMA_SQL`. -- `legacy_v013` — exact v0.1.3 legacy fingerprint; run +- `legacy_v013` — exact v0.1.3 legacy fingerprint plus the durable namespace + whitelist; run `WARD_AUDIT_MIGRATION_V014_SQL`. -- `current_v014` — exact v0.1.4 current fingerprint; continue without schema - work. +- `current_v014` — exact v0.1.4 current fingerprint plus the durable namespace + whitelist; continue without schema work. - `unknown` — every other shape; fail closed and investigate manually. The fingerprint is exact and table-local to `main.ward_audit`: @@ -38,21 +39,25 @@ The fingerprint is exact and table-local to `main.ward_audit`: The full stored table definition covers every declared table-level constraint — the `event_type` CHECK list, extra `CHECK` clauses, `UNIQUE` clauses, and foreign-key clauses — so any extra or missing constraint, column, index, or -trigger classifies as `unknown`. Any temp-schema table/view/index/trigger whose +trigger classifies as `unknown`. Across **all** durable states, the reserved +main-schema namespace is whitelisted to exactly these `main.ward_audit` +objects: the table itself, indexes `ward_audit_event_idx` and +`ward_audit_familiar_idx`, and triggers +`ward_audit_append_only_update` / `ward_audit_append_only_delete`, each attached +to `main.ward_audit`. Every other main-schema table/view/index/trigger whose name is exactly `ward_audit` or begins with `ward_audit_` also classifies as -`unknown`, even when `main.ward_audit` is otherwise exact current or legacy; -that fail-closed rule stops TEMP shadows from being mistaken for healthy -durable state and blocks unqualified daemon writes from proceeding under a false -`current_v014`. When `main.ward_audit` itself is absent, any preexisting -main-schema object whose name is reserved under the same durable namespace also -classifies as `unknown`; that prevents schema init from silently skipping -append-only objects already attached to other tables. No whitespace-destroying -normalization is applied: only the exact stored SQL variants SQLite emits for -the shipped schemas are accepted. For `current_v014`, that means the fresh -`CREATE TABLE ward_audit (...)` form and the quoted -`CREATE TABLE "ward_audit" (...)` form produced by the legacy rename path. For -`legacy_v013`, the fingerprint intentionally includes the inline comments -preserved from the shipped v0.1.3 DDL. +`unknown`, including `ward_audit_new`, backup/shadow tables, and reserved-name +indexes/triggers attached elsewhere. Any temp-schema table/view/index/trigger +whose name is exactly `ward_audit` or begins with `ward_audit_` likewise +classifies as `unknown`, even when `main.ward_audit` is otherwise exact current +or legacy; that fail-closed rule stops TEMP shadows from being mistaken for +healthy durable state and blocks unqualified daemon writes from proceeding under +a false `current_v014`. No whitespace-destroying normalization is applied: only +the exact stored SQL variants SQLite emits for the shipped schemas are +accepted. For `current_v014`, that means the fresh `CREATE TABLE ward_audit +(...)` form and the quoted `CREATE TABLE "ward_audit" (...)` form produced by +the legacy rename path. For `legacy_v013`, the fingerprint intentionally +includes the inline comments preserved from the shipped v0.1.3 DDL. ## Initialization design @@ -70,11 +75,12 @@ therefore: 5. `COMMIT;` This makes fresh installs atomic, makes exact current reruns idempotent, and -fails closed for `legacy_v013`, `unknown`, any temp shadow/reserved temp -object, or any malformed/silently skipped install result. If either guard -errors, callers must explicitly `ROLLBACK` before continuing so SQLite removes -any uncommitted `main.ward_audit` table created on the failed path while -preserving unrelated durable/temp objects that caused the collision. +fails closed for `legacy_v013`, `unknown`, any unexpected durable reserved-name +object, any temp shadow/reserved temp object, or any malformed/silently skipped +install result. If either guard errors, callers must explicitly `ROLLBACK` +before continuing so SQLite removes any uncommitted `main.ward_audit` table +created on the failed path while preserving unrelated durable/temp objects that +caused the collision. ## Migration design @@ -88,17 +94,21 @@ The v0.1.4 repair remains a table-local migration. The SQL still: The change is the guardrail: immediately after `BEGIN;`, the migration creates a uniquely named TEMP guard table with `CHECK (ok = 1)` and inserts `1` only if -the exact `legacy_v013` durable fingerprint holds **and** no temp -shadow/reserved temp object exists. Any missing/current/partial/temp-shadowed -schema inserts `0` instead, aborting before `ALTER TABLE`. This makes the -migration fail closed even if a caller skips the classification query, and keeps -initialization and migration aligned on the same exact state contract. - -If a later step fails after `ALTER TABLE` (for example, `ward_audit_new` -already exists), callers must explicitly `ROLLBACK` the failed transaction -before continuing so SQLite restores the untouched legacy table. Ward SQL never -reads or writes database-wide `PRAGMA user_version`, so unrelated shared-store -version state is preserved. +the exact `legacy_v013` durable fingerprint holds **and** no unexpected durable +reserved-name object or temp shadow/reserved temp object exists. Any +missing/current/partial/shadowed schema inserts `0` instead, aborting before +`ALTER TABLE`. This makes the migration fail closed even if a caller skips the +classification query, and keeps initialization and migration aligned on the +same exact state contract. + +If a later step fails after `ALTER TABLE`, callers must explicitly `ROLLBACK` +the failed transaction before continuing so SQLite restores the untouched +legacy table. A preexisting `main.ward_audit_new` is now itself durable drift +and must fail at the guard before `ALTER TABLE`; rollback semantics for the +production migration's post-`ALTER` failure contract are instead validated with +a controlled duplicate `CREATE TABLE main.ward_audit_new` later in the same +transaction. Ward SQL never reads or writes database-wide `PRAGMA user_version`, +so unrelated shared-store version state is preserved. ## Tests @@ -106,7 +116,8 @@ Executable rusqlite tests cover: 1. schema-state classification for `missing`, exact `legacy_v013`, and exact `current_v014`, including absent-table reserved durable-namespace collisions, - reserved temp objects, and TEMP `ward_audit` shadows classifying as + exact current/legacy shapes with extra durable reserved-name view/table + objects, reserved temp objects, and TEMP `ward_audit` shadows classifying as `unknown`; 2. fresh empty-store initialization classifying `missing`, running `WARD_AUDIT_SCHEMA_SQL` atomically to `current_v014`, and enforcing @@ -133,7 +144,9 @@ Executable rusqlite tests cover: 10. successful exact-legacy upgrade landing in `current_v014`, with fresh and migrated current schemas both classifying `current_v014` while matching only their controlled exact stored SQL variants; and -11. post-`ALTER` failure rollback restoring the full legacy table state; and +11. a controlled post-`ALTER` SQL failure inside the transaction validating + SQLite rollback semantics for the production migration contract and + restoring the full legacy table state; and 12. schema-qualified PRAGMA syntax on bundled SQLite plus the reason for the fail-closed contract: unqualified inserts hit TEMP first while the contract remains `unknown`. @@ -144,5 +157,7 @@ This repair does not add daemon migration orchestration or change the `WardAuditRecord` wire shape. The daemon remains responsible for calling `WARD_AUDIT_SCHEMA_STATE_SQL`, choosing fresh initialization vs legacy migration, treating `main.ward_audit` as the only durable audit contract, -running `WARD_AUDIT_SCHEMA_SQL` only through the allowed -`missing`/`current_v014` contract, and failing closed on `unknown`. +accepting only the exact durable whitelist of `main.ward_audit` plus its two +indexes and two append-only triggers, running `WARD_AUDIT_SCHEMA_SQL` only +through the allowed `missing`/`current_v014` contract, and failing closed on +`unknown`. From 73f285a86019a40fa779c706572ac2e1c8b68572 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 01:57:50 -0500 Subject: [PATCH 23/25] fix(audit): serialize schema guard transactions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 34 ++- crates/coven-threads-core/Cargo.toml | 2 +- crates/coven-threads-core/src/audit.rs | 288 ++++++++++++++++-- ...2026-07-19-apply-audit-migration-repair.md | 48 +-- ...-19-apply-audit-migration-repair-design.md | 51 +++- 5 files changed, 348 insertions(+), 75 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e133d9c..492f47b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,22 +32,31 @@ All notable changes to `coven-threads-core` are documented here. variants and the shipped v0.1.3 table SQL are accepted; no whitespace-destroying normalization is applied. - `WARD_AUDIT_SCHEMA_SQL` — atomic, self-guarding schema initialization for the - exact `current_v014` fingerprint. It begins a transaction, permits only - `missing` or exact `current_v014` before any mutation, uses idempotent `IF NOT - EXISTS` DDL for daemon compatibility, targets durable schema objects in - `main` wherever SQLite syntax permits, then requires exact `current_v014` - before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, any - unexpected durable reserved object, and any temp shadow/reserved temp object - fail closed; callers must `ROLLBACK` after any init error. + exact `current_v014` fingerprint. It now begins with `BEGIN IMMEDIATE`, so + the main-database write reservation is acquired before any guard read or + classification. That makes concurrent initializers serialize cleanly: the + second caller waits, re-runs the guard against the winner's committed schema, + and idempotently sees exact `current_v014` instead of racing into + `sqlite_master` lock errors. The SQL still permits only `missing` or exact + `current_v014` before any mutation, uses idempotent `IF NOT EXISTS` DDL for + daemon compatibility, targets durable schema objects in `main` wherever + SQLite syntax permits, then requires exact `current_v014` before `COMMIT`. + Exact `legacy_v013`, every drifted `unknown` shape, any unexpected durable + reserved object, and any temp shadow/reserved temp object fail closed; + callers must `ROLLBACK` after any init error. - `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving rebuild of `main.ward_audit` guarded by the exact `legacy_v013` fingerprint plus durable-whitelist and temp-shadow rejection. The daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, migrate only `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The - migration independently re-checks `legacy_v013` before any `ALTER`, using the - same exact stored-table + column/index/trigger predicate plus durable - namespace whitelist and temp-shadow rejection, then copies rows into the - replacement table without discarding evidence. + migration independently re-checks `legacy_v013` before any `ALTER`, now + inside `BEGIN IMMEDIATE` so concurrent migrators serialize before the guard + read. After the first caller upgrades, a second caller waits, re-runs the + guard against exact current, and fails only at the legacy guard instead of + racing into `sqlite_master` lock errors. The SQL otherwise uses the same + exact stored-table + column/index/trigger predicate plus durable namespace + whitelist and temp-shadow rejection, then copies rows into the replacement + table without discarding evidence. - Exhaustiveness test `schema_names_all_event_tags` extended to cover `ApplyAudit`. - New tests: `for_apply_produces_correct_shape`, `for_apply_roundtrips_json`, @@ -74,6 +83,7 @@ All notable changes to `coven-threads-core` are documented here. `current_schema_with_extra_reserved_main_view_or_table_is_unknown_and_schema_sql_preserves_state`, `current_schema_with_altered_trigger_error_literal_is_unknown`, `current_schema_with_altered_trigger_body_is_unknown`, + `schema_and_migration_sql_use_begin_immediate`, `absent_ward_audit_with_reserved_index_collision_is_unknown_and_schema_sql_preserves_other_objects`, `absent_ward_audit_with_reserved_trigger_collision_is_unknown_and_schema_sql_preserves_other_objects`, `unknown_partial_current_schema_rejects_schema_sql_and_preserves_state`, @@ -81,6 +91,8 @@ All notable changes to `coven-threads-core` are documented here. `legacy_schema_with_preexisting_main_ward_audit_new_is_unknown_and_guard_rejects_before_alter`, `legacy_schema_upgrades_and_preserves_append_only_behavior`, `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, + `concurrent_schema_initialization_serializes_without_locked_errors`, + `concurrent_legacy_migration_waits_then_rejects_current_at_guard`, `sqlite_post_alter_failure_rollback_restores_legacy_schema_for_production_migration_contract`, `schema_qualified_table_valued_pragmas_resolve_main_and_temp_separately`, `current_main_with_temp_shadow_is_unknown_and_guards_preserve_main_and_temp_rows`, diff --git a/crates/coven-threads-core/Cargo.toml b/crates/coven-threads-core/Cargo.toml index 94c465c..30e2582 100644 --- a/crates/coven-threads-core/Cargo.toml +++ b/crates/coven-threads-core/Cargo.toml @@ -21,4 +21,4 @@ sha2.workspace = true time.workspace = true [dev-dependencies] -rusqlite = { version = "0.31", features = ["bundled"] } +rusqlite = { version = "0.31", features = ["bundled", "hooks"] } diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index ef139fe..145b737 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -21,13 +21,16 @@ //! ## Schema shape and migration gating //! //! `WARD_AUDIT_SCHEMA_SQL` initializes or verifies the exact v0.1.4 -//! `main.ward_audit` shape in one transaction, without mutating database-wide -//! `PRAGMA user_version`. It is safe for the daemon to execute unconditionally -//! on every store open: a pre-install guard permits only `missing` or exact -//! `current_v014`, the durable DDL runs explicitly in `main` with `IF NOT -//! EXISTS` compatibility, and a post-install guard requires exact -//! `current_v014` before `COMMIT`. If either guard fails, callers must -//! explicitly `ROLLBACK` before continuing. +//! `main.ward_audit` shape in one `BEGIN IMMEDIATE` transaction, without +//! mutating database-wide `PRAGMA user_version`. It is safe for the daemon to +//! execute unconditionally on every store open: a pre-install guard permits +//! only `missing` or exact `current_v014`, the durable DDL runs explicitly in +//! `main` with `IF NOT EXISTS` compatibility, and a post-install guard requires +//! exact `current_v014` before `COMMIT`. The IMMEDIATE reservation means +//! concurrent initializers serialize before any guard/classification read, so a +//! second caller waits, re-runs the guard against the winner's committed +//! schema, and then idempotently sees `current_v014`. If either guard fails, +//! callers must explicitly `ROLLBACK` before continuing. //! `WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable, table-local fingerprint query //! that returns one of four stable tags: //! - `missing` — `main.ward_audit` does not exist, no unexpected durable @@ -59,10 +62,14 @@ //! comments preserved from the shipped v0.1.3 DDL. //! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` //! fingerprint with no unexpected durable reserved-namespace object and no temp -//! shadow. It independently re-checks that fingerprint inside the transaction -//! before any `ALTER`, then adds `detail` and rebuilds `main.ward_audit` so the -//! current CHECK set is installed and every existing row is preserved. If a -//! later migration step fails after +//! shadow. It independently re-checks that fingerprint inside one +//! `BEGIN IMMEDIATE` transaction before any `ALTER`, then adds `detail` and +//! rebuilds `main.ward_audit` so the current CHECK set is installed and every +//! existing row is preserved. The IMMEDIATE reservation means concurrent +//! migrators serialize before any legacy-guard read: the winner upgrades first, +//! and a second caller waits, reclassifies the durable table as current when +//! the guard re-runs, then fails closed at the legacy guard without a +//! `sqlite_master` lock race. If a later migration step fails after //! `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must explicitly roll //! back the failed transaction before continuing so SQLite restores the //! untouched legacy table. @@ -683,7 +690,8 @@ FROM ward_audit_shape; /// triggers (RFC-0001 §5.6: entries MUST NOT be deleted or modified). /// SQLite cannot `ALTER` a CHECK constraint on an existing table, so this /// transaction: -/// 1. re-checks the exact legacy fingerprint in SQL before any mutation, +/// 1. reserves the main database up front with `BEGIN IMMEDIATE`, then +/// re-checks the exact legacy fingerprint in SQL before any mutation, /// including exact stored `main.sqlite_master.sql` equality plus main /// column/index/trigger fingerprints, the durable reserved-namespace /// whitelist, and temp-shadow rejection; @@ -700,13 +708,17 @@ FROM ward_audit_shape; /// independently guards the same `legacy_v013` fingerprint, durable /// reserved-namespace whitelist, and temp-shadow rejection so callers cannot /// mutate a partial, already-current, or shadowed schema by skipping -/// classification. If a later step fails after `ALTER TABLE main.ward_audit ADD -/// COLUMN detail`, callers must `ROLLBACK` the failed transaction before -/// continuing so SQLite restores the untouched legacy table. This SQL does not -/// read or write database-wide `PRAGMA user_version`. +/// classification. Because the transaction begins IMMEDIATELY, concurrent +/// migrators serialize before the guard read; after the winner commits, a +/// second caller re-runs the guard against the now-current schema and fails +/// closed there rather than racing into a `sqlite_master` lock. If a later step +/// fails after `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must +/// `ROLLBACK` the failed transaction before continuing so SQLite restores the +/// untouched legacy table. This SQL does not read or write database-wide +/// `PRAGMA user_version`. pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = concat!( r#" -BEGIN; +BEGIN IMMEDIATE; CREATE TEMP TABLE coven_threads_ward_audit_migration_guard ( ok INTEGER NOT NULL CHECK (ok = 1) @@ -832,20 +844,23 @@ END; /// DDL for the durable `main.ward_audit` table inside `coven.sqlite3` (§3.4). /// -/// See module docs for the full schema-state contract. This transaction is -/// safe to run unconditionally on every store open: it permits only exact -/// `missing` or `current_v014` before any mutation, uses idempotent `IF NOT -/// EXISTS` DDL for daemon compatibility, then requires exact `current_v014` -/// before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, every -/// unexpected durable reserved-namespace object, and every temp -/// shadow/reserved temp object fail closed. Durable schema objects are -/// explicitly created in `main`; the temp guard tables live in `temp` under -/// unique non-reserved names. If this SQL errors, callers must explicitly -/// `ROLLBACK` before continuing so SQLite discards any uncommitted work. This -/// DDL never mutates database-wide `PRAGMA user_version`. +/// See module docs for the full schema-state contract. This `BEGIN IMMEDIATE` +/// transaction is safe to run unconditionally on every store open: it permits +/// only exact `missing` or `current_v014` before any mutation, uses idempotent +/// `IF NOT EXISTS` DDL for daemon compatibility, then requires exact +/// `current_v014` before `COMMIT`. Exact `legacy_v013`, every drifted +/// `unknown` shape, every unexpected durable reserved-namespace object, and +/// every temp shadow/reserved temp object fail closed. The IMMEDIATE +/// reservation serializes concurrent initializers before any guard read so the +/// loser waits, re-runs the guard against committed state, and sees exact +/// `current_v014` rather than racing into `sqlite_master` lock errors. Durable +/// schema objects are explicitly created in `main`; the temp guard tables live +/// in `temp` under unique non-reserved names. If this SQL errors, callers must +/// explicitly `ROLLBACK` before continuing so SQLite discards any uncommitted +/// work. This DDL never mutates database-wide `PRAGMA user_version`. pub const WARD_AUDIT_SCHEMA_SQL: &str = concat!( r#" -BEGIN; +BEGIN IMMEDIATE; CREATE TEMP TABLE coven_threads_ward_audit_schema_pre_guard ( ok INTEGER NOT NULL CHECK (ok = 1) @@ -894,7 +909,16 @@ mod tests { use super::*; use crate::ids::{SurfaceId, WriterId}; use rusqlite::{params, Connection}; - use std::collections::BTreeSet; + use std::{ + collections::BTreeSet, + fs, + io::ErrorKind, + path::{Path, PathBuf}, + sync::{Arc, Barrier}, + thread, + time::Duration, + }; + use uuid::Uuid; const FIXED_SUBMITTED_AT: &str = "2026-07-19T00:00:00.000Z"; const FIXED_DECIDED_AT: &str = "2026-07-19T00:01:00.000Z"; @@ -903,6 +927,7 @@ mod tests { const FIXED_DIFF_HASH: [u8; 32] = *b"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"; const FIXED_PREV_DETAIL: &str = r#"{"prev_sha256":"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc","bytes_written":42}"#; const FIXED_FILES_TOUCHED: &str = r#"["SOUL.md"]"#; + const CONCURRENT_DB_RUNS: usize = 3; /// Exact shipped v0.1.3 `ward_audit` DDL from `origin/main` / the PR base. /// Keep the inline comments: SQLite preserves them in `sqlite_master.sql`, @@ -983,6 +1008,97 @@ END; conn.pragma_update(None, "user_version", version).unwrap(); } + fn concurrent_db_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target/ward-audit-concurrency") + } + + fn sqlite_artifact_paths(path: &Path) -> Vec { + ["", "-journal", "-wal", "-shm"] + .into_iter() + .map(|suffix| { + let mut candidate = path.as_os_str().to_os_string(); + candidate.push(suffix); + PathBuf::from(candidate) + }) + .collect() + } + + fn cleanup_sqlite_artifacts(path: &Path) { + for candidate in sqlite_artifact_paths(path) { + match fs::remove_file(&candidate) { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => panic!("failed to remove {}: {err}", candidate.display()), + } + } + } + + fn assert_sqlite_artifacts_absent(path: &Path) { + for candidate in sqlite_artifact_paths(path) { + assert!( + !candidate.exists(), + "expected {} to be cleaned up", + candidate.display() + ); + } + } + + struct ScratchDbPath { + path: PathBuf, + } + + impl ScratchDbPath { + fn new(prefix: &str) -> Self { + let dir = concurrent_db_dir(); + fs::create_dir_all(&dir).unwrap(); + let path = dir.join(format!("{prefix}-{}.sqlite3", Uuid::new_v4())); + cleanup_sqlite_artifacts(&path); + Self { path } + } + + fn path(&self) -> &Path { + &self.path + } + } + + impl Drop for ScratchDbPath { + fn drop(&mut self) { + for candidate in sqlite_artifact_paths(&self.path) { + let _ = fs::remove_file(candidate); + } + } + } + + fn open_file_backed_connection(path: &Path) -> Connection { + let conn = Connection::open(path).unwrap(); + conn.busy_timeout(Duration::from_secs(5)).unwrap(); + conn + } + + fn run_concurrent_sql(path: &Path, sql: &'static str) -> Vec> { + let barrier = Arc::new(Barrier::new(2)); + let handles: Vec<_> = (0..2) + .map(|_| { + let barrier = Arc::clone(&barrier); + let path = path.to_path_buf(); + thread::spawn(move || { + let conn = open_file_backed_connection(&path); + conn.commit_hook(Some(|| { + thread::sleep(Duration::from_millis(150)); + false + })); + barrier.wait(); + conn.execute_batch(sql).map_err(|err| err.to_string()) + }) + }) + .collect(); + + handles + .into_iter() + .map(|handle| handle.join().unwrap()) + .collect() + } + fn ward_audit_schema_state(conn: &Connection) -> String { conn.query_row(WARD_AUDIT_SCHEMA_STATE_SQL, [], |row| row.get(0)) .unwrap() @@ -1678,6 +1794,19 @@ END; assert!(WARD_AUDIT_SCHEMA_SQL.contains("append-only (RFC-0001 §5.6)")); } + #[test] + fn schema_and_migration_sql_use_begin_immediate() { + for (label, sql) in [ + ("schema", WARD_AUDIT_SCHEMA_SQL), + ("migration", WARD_AUDIT_MIGRATION_V014_SQL), + ] { + assert!( + sql.trim_start().starts_with("BEGIN IMMEDIATE;"), + "{label} SQL must reserve the main database before guard reads" + ); + } + } + #[test] fn for_apply_produces_correct_shape() { let now = OffsetDateTime::now_utc(); @@ -2747,6 +2876,105 @@ END; assert_eq!(user_version(&conn), 37); } + #[test] + fn concurrent_schema_initialization_serializes_without_locked_errors() { + for attempt in 0..CONCURRENT_DB_RUNS { + let path = { + let db = ScratchDbPath::new("concurrent-schema-init"); + let path = db.path().to_path_buf(); + + { + let setup = open_file_backed_connection(db.path()); + set_user_version(&setup, 41); + } + + let outcomes = run_concurrent_sql(db.path(), WARD_AUDIT_SCHEMA_SQL); + assert!( + outcomes.iter().all(|result| result.is_ok()), + "attempt {attempt} concurrent init outcomes: {outcomes:?}" + ); + + { + let final_conn = open_file_backed_connection(db.path()); + assert_eq!(user_version(&final_conn), 41); + assert_schema_state(&final_conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!( + explicit_index_names(&final_conn), + expected_explicit_index_names() + ); + assert_eq!(trigger_names(&final_conn), expected_trigger_names()); + } + + path + }; + + assert_sqlite_artifacts_absent(&path); + } + } + + #[test] + fn concurrent_legacy_migration_waits_then_rejects_current_at_guard() { + for attempt in 0..CONCURRENT_DB_RUNS { + let path = { + let db = ScratchDbPath::new("concurrent-legacy-migration"); + let path = db.path().to_path_buf(); + + let (row_id, before) = { + let setup = open_file_backed_connection(db.path()); + set_user_version(&setup, 37); + setup.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_legacy_ward_updated_row(&setup); + let before = load_legacy_audit_row(&setup, row_id); + (row_id, before) + }; + + let outcomes = run_concurrent_sql(db.path(), WARD_AUDIT_MIGRATION_V014_SQL); + let success_count = outcomes.iter().filter(|result| result.is_ok()).count(); + let errors: Vec<_> = outcomes + .iter() + .filter_map(|result| result.as_ref().err()) + .collect(); + assert_eq!( + success_count, 1, + "attempt {attempt} concurrent migration outcomes: {outcomes:?}" + ); + assert_eq!( + errors.len(), + 1, + "attempt {attempt} concurrent migration outcomes: {outcomes:?}" + ); + let error = errors[0].to_lowercase(); + assert!( + error.contains("check constraint failed"), + "attempt {attempt} unexpected migration error: {}", + errors[0] + ); + assert!( + !error.contains("locked"), + "attempt {attempt} migration must wait then fail at the guard, not lock: {}", + errors[0] + ); + + { + let final_conn = open_file_backed_connection(db.path()); + assert_eq!(user_version(&final_conn), 37); + assert_schema_state(&final_conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_eq!(ward_audit_row_count(&final_conn, "main"), 1); + assert_eq!(load_audit_row(&final_conn, row_id), before); + assert_eq!( + explicit_index_names(&final_conn), + expected_explicit_index_names() + ); + assert_eq!(trigger_names(&final_conn), expected_trigger_names()); + } + + path + }; + + assert_sqlite_artifacts_absent(&path); + } + } + #[test] fn unqualified_insert_targets_temp_shadow_while_schema_state_stays_unknown() { let conn = Connection::open_in_memory().unwrap(); diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 75d6270..1be8ccd 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -14,27 +14,29 @@ Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on `missing` / `legacy_v013` / `current_v014` / `unknown`, reserve `missing` for an absent `main.ward_audit` with no unexpected durable reserved-name object and no temp shadow/reserved temp object, require the same exact durable whitelist -for `legacy_v013` and `current_v014`, wrap `WARD_AUDIT_SCHEMA_SQL` in atomic -pre/post guards that allow only `missing`/`current_v014`, and embed the same -exact `legacy_v013` + no-unexpected-durable-object + no-temp-shadow predicate -inside `WARD_AUDIT_MIGRATION_V014_SQL` before any `ALTER TABLE`. Durable -DDL/DML stays explicitly qualified to `main` wherever SQLite permits it, and -the init/migration paths never write database-wide `user_version`. +for `legacy_v013` and `current_v014`, wrap both `WARD_AUDIT_SCHEMA_SQL` and +`WARD_AUDIT_MIGRATION_V014_SQL` in `BEGIN IMMEDIATE` transactions so guard +reads happen under the main-database write reservation, let concurrent init +serialize cleanly to `current_v014`, and make a concurrent second migration +wait, then reclassify current when the legacy guard re-runs. Durable DDL/DML +stays explicitly qualified to `main` wherever SQLite permits it, and the +init/migration paths never write database-wide `user_version`. -**Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` for in-memory -tests. +**Tech Stack:** Rust 2021, SQLite, Cargo, bundled `rusqlite` (hooks-enabled in +dev-dependencies) for in-memory and file-backed concurrency tests. --- ## Final File Map -For this follow-up fingerprint repair, the diff stays scoped to these four +For this follow-up fingerprint repair, the diff stays scoped to these five files: 1. `CHANGELOG.md` -2. `crates/coven-threads-core/src/audit.rs` -3. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` -4. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` +2. `crates/coven-threads-core/Cargo.toml` +3. `crates/coven-threads-core/src/audit.rs` +4. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` +5. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` --- @@ -61,6 +63,10 @@ files: `current_v014`, and append-only UPDATE/DELETE still abort; - exact current schema → rerunning `WARD_AUDIT_SCHEMA_SQL` is idempotent and preserves rows/objects; + - file-backed two-connection concurrent init with `busy_timeout` and repeated + synchronized starts → both callers complete successfully because + `BEGIN IMMEDIATE` serializes before guard reads, with no locked/schema-locked + errors and exact final `current_v014`; - exact legacy schema → `WARD_AUDIT_SCHEMA_SQL` rejects, requires explicit `ROLLBACK`, and preserves state/data; - exact current `main.ward_audit` + TEMP shadow clone → init rejects, @@ -92,6 +98,10 @@ files: the new guard and always `ROLLBACK` with `unwrap()`. - Add legacy TEMP-shadow migration coverage proving the guard rejects before any main/temp mutation and rollback preserves the exact legacy durable table. +- Add file-backed two-connection concurrent legacy migration coverage with + `busy_timeout` and repeated synchronized starts proving one caller upgrades, + the other waits, then fails only at the legacy guard after seeing exact + current (never with a lock error), while preserving the legacy row. - Add reason-demo coverage proving an unqualified `INSERT INTO ward_audit ...` lands in TEMP while the durable contract remains `unknown`. - Add a controlled post-`ALTER` rollback coverage path that starts from exact @@ -128,7 +138,8 @@ files: - Re-export the state query and tags from `crates/coven-threads-core/src/lib.rs` for root-API callers. - Update `WARD_AUDIT_SCHEMA_SQL` so it: - - starts a transaction; + - starts with `BEGIN IMMEDIATE` so the main-database write reservation is + acquired before any guard read/classification; - creates a uniquely named TEMP pre-install guard with `CHECK (ok = 1)`; - inserts `1` only when the shared schema-state expression returns `missing` or `current_v014`, otherwise inserts `0` and aborts before any @@ -141,7 +152,8 @@ files: - commits on success; and - requires callers to `ROLLBACK` after any init error. - Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: - - starts a transaction; + - starts with `BEGIN IMMEDIATE` so concurrent migrators serialize before the + legacy guard read; - creates a uniquely named TEMP guard table with `CHECK (ok = 1)`; - inserts `1` only when the full exact legacy durable predicate holds and no unexpected durable reserved-name object or temp shadow/reserved temp object @@ -159,8 +171,10 @@ files: `main.sqlite_master.sql` equality covers every declared table-level constraint, schema-qualified PRAGMAs are required for durable column/index inspection, the exact durable namespace whitelist applies in every state, - temp shadows/reserved temp objects fail closed, and no whitespace-destroying - normalization is allowed. + temp shadows/reserved temp objects fail closed, guard reads happen under an + IMMEDIATE transaction, concurrent init serializes, concurrent second + migration callers must reclassify after the legacy guard rejects current, and + no whitespace-destroying normalization is allowed. - Document the caller contract explicitly: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - initialize on `missing` only; @@ -181,7 +195,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): reject durable namespace drift +fix(audit): serialize schema guard transactions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index ec5905a..f9019dd 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -65,7 +65,8 @@ includes the inline comments preserved from the shipped v0.1.3 DDL. the daemon executes it unconditionally on every store open. The safe contract is therefore: -1. `BEGIN;` +1. `BEGIN IMMEDIATE;` to reserve the main-database write slot before any guard + read/classification; 2. run a uniquely named TEMP pre-install guard that reuses the exact schema-state CTEs and permits only `missing` or exact `current_v014`; 3. create the durable `main.ward_audit` table, explicit main indexes, and @@ -75,12 +76,15 @@ therefore: 5. `COMMIT;` This makes fresh installs atomic, makes exact current reruns idempotent, and -fails closed for `legacy_v013`, `unknown`, any unexpected durable reserved-name -object, any temp shadow/reserved temp object, or any malformed/silently skipped -install result. If either guard errors, callers must explicitly `ROLLBACK` -before continuing so SQLite removes any uncommitted `main.ward_audit` table -created on the failed path while preserving unrelated durable/temp objects that -caused the collision. +serializes concurrent initializers before they read `main.sqlite_master`: the +winner reserves the main-database write slot first, and the second caller waits +until commit, re-runs the guard against the committed schema, and then +idempotently sees exact `current_v014`. The path still fails closed for +`legacy_v013`, `unknown`, any unexpected durable reserved-name object, any temp +shadow/reserved temp object, or any malformed/silently skipped install result. +If either guard errors, callers must explicitly `ROLLBACK` before continuing so +SQLite removes any uncommitted `main.ward_audit` table created on the failed +path while preserving unrelated durable/temp objects that caused the collision. ## Migration design @@ -92,14 +96,18 @@ The v0.1.4 repair remains a table-local migration. The SQL still: 4. swaps tables; and 5. re-creates the required explicit main indexes and append-only main triggers. -The change is the guardrail: immediately after `BEGIN;`, the migration creates -a uniquely named TEMP guard table with `CHECK (ok = 1)` and inserts `1` only if -the exact `legacy_v013` durable fingerprint holds **and** no unexpected durable -reserved-name object or temp shadow/reserved temp object exists. Any -missing/current/partial/shadowed schema inserts `0` instead, aborting before -`ALTER TABLE`. This makes the migration fail closed even if a caller skips the -classification query, and keeps initialization and migration aligned on the -same exact state contract. +The change is the guardrail: immediately after `BEGIN IMMEDIATE;`, the +migration creates a uniquely named TEMP guard table with `CHECK (ok = 1)` and +inserts `1` only if the exact `legacy_v013` durable fingerprint holds **and** +no unexpected durable reserved-name object or temp shadow/reserved temp object +exists. Any missing/current/partial/shadowed schema inserts `0` instead, +aborting before `ALTER TABLE`. That IMMEDIATE reservation also serializes +concurrent migrators before the guard read: the winner upgrades first, and a +second caller waits, re-runs the guard against the now-current durable table, +and fails closed there instead of racing into a `sqlite_master` lock. This +makes the migration fail closed even if a caller skips the classification query, +and keeps initialization and migration aligned on the same exact state +contract. If a later step fails after `ALTER TABLE`, callers must explicitly `ROLLBACK` the failed transaction before continuing so SQLite restores the untouched @@ -150,6 +158,15 @@ Executable rusqlite tests cover: 12. schema-qualified PRAGMA syntax on bundled SQLite plus the reason for the fail-closed contract: unqualified inserts hit TEMP first while the contract remains `unknown`. +13. file-backed multi-connection initialization with two simultaneous callers, + `busy_timeout`, and repeated runs proving both `WARD_AUDIT_SCHEMA_SQL` + executions complete without locked/schema-locked errors because + `BEGIN IMMEDIATE` serializes them before guard reads; and +14. file-backed multi-connection legacy migration with two simultaneous callers, + `busy_timeout`, and repeated runs proving one migration succeeds while the + second waits, re-runs the guard against exact `current_v014`, and fails only + at the legacy guard (never with a lock error), while preserving the legacy + row data. ## Scope @@ -160,4 +177,6 @@ migration, treating `main.ward_audit` as the only durable audit contract, accepting only the exact durable whitelist of `main.ward_audit` plus its two indexes and two append-only triggers, running `WARD_AUDIT_SCHEMA_SQL` only through the allowed `missing`/`current_v014` contract, and failing closed on -`unknown`. +`unknown`. Concurrent callers must also treat a migration guard rejection after +waiting as a signal to reclassify: another writer may already have serialized +the schema to exact `current_v014`. From f5bfe40066ec628b1733738ff372e5f4ceacd1e9 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 02:45:36 -0500 Subject: [PATCH 24/25] fix(audit): verify migration postcondition Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 14 +- crates/coven-threads-core/src/audit.rs | 147 ++++++++++++++++-- docs/architecture.md | 2 +- ...2026-07-19-apply-audit-migration-repair.md | 14 +- ...-19-apply-audit-migration-repair-design.md | 49 ++++-- 5 files changed, 188 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 492f47b..17f2fbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,9 +51,13 @@ All notable changes to `coven-threads-core` are documented here. `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The migration independently re-checks `legacy_v013` before any `ALTER`, now inside `BEGIN IMMEDIATE` so concurrent migrators serialize before the guard - read. After the first caller upgrades, a second caller waits, re-runs the - guard against exact current, and fails only at the legacy guard instead of - racing into `sqlite_master` lock errors. The SQL otherwise uses the same + read. After the rebuild, a distinct TEMP postcondition guard reruns the same + shared schema-state CTE/predicates and requires exact `current_v014` before + `COMMIT`, so a caller cannot durably commit a self-unknown rebuilt schema if + unexpected `main.ward_audit` / `main.ward_audit_*` drift appears mid- + transaction. After the first caller upgrades, a second caller waits, re-runs + the guard against exact current, and fails only at the legacy guard instead + of racing into `sqlite_master` lock errors. The SQL otherwise uses the same exact stored-table + column/index/trigger predicate plus durable namespace whitelist and temp-shadow rejection, then copies rows into the replacement table without discarding evidence. @@ -84,12 +88,14 @@ All notable changes to `coven-threads-core` are documented here. `current_schema_with_altered_trigger_error_literal_is_unknown`, `current_schema_with_altered_trigger_body_is_unknown`, `schema_and_migration_sql_use_begin_immediate`, + `migration_sql_uses_distinct_pre_and_post_guards_before_commit`, `absent_ward_audit_with_reserved_index_collision_is_unknown_and_schema_sql_preserves_other_objects`, `absent_ward_audit_with_reserved_trigger_collision_is_unknown_and_schema_sql_preserves_other_objects`, `unknown_partial_current_schema_rejects_schema_sql_and_preserves_state`, `migration_rejects_current_schema_rows_with_detail_and_preserves_state`, `legacy_schema_with_preexisting_main_ward_audit_new_is_unknown_and_guard_rejects_before_alter`, - `legacy_schema_upgrades_and_preserves_append_only_behavior`, + `legacy_schema_upgrade_passes_post_guard_and_preserves_append_only_behavior`, + `post_guard_rejects_durable_drift_and_rollback_restores_exact_legacy_state`, `rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows`, `concurrent_schema_initialization_serializes_without_locked_errors`, `concurrent_legacy_migration_waits_then_rejects_current_at_guard`, diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index 145b737..c4c8f5c 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -65,11 +65,15 @@ //! shadow. It independently re-checks that fingerprint inside one //! `BEGIN IMMEDIATE` transaction before any `ALTER`, then adds `detail` and //! rebuilds `main.ward_audit` so the current CHECK set is installed and every -//! existing row is preserved. The IMMEDIATE reservation means concurrent -//! migrators serialize before any legacy-guard read: the winner upgrades first, -//! and a second caller waits, reclassifies the durable table as current when -//! the guard re-runs, then fails closed at the legacy guard without a -//! `sqlite_master` lock race. If a later migration step fails after +//! existing row is preserved. Before `COMMIT`, a distinct TEMP postcondition +//! guard reruns the shared schema-state CTE/predicates and requires exact +//! `current_v014`, so callers cannot durably commit a rebuilt-but-self-unknown +//! namespace if extra `ward_audit` / `ward_audit_*` objects appear mid-transaction. +//! The IMMEDIATE reservation means concurrent migrators serialize before any +//! legacy-guard read: the winner upgrades first, and a second caller waits, +//! reclassifies the durable table as current when the guard re-runs, then fails +//! closed at the legacy guard without a `sqlite_master` lock race. If the +//! postcondition guard or any later migration step fails after //! `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must explicitly roll //! back the failed transaction before continuing so SQLite restores the //! untouched legacy table. @@ -700,18 +704,22 @@ FROM ward_audit_shape; /// 3. creates `main.ward_audit_new` with the updated CHECK; /// 4. copies every existing row, preserving `detail`; /// 5. swaps the tables; and -/// 6. re-creates the exact explicit main indexes and append-only main triggers. +/// 6. re-creates the exact explicit main indexes and append-only main triggers; +/// and +/// 7. re-runs the shared schema-state expression in a distinct TEMP +/// postcondition guard and requires exact `current_v014` before `COMMIT`. /// /// Callers should still branch on [`WARD_AUDIT_SCHEMA_STATE_SQL`] first: /// initialize when the state is `missing`, migrate only `legacy_v013`, /// continue on `current_v014`, and fail closed on `unknown`. This migration /// independently guards the same `legacy_v013` fingerprint, durable -/// reserved-namespace whitelist, and temp-shadow rejection so callers cannot -/// mutate a partial, already-current, or shadowed schema by skipping -/// classification. Because the transaction begins IMMEDIATELY, concurrent -/// migrators serialize before the guard read; after the winner commits, a -/// second caller re-runs the guard against the now-current schema and fails -/// closed there rather than racing into a `sqlite_master` lock. If a later step +/// reserved-namespace whitelist, temp-shadow rejection, and exact-current +/// postcondition so callers cannot mutate a partial, already-current, +/// shadowed, or rebuilt-but-drifted schema by skipping classification. Because +/// the transaction begins IMMEDIATELY, concurrent migrators serialize before +/// the guard read; after the winner commits, a second caller re-runs the guard +/// against the now-current schema and fails closed there rather than racing +/// into a `sqlite_master` lock. If the postcondition guard or a later step /// fails after `ALTER TABLE main.ward_audit ADD COLUMN detail`, callers must /// `ROLLBACK` the failed transaction before continuing so SQLite restores the /// untouched legacy table. This SQL does not read or write database-wide @@ -792,6 +800,24 @@ BEGIN SELECT RAISE(ABORT, 'ward_audit is append-only (RFC-0001 §5.6)'); END; +CREATE TEMP TABLE coven_threads_ward_audit_migration_post_guard ( + ok INTEGER NOT NULL CHECK (ok = 1) +); + +INSERT INTO coven_threads_ward_audit_migration_post_guard (ok) +"#, + ward_audit_schema_state_ctes_sql!(), + r#" +SELECT CASE + WHEN ("#, + ward_audit_schema_state_case_sql!(), + r#") = 'current_v014' THEN 1 + ELSE 0 +END +FROM ward_audit_shape; + +DROP TABLE coven_threads_ward_audit_migration_post_guard; + COMMIT; "#, ); @@ -1230,6 +1256,24 @@ END; .unwrap(); } + fn inject_sql_before_anchor(base_sql: &str, anchor: &str, injection_sql: &str) -> String { + let replacement = format!("{injection_sql}\n\n{anchor}"); + let updated = base_sql.replacen(anchor, &replacement, 1); + assert_ne!( + updated, base_sql, + "expected to inject before anchor {anchor}" + ); + updated + } + + fn migration_sql_with_durable_drift_before_post_guard(drift_sql: &str) -> String { + inject_sql_before_anchor( + WARD_AUDIT_MIGRATION_V014_SQL, + "CREATE TEMP TABLE coven_threads_ward_audit_migration_post_guard (", + drift_sql, + ) + } + fn legacy_schema_with_extra_table_constraint(constraint_sql: &str) -> String { schema_sql_with_extra_table_constraint(LEGACY_WARD_AUDIT_SCHEMA_SQL, constraint_sql) } @@ -1807,6 +1851,35 @@ END; } } + #[test] + fn migration_sql_uses_distinct_pre_and_post_guards_before_commit() { + let pre_guard_offset = WARD_AUDIT_MIGRATION_V014_SQL + .find("CREATE TEMP TABLE coven_threads_ward_audit_migration_guard (") + .expect("migration SQL must define a precondition guard"); + let post_guard_offset = WARD_AUDIT_MIGRATION_V014_SQL + .find("CREATE TEMP TABLE coven_threads_ward_audit_migration_post_guard (") + .expect("migration SQL must define a postcondition guard"); + let post_guard_drop_offset = WARD_AUDIT_MIGRATION_V014_SQL + .find("DROP TABLE coven_threads_ward_audit_migration_post_guard;") + .expect("migration SQL must drop the postcondition guard before commit"); + let commit_offset = WARD_AUDIT_MIGRATION_V014_SQL + .rfind("COMMIT;") + .expect("migration SQL must commit on success"); + + assert!( + pre_guard_offset < post_guard_offset, + "postcondition guard must run after the legacy precondition guard" + ); + assert!( + post_guard_offset < post_guard_drop_offset, + "postcondition guard must be dropped on the success path" + ); + assert!( + post_guard_drop_offset < commit_offset, + "postcondition guard must run before COMMIT" + ); + } + #[test] fn for_apply_produces_correct_shape() { let now = OffsetDateTime::now_utc(); @@ -2792,7 +2865,7 @@ END; } #[test] - fn legacy_schema_upgrades_and_preserves_append_only_behavior() { + fn legacy_schema_upgrade_passes_post_guard_and_preserves_append_only_behavior() { let conn = Connection::open_in_memory().unwrap(); set_user_version(&conn, 37); conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); @@ -2846,6 +2919,54 @@ END; assert!(delete_err.to_string().contains("append-only")); } + #[test] + fn post_guard_rejects_durable_drift_and_rollback_restores_exact_legacy_state() { + let conn = Connection::open_in_memory().unwrap(); + set_user_version(&conn, 37); + conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); + let row_id = insert_legacy_ward_updated_row(&conn); + let before = load_legacy_audit_row(&conn, row_id); + + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + + let drift_sql = migration_sql_with_durable_drift_before_post_guard( + "CREATE VIEW main.ward_audit_history AS SELECT id, decision FROM main.ward_audit;", + ); + let err = conn + .execute_batch(&drift_sql) + .expect_err("postcondition guard must reject durable drift before commit"); + assert!( + err.to_string().contains("CHECK constraint failed"), + "unexpected migration error: {err}" + ); + assert!(has_column(&conn, "detail")); + assert!(main_schema_object_exists( + &conn, + "view", + "ward_audit_history" + )); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); + + conn.execute_batch("ROLLBACK;").unwrap(); + + assert_eq!(load_legacy_audit_row(&conn, row_id), before); + assert_eq!(user_version(&conn), 37); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); + assert!(!has_column(&conn, "detail")); + assert!(!main_schema_object_exists(&conn, "table", "ward_audit_new")); + assert!(!main_schema_object_exists( + &conn, + "view", + "ward_audit_history" + )); + assert_eq!( + stored_table_sql(&conn), + sql_literal_value(&conn, ward_audit_exact_legacy_table_sql_sql!()) + ); + assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); + assert_eq!(trigger_names(&conn), expected_trigger_names()); + } + #[test] fn rerunning_migration_after_legacy_upgrade_errors_and_preserves_rows() { let conn = Connection::open_in_memory().unwrap(); diff --git a/docs/architecture.md b/docs/architecture.md index 9148191..a47c2a8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -62,7 +62,7 @@ The rationale stacks three facts: - RFC-0001 §5.6 defines the audit-log entry shape, including `ward_hash`, and requires append-only behavior: entries MUST NOT be deleted or modified. - The daemon already owns `coven.sqlite3`, so putting the table there inherits the existing ownership and access boundary for free. -Every gate verdict is auditable, and WARD-C6's compaction ledger rides the same table rather than a second store. The implemented contract (`audit.rs`, `[IMPLEMENTED, NOT ENFORCING]`) mirrors RFC-0001 §5.6's event vocabulary (`proposal_submitted`, `proposal_approved`, `proposal_rejected`, `proposal_vetoed`, `ward_updated`) and adds the coven-threads extensions (gate verdicts, compaction ledger entries). One implementation note: the SQL table is spelled `ward_audit`, because a literal dot in the name would collide with SQLite's attached-database syntax — and an attached `ward.*` database would *be* the forbidden sidecar. +Every gate verdict is auditable, and WARD-C6's compaction ledger rides the same table rather than a second store. The implemented contract (`audit.rs`, `[IMPLEMENTED, NOT ENFORCING]`) mirrors RFC-0001 §5.6's event vocabulary (`proposal_submitted`, `proposal_approved`, `proposal_rejected`, `proposal_vetoed`, `ward_updated`) and adds the coven-threads extensions (gate verdicts, compaction ledger entries). One implementation note: the SQL table is spelled `ward_audit`, because a literal dot in the name would collide with SQLite's attached-database syntax — and an attached `ward.*` database would *be* the forbidden sidecar. Fresh init and the legacy→v0.1.4 migration both run exact pre- and postcondition guards inside `BEGIN IMMEDIATE`, so callers only commit exact `current_v014` and must `ROLLBACK` on any drift. ## Compatibility contract diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 1be8ccd..411fff2 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -102,6 +102,11 @@ files: `busy_timeout` and repeated synchronized starts proving one caller upgrades, the other waits, then fails only at the legacy guard after seeing exact current (never with a lock error), while preserving the legacy row. +- Add a post-rebuild drift coverage path that reuses + `WARD_AUDIT_MIGRATION_V014_SQL`, injects an unexpected durable + `main.ward_audit_*` object immediately before the real postcondition guard, + requires a guard failure, and proves explicit rollback restores exact legacy + row/schema/constraints/user_version while removing the injected object. - Add reason-demo coverage proving an unqualified `INSERT INTO ward_audit ...` lands in TEMP while the durable contract remains `unknown`. - Add a controlled post-`ALTER` rollback coverage path that starts from exact @@ -154,11 +159,14 @@ files: - Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: - starts with `BEGIN IMMEDIATE` so concurrent migrators serialize before the legacy guard read; - - creates a uniquely named TEMP guard table with `CHECK (ok = 1)`; + - creates a uniquely named TEMP precondition guard table with `CHECK (ok = 1)`; - inserts `1` only when the full exact legacy durable predicate holds and no unexpected durable reserved-name object or temp shadow/reserved temp object exists, otherwise inserts `0` and aborts; - - drops the guard on the success path; + - after the rebuild, creates a distinct TEMP postcondition guard that reruns + the shared schema-state CTE/predicates and requires exact `current_v014` + before `COMMIT`; + - drops both guards on the success path; - keeps `ALTER ADD detail`, strict replacement-table creation, detail copy, and explicit main index/trigger recreation, all qualified to `main` wherever SQLite syntax permits; @@ -195,7 +203,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): serialize schema guard transactions +fix(audit): verify migration postcondition Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index f9019dd..16810ce 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -94,10 +94,12 @@ The v0.1.4 repair remains a table-local migration. The SQL still: 2. creates a strict replacement `main.ward_audit_new`; 3. copies every row, preserving `detail`; 4. swaps tables; and -5. re-creates the required explicit main indexes and append-only main triggers. +5. re-creates the required explicit main indexes and append-only main triggers; + and +6. runs a distinct TEMP post-migration guard before `COMMIT`. The change is the guardrail: immediately after `BEGIN IMMEDIATE;`, the -migration creates a uniquely named TEMP guard table with `CHECK (ok = 1)` and +migration creates a uniquely named TEMP precondition guard table with `CHECK (ok = 1)` and inserts `1` only if the exact `legacy_v013` durable fingerprint holds **and** no unexpected durable reserved-name object or temp shadow/reserved temp object exists. Any missing/current/partial/shadowed schema inserts `0` instead, @@ -109,6 +111,15 @@ makes the migration fail closed even if a caller skips the classification query, and keeps initialization and migration aligned on the same exact state contract. +Before `COMMIT`, the migration now creates a second uniquely named TEMP guard +table that reruns the same shared schema-state CTE/predicates and requires exact +`current_v014`. That postcondition guard catches rebuilt tables that have become +self-unknown inside the still-open `BEGIN IMMEDIATE` transaction — for example, +if an unexpected durable `main.ward_audit_*` view or index appears after the +swap but before commit. On that failure path, the caller's explicit `ROLLBACK` +restores the untouched exact legacy table, its rows, and the pre-migration +namespace while removing the injected drift object. + If a later step fails after `ALTER TABLE`, callers must explicitly `ROLLBACK` the failed transaction before continuing so SQLite restores the untouched legacy table. A preexisting `main.ward_audit_new` is now itself durable drift @@ -150,23 +161,27 @@ Executable rusqlite tests cover: `recorded_at DESC` index, a `COLLATE NOCASE` index, or altered append-only trigger SQL classifying as `unknown`; 10. successful exact-legacy upgrade landing in `current_v014`, with fresh and - migrated current schemas both classifying `current_v014` while matching only - their controlled exact stored SQL variants; and -11. a controlled post-`ALTER` SQL failure inside the transaction validating + migrated current schemas both classifying `current_v014` while matching only + their controlled exact stored SQL variants; +11. a test-only migration variant that injects durable reserved-name drift after + the rebuild but before the real postcondition guard, proving the guard fails + closed and explicit rollback restores the exact legacy row/schema/ + constraints/user_version while removing the injected object; +12. a controlled post-`ALTER` SQL failure inside the transaction validating SQLite rollback semantics for the production migration contract and restoring the full legacy table state; and -12. schema-qualified PRAGMA syntax on bundled SQLite plus the reason for the - fail-closed contract: unqualified inserts hit TEMP first while the contract - remains `unknown`. -13. file-backed multi-connection initialization with two simultaneous callers, - `busy_timeout`, and repeated runs proving both `WARD_AUDIT_SCHEMA_SQL` - executions complete without locked/schema-locked errors because - `BEGIN IMMEDIATE` serializes them before guard reads; and -14. file-backed multi-connection legacy migration with two simultaneous callers, - `busy_timeout`, and repeated runs proving one migration succeeds while the - second waits, re-runs the guard against exact `current_v014`, and fails only - at the legacy guard (never with a lock error), while preserving the legacy - row data. +13. schema-qualified PRAGMA syntax on bundled SQLite plus the reason for the + fail-closed contract: unqualified inserts hit TEMP first while the contract + remains `unknown`. +14. file-backed multi-connection initialization with two simultaneous callers, + `busy_timeout`, and repeated runs proving both `WARD_AUDIT_SCHEMA_SQL` + executions complete without locked/schema-locked errors because + `BEGIN IMMEDIATE` serializes them before guard reads; and +15. file-backed multi-connection legacy migration with two simultaneous callers, + `busy_timeout`, and repeated runs proving one migration succeeds while the + second waits, re-runs the guard against exact `current_v014`, and fails only + at the legacy guard (never with a lock error), while preserving the legacy + row data. ## Scope From 31ae385c5e297cc584da8377600592294d6d62a0 Mon Sep 17 00:00:00 2001 From: Val Alexander Date: Mon, 20 Jul 2026 02:57:44 -0500 Subject: [PATCH 25/25] fix(audit): release ApplyAudit as v0.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 31 ++-- Cargo.lock | 2 +- Cargo.toml | 2 +- crates/coven-threads-core/src/audit.rs | 133 +++++++++--------- crates/coven-threads-core/src/lib.rs | 4 +- docs/architecture.md | 2 +- ...2026-07-19-apply-audit-migration-repair.md | 56 +++++--- ...-19-apply-audit-migration-repair-design.md | 37 ++--- 8 files changed, 146 insertions(+), 121 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 17f2fbc..fc3aecb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,12 @@ All notable changes to `coven-threads-core` are documented here. -## [0.1.4] — unreleased +## [0.2.0] — unreleased + +Compatibility note: `AuditEventType::ApplyAudit` and `WardAuditRecord::detail` +expand public exhaustive APIs. Downstream `match` arms and struct literals can +break under Cargo `^0.1`, so this ships on the `0.2.x` line rather than as a +`0.1.x` patch release. ### Added @@ -14,7 +19,7 @@ All notable changes to `coven-threads-core` are documented here. - `WardAuditRecord::apply_prev_sha256_hex()` / `apply_bytes_written()` — decode helpers for `ApplyAudit` detail. - `APPLY_AUDIT_DETAIL_KEY_PREV` / `APPLY_AUDIT_DETAIL_KEY_BYTES` — stable JSON key constants. - `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable state tags (`missing`, - `legacy_v013`, `current_v014`, `unknown`) — reusable table-local schema + `legacy_v013`, `current_v020`, `unknown`) — reusable table-local schema fingerprint contract for daemon callers, using exact stored `main.sqlite_master.sql` fingerprints for the durable `main.ward_audit` table, explicit main indexes, and append-only main triggers, plus ordered @@ -28,31 +33,31 @@ All notable changes to `coven-threads-core` are documented here. means `main.ward_audit` is absent, no unexpected durable reserved object exists, **and** no temp shadow/reserved temp object exists; any temp-schema table/view/index/trigger named `ward_audit` or `ward_audit_*` also - fail-closes as `unknown`. Only the controlled fresh/migrated v0.1.4 table SQL + fail-closes as `unknown`. Only the controlled fresh/migrated v0.2.0 table SQL variants and the shipped v0.1.3 table SQL are accepted; no whitespace-destroying normalization is applied. - `WARD_AUDIT_SCHEMA_SQL` — atomic, self-guarding schema initialization for the - exact `current_v014` fingerprint. It now begins with `BEGIN IMMEDIATE`, so + exact `current_v020` fingerprint. It now begins with `BEGIN IMMEDIATE`, so the main-database write reservation is acquired before any guard read or classification. That makes concurrent initializers serialize cleanly: the second caller waits, re-runs the guard against the winner's committed schema, - and idempotently sees exact `current_v014` instead of racing into + and idempotently sees exact `current_v020` instead of racing into `sqlite_master` lock errors. The SQL still permits only `missing` or exact - `current_v014` before any mutation, uses idempotent `IF NOT EXISTS` DDL for + `current_v020` before any mutation, uses idempotent `IF NOT EXISTS` DDL for daemon compatibility, targets durable schema objects in `main` wherever - SQLite syntax permits, then requires exact `current_v014` before `COMMIT`. + SQLite syntax permits, then requires exact `current_v020` before `COMMIT`. Exact `legacy_v013`, every drifted `unknown` shape, any unexpected durable reserved object, and any temp shadow/reserved temp object fail closed; callers must `ROLLBACK` after any init error. -- `WARD_AUDIT_MIGRATION_V014_SQL` — transaction SQL for a detail-preserving +- `WARD_AUDIT_MIGRATION_V020_SQL` — transaction SQL for a detail-preserving rebuild of `main.ward_audit` guarded by the exact `legacy_v013` fingerprint plus durable-whitelist and temp-shadow rejection. The daemon should query `WARD_AUDIT_SCHEMA_STATE_SQL`, initialize when missing, migrate only - `legacy_v013`, continue on `current_v014`, and fail closed on `unknown`. The + `legacy_v013`, continue on `current_v020`, and fail closed on `unknown`. The migration independently re-checks `legacy_v013` before any `ALTER`, now inside `BEGIN IMMEDIATE` so concurrent migrators serialize before the guard read. After the rebuild, a distinct TEMP postcondition guard reruns the same - shared schema-state CTE/predicates and requires exact `current_v014` before + shared schema-state CTE/predicates and requires exact `current_v020` before `COMMIT`, so a caller cannot durably commit a self-unknown rebuilt schema if unexpected `main.ward_audit` / `main.ward_audit_*` drift appears mid- transaction. After the first caller upgrades, a second caller waits, re-runs @@ -67,7 +72,7 @@ All notable changes to `coven-threads-core` are documented here. `schema_state_query_returns_missing_on_empty_db`, `fresh_schema_sql_initializes_current_schema_atomically_and_enforces_append_only`, `exact_legacy_fixture_returns_legacy_v013`, - `exact_current_schema_returns_current_v014`, + `exact_current_schema_returns_current_v020`, `current_schema_sql_reruns_idempotently_and_preserves_rows_and_objects`, `fresh_and_migrated_current_schemas_use_controlled_exact_sql_variants`, `current_schema_with_spaced_event_type_literal_is_unknown`, @@ -112,8 +117,8 @@ All notable changes to `coven-threads-core` are documented here. - **Column approach (Option A):** `diff_hash` carries `next_sha256`; `prev_sha256` + `bytes_written` ride in the new `detail` TEXT column as JSON. This avoids a second table rebuild by not adding typed hash columns while keeping the data query-accessible via SQLite JSON functions. - **Migration approach (Option B-lite):** exports `WARD_AUDIT_SCHEMA_STATE_SQL`, `WARD_AUDIT_SCHEMA_SQL`, and - `WARD_AUDIT_MIGRATION_V014_SQL` so the daemon can classify `missing` / - `legacy_v013` / `current_v014` / `unknown`, perform a quiet fail-closed init + `WARD_AUDIT_MIGRATION_V020_SQL` so the daemon can classify `missing` / + `legacy_v013` / `current_v020` / `unknown`, perform a quiet fail-closed init or upgrade, and recover cleanly with explicit rollback after any init or migration error. The durable contract is explicitly `main.ward_audit`, and the reserved durable namespace is valid only for that table plus its two diff --git a/Cargo.lock b/Cargo.lock index 1091649..0925248 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -85,7 +85,7 @@ checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" [[package]] name = "coven-threads-core" -version = "0.1.4" +version = "0.2.0" dependencies = [ "blake3", "rusqlite", diff --git a/Cargo.toml b/Cargo.toml index 633c4b6..a13a952 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["crates/coven-threads-core"] [workspace.package] -version = "0.1.4" +version = "0.2.0" edition = "2021" rust-version = "1.75" license = "MIT" diff --git a/crates/coven-threads-core/src/audit.rs b/crates/coven-threads-core/src/audit.rs index c4c8f5c..55b34b9 100644 --- a/crates/coven-threads-core/src/audit.rs +++ b/crates/coven-threads-core/src/audit.rs @@ -17,19 +17,22 @@ //! //! **Gate-4 applied writes** are also auditable here via //! `AuditEventType::ApplyAudit` (see §3.4, coven-threads#5). +//! Because [`AuditEventType`] and [`WardAuditRecord`] are public exhaustive +//! surfaces, that new variant plus [`WardAuditRecord::detail`] define the +//! v0.2.0 contract rather than a v0.1.x-compatible patch. //! //! ## Schema shape and migration gating //! -//! `WARD_AUDIT_SCHEMA_SQL` initializes or verifies the exact v0.1.4 +//! `WARD_AUDIT_SCHEMA_SQL` initializes or verifies the exact v0.2.0 //! `main.ward_audit` shape in one `BEGIN IMMEDIATE` transaction, without //! mutating database-wide `PRAGMA user_version`. It is safe for the daemon to //! execute unconditionally on every store open: a pre-install guard permits -//! only `missing` or exact `current_v014`, the durable DDL runs explicitly in +//! only `missing` or exact `current_v020`, the durable DDL runs explicitly in //! `main` with `IF NOT EXISTS` compatibility, and a post-install guard requires -//! exact `current_v014` before `COMMIT`. The IMMEDIATE reservation means +//! exact `current_v020` before `COMMIT`. The IMMEDIATE reservation means //! concurrent initializers serialize before any guard/classification read, so a //! second caller waits, re-runs the guard against the winner's committed -//! schema, and then idempotently sees `current_v014`. If either guard fails, +//! schema, and then idempotently sees `current_v020`. If either guard fails, //! callers must explicitly `ROLLBACK` before continuing. //! `WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable, table-local fingerprint query //! that returns one of four stable tags: @@ -40,8 +43,8 @@ //! - `legacy_v013` — `main.ward_audit` exactly matches the v0.1.3 legacy //! fingerprint, the durable reserved namespace contains only the expected //! table/index/trigger objects attached to `main.ward_audit`, and no temp -//! shadow exists; run `WARD_AUDIT_MIGRATION_V014_SQL`; -//! - `current_v014` — `main.ward_audit` exactly matches the v0.1.4 current +//! shadow exists; run `WARD_AUDIT_MIGRATION_V020_SQL`; +//! - `current_v020` — `main.ward_audit` exactly matches the v0.2.0 current //! fingerprint, the durable reserved namespace contains only the expected //! table/index/trigger objects attached to `main.ward_audit`, and no temp //! shadow exists; continue without schema work; @@ -55,19 +58,19 @@ //! table, explicit durable indexes, and append-only durable triggers, plus //! ordered `pragma_table_info('ward_audit', 'main')` metadata and //! `pragma_index_list('ward_audit', 'main')` for explicit index discovery. It -//! does **not** normalize whitespace: the only accepted `current_v014` +//! does **not** normalize whitespace: the only accepted `current_v020` //! table-SQL variants are the fresh `CREATE TABLE ward_audit` form and the //! quoted `CREATE TABLE "ward_audit"` form SQLite stores after the exact legacy //! migration path, and the `legacy_v013` fingerprint includes the inline //! comments preserved from the shipped v0.1.3 DDL. -//! `WARD_AUDIT_MIGRATION_V014_SQL` exists only for the exact `legacy_v013` +//! `WARD_AUDIT_MIGRATION_V020_SQL` exists only for the exact `legacy_v013` //! fingerprint with no unexpected durable reserved-namespace object and no temp //! shadow. It independently re-checks that fingerprint inside one //! `BEGIN IMMEDIATE` transaction before any `ALTER`, then adds `detail` and //! rebuilds `main.ward_audit` so the current CHECK set is installed and every //! existing row is preserved. Before `COMMIT`, a distinct TEMP postcondition //! guard reruns the shared schema-state CTE/predicates and requires exact -//! `current_v014`, so callers cannot durably commit a rebuilt-but-self-unknown +//! `current_v020`, so callers cannot durably commit a rebuilt-but-self-unknown //! namespace if extra `ward_audit` / `ward_audit_*` objects appear mid-transaction. //! The IMMEDIATE reservation means concurrent migrators serialize before any //! legacy-guard read: the winner upgrades first, and a second caller waits, @@ -358,9 +361,9 @@ pub const WARD_AUDIT_SCHEMA_STATE_MISSING: &str = "missing"; /// durable reserved-namespace object and no temp shadow. pub const WARD_AUDIT_SCHEMA_STATE_LEGACY_V013: &str = "legacy_v013"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for the exact -/// `main.ward_audit` v0.1.4 current schema fingerprint, with no unexpected +/// `main.ward_audit` v0.2.0 current schema fingerprint, with no unexpected /// durable reserved-namespace object and no temp shadow. -pub const WARD_AUDIT_SCHEMA_STATE_CURRENT_V014: &str = "current_v014"; +pub const WARD_AUDIT_SCHEMA_STATE_CURRENT_V020: &str = "current_v020"; /// Stable tag returned by [`WARD_AUDIT_SCHEMA_STATE_SQL`] for every other /// `main.ward_audit` shape, plus any unexpected durable reserved-namespace /// object or any temp-schema `ward_audit` / `ward_audit_*` shadow object. @@ -631,7 +634,7 @@ macro_rules! ward_audit_schema_state_case_sql { r#" THEN 'legacy_v013' WHEN "#, ward_audit_exact_current_predicate_sql!(), - r#" THEN 'current_v014' + r#" THEN 'current_v020' ELSE 'unknown' END"# ) @@ -647,8 +650,8 @@ END"# /// `ward_audit_*` exists, and no temp shadow/reserved object exists; /// initialize with [`WARD_AUDIT_SCHEMA_SQL`]; /// - [`WARD_AUDIT_SCHEMA_STATE_LEGACY_V013`] — run -/// [`WARD_AUDIT_MIGRATION_V014_SQL`]; -/// - [`WARD_AUDIT_SCHEMA_STATE_CURRENT_V014`] — continue without schema work; +/// [`WARD_AUDIT_MIGRATION_V020_SQL`]; +/// - [`WARD_AUDIT_SCHEMA_STATE_CURRENT_V020`] — continue without schema work; /// - [`WARD_AUDIT_SCHEMA_STATE_UNKNOWN`] — fail closed and investigate the /// table manually. /// @@ -672,7 +675,7 @@ END"# /// `ward_audit_` also returns `unknown`, even when `main.ward_audit` itself is /// exact current or legacy, so callers cannot treat a shadowed namespace as /// healthy durable state. No whitespace-destroying normalization is applied: -/// the only accepted `current_v014` table SQL variants are the fresh +/// the only accepted `current_v020` table SQL variants are the fresh /// `CREATE TABLE ward_audit (...)` form and SQLite's quoted /// `CREATE TABLE "ward_audit" (...)` form produced by the exact legacy /// migration path, while the `legacy_v013` fingerprint intentionally includes @@ -707,11 +710,11 @@ FROM ward_audit_shape; /// 6. re-creates the exact explicit main indexes and append-only main triggers; /// and /// 7. re-runs the shared schema-state expression in a distinct TEMP -/// postcondition guard and requires exact `current_v014` before `COMMIT`. +/// postcondition guard and requires exact `current_v020` before `COMMIT`. /// /// Callers should still branch on [`WARD_AUDIT_SCHEMA_STATE_SQL`] first: /// initialize when the state is `missing`, migrate only `legacy_v013`, -/// continue on `current_v014`, and fail closed on `unknown`. This migration +/// continue on `current_v020`, and fail closed on `unknown`. This migration /// independently guards the same `legacy_v013` fingerprint, durable /// reserved-namespace whitelist, temp-shadow rejection, and exact-current /// postcondition so callers cannot mutate a partial, already-current, @@ -724,7 +727,7 @@ FROM ward_audit_shape; /// `ROLLBACK` the failed transaction before continuing so SQLite restores the /// untouched legacy table. This SQL does not read or write database-wide /// `PRAGMA user_version`. -pub const WARD_AUDIT_MIGRATION_V014_SQL: &str = concat!( +pub const WARD_AUDIT_MIGRATION_V020_SQL: &str = concat!( r#" BEGIN IMMEDIATE; @@ -811,7 +814,7 @@ INSERT INTO coven_threads_ward_audit_migration_post_guard (ok) SELECT CASE WHEN ("#, ward_audit_schema_state_case_sql!(), - r#") = 'current_v014' THEN 1 + r#") = 'current_v020' THEN 1 ELSE 0 END FROM ward_audit_shape; @@ -822,7 +825,7 @@ COMMIT; "#, ); -// Shared durable-main current-v0.1.4 DDL body used by the guarded init SQL and +// Shared durable-main current-v0.2.0 DDL body used by the guarded init SQL and // drift tests. macro_rules! ward_audit_current_objects_sql { () => { @@ -872,14 +875,14 @@ END; /// /// See module docs for the full schema-state contract. This `BEGIN IMMEDIATE` /// transaction is safe to run unconditionally on every store open: it permits -/// only exact `missing` or `current_v014` before any mutation, uses idempotent +/// only exact `missing` or `current_v020` before any mutation, uses idempotent /// `IF NOT EXISTS` DDL for daemon compatibility, then requires exact -/// `current_v014` before `COMMIT`. Exact `legacy_v013`, every drifted +/// `current_v020` before `COMMIT`. Exact `legacy_v013`, every drifted /// `unknown` shape, every unexpected durable reserved-namespace object, and /// every temp shadow/reserved temp object fail closed. The IMMEDIATE /// reservation serializes concurrent initializers before any guard read so the /// loser waits, re-runs the guard against committed state, and sees exact -/// `current_v014` rather than racing into `sqlite_master` lock errors. Durable +/// `current_v020` rather than racing into `sqlite_master` lock errors. Durable /// schema objects are explicitly created in `main`; the temp guard tables live /// in `temp` under unique non-reserved names. If this SQL errors, callers must /// explicitly `ROLLBACK` before continuing so SQLite discards any uncommitted @@ -899,7 +902,7 @@ INSERT INTO coven_threads_ward_audit_schema_pre_guard (ok) SELECT CASE WHEN ("#, ward_audit_schema_state_case_sql!(), - r#") IN ('missing', 'current_v014') THEN 1 + r#") IN ('missing', 'current_v020') THEN 1 ELSE 0 END FROM ward_audit_shape; @@ -919,7 +922,7 @@ INSERT INTO coven_threads_ward_audit_schema_post_guard (ok) SELECT CASE WHEN ("#, ward_audit_schema_state_case_sql!(), - r#") = 'current_v014' THEN 1 + r#") = 'current_v020' THEN 1 ELSE 0 END FROM ward_audit_shape; @@ -1268,7 +1271,7 @@ END; fn migration_sql_with_durable_drift_before_post_guard(drift_sql: &str) -> String { inject_sql_before_anchor( - WARD_AUDIT_MIGRATION_V014_SQL, + WARD_AUDIT_MIGRATION_V020_SQL, "CREATE TEMP TABLE coven_threads_ward_audit_migration_post_guard (", drift_sql, ) @@ -1440,7 +1443,7 @@ END; conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); assert_eq!(user_version(&conn), initial_version); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); let row_id = insert_current_apply_audit_row(&conn); let row = load_audit_row(&conn, row_id); @@ -1462,7 +1465,7 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("drifted legacy schema must fail at the migration guard"); assert!( err.to_string().contains("CHECK constraint failed"), @@ -1611,7 +1614,7 @@ END; "apply_audit", Some("proposal-current"), "familiar-current", - Some("0.1.4"), + Some("0.2.0"), FIXED_WARD_HASH.as_ref(), Some("tier_2"), "applied", @@ -1651,7 +1654,7 @@ END; "apply_audit", Some("proposal-current"), "familiar-current", - Some("0.1.4"), + Some("0.2.0"), FIXED_WARD_HASH.as_ref(), Some("tier_2"), "applied", @@ -1842,7 +1845,7 @@ END; fn schema_and_migration_sql_use_begin_immediate() { for (label, sql) in [ ("schema", WARD_AUDIT_SCHEMA_SQL), - ("migration", WARD_AUDIT_MIGRATION_V014_SQL), + ("migration", WARD_AUDIT_MIGRATION_V020_SQL), ] { assert!( sql.trim_start().starts_with("BEGIN IMMEDIATE;"), @@ -1853,16 +1856,16 @@ END; #[test] fn migration_sql_uses_distinct_pre_and_post_guards_before_commit() { - let pre_guard_offset = WARD_AUDIT_MIGRATION_V014_SQL + let pre_guard_offset = WARD_AUDIT_MIGRATION_V020_SQL .find("CREATE TEMP TABLE coven_threads_ward_audit_migration_guard (") .expect("migration SQL must define a precondition guard"); - let post_guard_offset = WARD_AUDIT_MIGRATION_V014_SQL + let post_guard_offset = WARD_AUDIT_MIGRATION_V020_SQL .find("CREATE TEMP TABLE coven_threads_ward_audit_migration_post_guard (") .expect("migration SQL must define a postcondition guard"); - let post_guard_drop_offset = WARD_AUDIT_MIGRATION_V014_SQL + let post_guard_drop_offset = WARD_AUDIT_MIGRATION_V020_SQL .find("DROP TABLE coven_threads_ward_audit_migration_post_guard;") .expect("migration SQL must drop the postcondition guard before commit"); - let commit_offset = WARD_AUDIT_MIGRATION_V014_SQL + let commit_offset = WARD_AUDIT_MIGRATION_V020_SQL .rfind("COMMIT;") .expect("migration SQL must commit on success"); @@ -2019,7 +2022,7 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); let migration_err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("temp shadow must make migration fail closed"); assert!( migration_err @@ -2041,7 +2044,7 @@ END; conn.execute_batch("DROP TABLE temp.ward_audit;").unwrap(); assert!(!temp_schema_object_exists(&conn, "table", "ward_audit")); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(load_audit_row(&conn, main_row_id), main_before); } @@ -2142,7 +2145,7 @@ END; conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(user_version(&conn), 11); assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); assert_eq!(trigger_names(&conn), expected_trigger_names()); @@ -2186,10 +2189,10 @@ END; } #[test] - fn exact_current_schema_returns_current_v014() { + fn exact_current_schema_returns_current_v020() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!( reserved_main_namespace_objects(&conn), expected_reserved_main_namespace_objects() @@ -2226,7 +2229,7 @@ END; assert_eq!(explicit_index_names(&conn), before_indexes); assert_eq!(trigger_names(&conn), before_triggers); assert_eq!(user_version(&conn), 29); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); } #[test] @@ -2239,11 +2242,11 @@ END; .execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL) .unwrap(); migrated - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .unwrap(); - assert_schema_state(&fresh, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); - assert_schema_state(&migrated, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&fresh, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); + assert_schema_state(&migrated, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!( stored_table_sql(&fresh), sql_literal_value(&fresh, ward_audit_exact_current_fresh_table_sql_sql!()) @@ -2312,7 +2315,7 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("temp shadow must make legacy migration fail before mutation"); assert!( err.to_string().contains("CHECK constraint failed"), @@ -2403,7 +2406,7 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("partial legacy schema must fail at the migration guard"); assert!( err.to_string().contains("CHECK constraint failed"), @@ -2485,7 +2488,7 @@ END; fn current_schema_missing_append_only_trigger_is_unknown_and_update_succeeds() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); let row_id = insert_current_apply_audit_row(&conn); conn.execute_batch("DROP TRIGGER ward_audit_append_only_update;") @@ -2504,7 +2507,7 @@ END; fn current_schema_with_extra_index_is_unknown() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); conn.execute_batch("CREATE INDEX ward_audit_decision_idx ON ward_audit (decision);") .unwrap(); @@ -2516,7 +2519,7 @@ END; fn current_schema_with_desc_index_is_unknown() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); conn.execute_batch( "DROP INDEX ward_audit_event_idx; @@ -2536,7 +2539,7 @@ END; fn current_schema_with_collated_index_is_unknown() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); conn.execute_batch( "DROP INDEX ward_audit_event_idx; @@ -2628,7 +2631,7 @@ END; fn current_schema_with_altered_trigger_error_literal_is_unknown() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); conn.execute_batch( "DROP TRIGGER ward_audit_append_only_update; @@ -2652,7 +2655,7 @@ END; fn current_schema_with_altered_trigger_body_is_unknown() { let conn = Connection::open_in_memory().unwrap(); conn.execute_batch(WARD_AUDIT_SCHEMA_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); conn.execute_batch( "DROP TRIGGER ward_audit_append_only_delete; @@ -2800,10 +2803,10 @@ END; let before = load_audit_row(&conn, row_id); let before_version = user_version(&conn); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("current schema migration must fail at the legacy guard"); assert!( err.to_string().contains("CHECK constraint failed"), @@ -2812,7 +2815,7 @@ END; conn.execute_batch("ROLLBACK;").unwrap(); assert_eq!(load_audit_row(&conn, row_id), before); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(user_version(&conn), before_version); } @@ -2846,7 +2849,7 @@ END; assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_UNKNOWN); let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("preexisting ward_audit_new must make migration fail before ALTER"); assert!( err.to_string().contains("CHECK constraint failed"), @@ -2874,9 +2877,9 @@ END; assert_eq!(user_version(&conn), 37); assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013); - conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); + conn.execute_batch(WARD_AUDIT_MIGRATION_V020_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(user_version(&conn), 37); assert_eq!(explicit_index_names(&conn), expected_explicit_index_names()); assert_eq!(trigger_names(&conn), expected_trigger_names()); @@ -2974,8 +2977,8 @@ END; conn.execute_batch(LEGACY_WARD_AUDIT_SCHEMA_SQL).unwrap(); let legacy_row_id = insert_legacy_ward_updated_row(&conn); - conn.execute_batch(WARD_AUDIT_MIGRATION_V014_SQL).unwrap(); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + conn.execute_batch(WARD_AUDIT_MIGRATION_V020_SQL).unwrap(); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(user_version(&conn), 37); let apply_row_id = insert_current_apply_audit_row(&conn); @@ -2983,7 +2986,7 @@ END; let apply_before = load_audit_row(&conn, apply_row_id); let err = conn - .execute_batch(WARD_AUDIT_MIGRATION_V014_SQL) + .execute_batch(WARD_AUDIT_MIGRATION_V020_SQL) .expect_err("rerunning the migration must fail at the legacy guard"); assert!( err.to_string().contains("CHECK constraint failed"), @@ -2993,7 +2996,7 @@ END; assert_eq!(load_audit_row(&conn, legacy_row_id), legacy_before); assert_eq!(load_audit_row(&conn, apply_row_id), apply_before); - assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(user_version(&conn), 37); } @@ -3018,7 +3021,7 @@ END; { let final_conn = open_file_backed_connection(db.path()); assert_eq!(user_version(&final_conn), 41); - assert_schema_state(&final_conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&final_conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!( explicit_index_names(&final_conn), expected_explicit_index_names() @@ -3049,7 +3052,7 @@ END; (row_id, before) }; - let outcomes = run_concurrent_sql(db.path(), WARD_AUDIT_MIGRATION_V014_SQL); + let outcomes = run_concurrent_sql(db.path(), WARD_AUDIT_MIGRATION_V020_SQL); let success_count = outcomes.iter().filter(|result| result.is_ok()).count(); let errors: Vec<_> = outcomes .iter() @@ -3079,7 +3082,7 @@ END; { let final_conn = open_file_backed_connection(db.path()); assert_eq!(user_version(&final_conn), 37); - assert_schema_state(&final_conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V014); + assert_schema_state(&final_conn, WARD_AUDIT_SCHEMA_STATE_CURRENT_V020); assert_eq!(ward_audit_row_count(&final_conn, "main"), 1); assert_eq!(load_audit_row(&final_conn, row_id), before); assert_eq!( diff --git a/crates/coven-threads-core/src/lib.rs b/crates/coven-threads-core/src/lib.rs index 052e578..712d9a2 100644 --- a/crates/coven-threads-core/src/lib.rs +++ b/crates/coven-threads-core/src/lib.rs @@ -67,8 +67,8 @@ pub mod validate; pub mod weave; pub use audit::{ - AuditEventType, WardAuditRecord, WARD_AUDIT_MIGRATION_V014_SQL, WARD_AUDIT_SCHEMA_SQL, - WARD_AUDIT_SCHEMA_STATE_CURRENT_V014, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013, + AuditEventType, WardAuditRecord, WARD_AUDIT_MIGRATION_V020_SQL, WARD_AUDIT_SCHEMA_SQL, + WARD_AUDIT_SCHEMA_STATE_CURRENT_V020, WARD_AUDIT_SCHEMA_STATE_LEGACY_V013, WARD_AUDIT_SCHEMA_STATE_MISSING, WARD_AUDIT_SCHEMA_STATE_SQL, WARD_AUDIT_SCHEMA_STATE_UNKNOWN, }; pub use channel::Channel; diff --git a/docs/architecture.md b/docs/architecture.md index a47c2a8..be64efc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -62,7 +62,7 @@ The rationale stacks three facts: - RFC-0001 §5.6 defines the audit-log entry shape, including `ward_hash`, and requires append-only behavior: entries MUST NOT be deleted or modified. - The daemon already owns `coven.sqlite3`, so putting the table there inherits the existing ownership and access boundary for free. -Every gate verdict is auditable, and WARD-C6's compaction ledger rides the same table rather than a second store. The implemented contract (`audit.rs`, `[IMPLEMENTED, NOT ENFORCING]`) mirrors RFC-0001 §5.6's event vocabulary (`proposal_submitted`, `proposal_approved`, `proposal_rejected`, `proposal_vetoed`, `ward_updated`) and adds the coven-threads extensions (gate verdicts, compaction ledger entries). One implementation note: the SQL table is spelled `ward_audit`, because a literal dot in the name would collide with SQLite's attached-database syntax — and an attached `ward.*` database would *be* the forbidden sidecar. Fresh init and the legacy→v0.1.4 migration both run exact pre- and postcondition guards inside `BEGIN IMMEDIATE`, so callers only commit exact `current_v014` and must `ROLLBACK` on any drift. +Every gate verdict is auditable, and WARD-C6's compaction ledger rides the same table rather than a second store. The implemented contract (`audit.rs`, `[IMPLEMENTED, NOT ENFORCING]`) mirrors RFC-0001 §5.6's event vocabulary (`proposal_submitted`, `proposal_approved`, `proposal_rejected`, `proposal_vetoed`, `ward_updated`) and adds the coven-threads extensions (gate verdicts, compaction ledger entries). One implementation note: the SQL table is spelled `ward_audit`, because a literal dot in the name would collide with SQLite's attached-database syntax — and an attached `ward.*` database would *be* the forbidden sidecar. Fresh init and the legacy→v0.2.0 migration both run exact pre- and postcondition guards inside `BEGIN IMMEDIATE`, so callers only commit exact `current_v020` and must `ROLLBACK` on any drift. ## Compatibility contract diff --git a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md index 411fff2..e5877b4 100644 --- a/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md +++ b/docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md @@ -9,15 +9,19 @@ closed on any TEMP shadow/reserved temp object, make schema initialization fail closed and atomic, preserve evidence and unrelated `user_version`, and prove rollback behavior with executable rusqlite tests. +Because PR #7 also adds the public `AuditEventType::ApplyAudit` enum variant +and the public `WardAuditRecord::detail` field, the release must bump to v0.2.0 +instead of claiming `0.1.x` compatibility under Cargo `^0.1`. + **Architecture:** Keep ownership in `crates/coven-threads-core/src/audit.rs`. Expose `WARD_AUDIT_SCHEMA_STATE_SQL` plus stable tags so callers can branch on -`missing` / `legacy_v013` / `current_v014` / `unknown`, reserve `missing` for +`missing` / `legacy_v013` / `current_v020` / `unknown`, reserve `missing` for an absent `main.ward_audit` with no unexpected durable reserved-name object and no temp shadow/reserved temp object, require the same exact durable whitelist -for `legacy_v013` and `current_v014`, wrap both `WARD_AUDIT_SCHEMA_SQL` and -`WARD_AUDIT_MIGRATION_V014_SQL` in `BEGIN IMMEDIATE` transactions so guard +for `legacy_v013` and `current_v020`, wrap both `WARD_AUDIT_SCHEMA_SQL` and +`WARD_AUDIT_MIGRATION_V020_SQL` in `BEGIN IMMEDIATE` transactions so guard reads happen under the main-database write reservation, let concurrent init -serialize cleanly to `current_v014`, and make a concurrent second migration +serialize cleanly to `current_v020`, and make a concurrent second migration wait, then reclassify current when the legacy guard re-runs. Durable DDL/DML stays explicitly qualified to `main` wherever SQLite permits it, and the init/migration paths never write database-wide `user_version`. @@ -29,14 +33,17 @@ dev-dependencies) for in-memory and file-backed concurrency tests. ## Final File Map -For this follow-up fingerprint repair, the diff stays scoped to these five -files: +For this semver-corrected follow-up, the repair delta relative to +`origin/cody/apply-audit-v014` stays scoped to these eight tracked files: 1. `CHANGELOG.md` -2. `crates/coven-threads-core/Cargo.toml` -3. `crates/coven-threads-core/src/audit.rs` -4. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` -5. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` +2. `Cargo.lock` +3. `Cargo.toml` +4. `crates/coven-threads-core/src/audit.rs` +5. `crates/coven-threads-core/src/lib.rs` +6. `docs/architecture.md` +7. `docs/superpowers/plans/2026-07-19-apply-audit-migration-repair.md` +8. `docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md` --- @@ -45,7 +52,7 @@ files: - Add schema-state tests for: - empty DB → `missing`; - exact legacy fixture → `legacy_v013`; - - exact current schema → `current_v014`; + - exact current schema → `current_v020`; - exact legacy/current with only the durable whitelist objects still classifying correctly; - absent-table reserved-name collisions (`ward_audit_event_idx`, @@ -60,18 +67,18 @@ files: - `pragma_index_list('ward_audit', 'main')` sees durable explicit indexes. - Add init-safety tests for: - clean empty DB → `WARD_AUDIT_SCHEMA_SQL` succeeds atomically, lands on - `current_v014`, and append-only UPDATE/DELETE still abort; + `current_v020`, and append-only UPDATE/DELETE still abort; - exact current schema → rerunning `WARD_AUDIT_SCHEMA_SQL` is idempotent and preserves rows/objects; - file-backed two-connection concurrent init with `busy_timeout` and repeated synchronized starts → both callers complete successfully because `BEGIN IMMEDIATE` serializes before guard reads, with no locked/schema-locked - errors and exact final `current_v014`; + errors and exact final `current_v020`; - exact legacy schema → `WARD_AUDIT_SCHEMA_SQL` rejects, requires explicit `ROLLBACK`, and preserves state/data; - exact current `main.ward_audit` + TEMP shadow clone → init rejects, explicit rollback preserves both main and temp rows, and dropping the TEMP - shadow restores `current_v014`; + shadow restores `current_v020`; - missing main + TEMP `ward_audit` / reserved `ward_audit_*` temp object → init rejects and never creates `main.ward_audit`; - unknown partial current schema → `WARD_AUDIT_SCHEMA_SQL` rejects and @@ -103,7 +110,7 @@ files: the other waits, then fails only at the legacy guard after seeing exact current (never with a lock error), while preserving the legacy row. - Add a post-rebuild drift coverage path that reuses - `WARD_AUDIT_MIGRATION_V014_SQL`, injects an unexpected durable + `WARD_AUDIT_MIGRATION_V020_SQL`, injects an unexpected durable `main.ward_audit_*` object immediately before the real postcondition guard, requires a guard failure, and proves explicit rollback restores exact legacy row/schema/constraints/user_version while removing the injected object. @@ -115,7 +122,7 @@ files: inside the same transaction; require the error, explicit rollback success, full row/schema/user_version restoration, and `legacy_v013` classification. - Add exact-stored-SQL tests proving fresh current and successfully migrated - current both classify `current_v014` while matching only their controlled + current both classify `current_v020` while matching only their controlled SQLite-emitted table-SQL variants, and that the shipped legacy fixture matches `legacy_v013`. @@ -147,16 +154,16 @@ files: acquired before any guard read/classification; - creates a uniquely named TEMP pre-install guard with `CHECK (ok = 1)`; - inserts `1` only when the shared schema-state expression returns - `missing` or `current_v014`, otherwise inserts `0` and aborts before any + `missing` or `current_v020`, otherwise inserts `0` and aborts before any mutation; - keeps `CREATE TABLE/INDEX/TRIGGER IF NOT EXISTS` for current daemon compatibility while targeting `main` unambiguously wherever SQLite syntax allows it; - creates a uniquely named TEMP post-install guard that requires exact - `current_v014`, aborting on any silent no-op/collision or malformed result; + `current_v020`, aborting on any silent no-op/collision or malformed result; - commits on success; and - requires callers to `ROLLBACK` after any init error. -- Update `WARD_AUDIT_MIGRATION_V014_SQL` so it: +- Update `WARD_AUDIT_MIGRATION_V020_SQL` so it: - starts with `BEGIN IMMEDIATE` so concurrent migrators serialize before the legacy guard read; - creates a uniquely named TEMP precondition guard table with `CHECK (ok = 1)`; @@ -164,7 +171,7 @@ files: unexpected durable reserved-name object or temp shadow/reserved temp object exists, otherwise inserts `0` and aborts; - after the rebuild, creates a distinct TEMP postcondition guard that reruns - the shared schema-state CTE/predicates and requires exact `current_v014` + the shared schema-state CTE/predicates and requires exact `current_v020` before `COMMIT`; - drops both guards on the success path; - keeps `ALTER ADD detail`, strict replacement-table creation, detail copy, @@ -174,6 +181,9 @@ files: ## Task 3: Update the written contract and validate +- Bump `[workspace.package].version` in the root `Cargo.toml` to `0.2.0`, let + Cargo refresh the `coven-threads-core` lock entry, and verify + `cargo metadata --no-deps --format-version 1` reports `0.2.0`. - Update `audit.rs` docs, the design doc, the plan doc, and `CHANGELOG.md` to say that the durable audit contract is `main.ward_audit`, exact stored `main.sqlite_master.sql` equality covers every declared table-level @@ -187,13 +197,15 @@ files: - query `WARD_AUDIT_SCHEMA_STATE_SQL`; - initialize on `missing` only; - migrate only `legacy_v013`; - - continue on `current_v014`; + - continue on `current_v020`; - fail closed on `unknown`; - `ROLLBACK` after any init or migration error; and - rely on the init/migration independent guards as second lines of defense. - Run: - `cargo fmt` + - `cargo clippy --workspace --all-targets -- -D warnings` - focused audit tests + - `cargo metadata --no-deps --format-version 1` - `cargo test --workspace` if focused tests pass - `git diff --check` - stale-wording searches for the old partial-gate terminology. @@ -203,7 +215,7 @@ files: Create a new commit (no amend) with: ```text -fix(audit): verify migration postcondition +fix(audit): release ApplyAudit as v0.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> ``` diff --git a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md index 16810ce..07a9e31 100644 --- a/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md +++ b/docs/superpowers/specs/2026-07-19-apply-audit-migration-repair-design.md @@ -8,6 +8,11 @@ immutable evidence, preserve unrelated shared-store `user_version`, reject partial or drifted schemas before mutation, and remain scoped to the audit schema contract. +Because PR #7 adds the public `AuditEventType::ApplyAudit` enum variant and the +public `WardAuditRecord::detail` field, the corrected release line is v0.2.0. +Cargo `^0.1` consumers can break on exhaustive `match` arms or struct literals, +so the current schema/state contract is named `current_v020`. + ## Schema-state contract `crates/coven-threads-core::WARD_AUDIT_SCHEMA_STATE_SQL` is the reusable query @@ -19,8 +24,8 @@ stable tag: shadow/reserved temp object exists; initialize with `WARD_AUDIT_SCHEMA_SQL`. - `legacy_v013` — exact v0.1.3 legacy fingerprint plus the durable namespace whitelist; run - `WARD_AUDIT_MIGRATION_V014_SQL`. -- `current_v014` — exact v0.1.4 current fingerprint plus the durable namespace + `WARD_AUDIT_MIGRATION_V020_SQL`. +- `current_v020` — exact v0.2.0 current fingerprint plus the durable namespace whitelist; continue without schema work. - `unknown` — every other shape; fail closed and investigate manually. @@ -52,9 +57,9 @@ whose name is exactly `ward_audit` or begins with `ward_audit_` likewise classifies as `unknown`, even when `main.ward_audit` is otherwise exact current or legacy; that fail-closed rule stops TEMP shadows from being mistaken for healthy durable state and blocks unqualified daemon writes from proceeding under -a false `current_v014`. No whitespace-destroying normalization is applied: only +a false `current_v020`. No whitespace-destroying normalization is applied: only the exact stored SQL variants SQLite emits for the shipped schemas are -accepted. For `current_v014`, that means the fresh `CREATE TABLE ward_audit +accepted. For `current_v020`, that means the fresh `CREATE TABLE ward_audit (...)` form and the quoted `CREATE TABLE "ward_audit" (...)` form produced by the legacy rename path. For `legacy_v013`, the fingerprint intentionally includes the inline comments preserved from the shipped v0.1.3 DDL. @@ -68,18 +73,18 @@ therefore: 1. `BEGIN IMMEDIATE;` to reserve the main-database write slot before any guard read/classification; 2. run a uniquely named TEMP pre-install guard that reuses the exact - schema-state CTEs and permits only `missing` or exact `current_v014`; + schema-state CTEs and permits only `missing` or exact `current_v020`; 3. create the durable `main.ward_audit` table, explicit main indexes, and append-only main triggers with `IF NOT EXISTS`; 4. run a uniquely named TEMP post-install guard that reuses the same - schema-state expression and requires exact `current_v014`; and + schema-state expression and requires exact `current_v020`; and 5. `COMMIT;` This makes fresh installs atomic, makes exact current reruns idempotent, and serializes concurrent initializers before they read `main.sqlite_master`: the winner reserves the main-database write slot first, and the second caller waits until commit, re-runs the guard against the committed schema, and then -idempotently sees exact `current_v014`. The path still fails closed for +idempotently sees exact `current_v020`. The path still fails closed for `legacy_v013`, `unknown`, any unexpected durable reserved-name object, any temp shadow/reserved temp object, or any malformed/silently skipped install result. If either guard errors, callers must explicitly `ROLLBACK` before continuing so @@ -88,7 +93,7 @@ path while preserving unrelated durable/temp objects that caused the collision. ## Migration design -The v0.1.4 repair remains a table-local migration. The SQL still: +The v0.2.0 repair remains a table-local migration. The SQL still: 1. `ALTER TABLE main.ward_audit ADD COLUMN detail TEXT`; 2. creates a strict replacement `main.ward_audit_new`; @@ -113,7 +118,7 @@ contract. Before `COMMIT`, the migration now creates a second uniquely named TEMP guard table that reruns the same shared schema-state CTE/predicates and requires exact -`current_v014`. That postcondition guard catches rebuilt tables that have become +`current_v020`. That postcondition guard catches rebuilt tables that have become self-unknown inside the still-open `BEGIN IMMEDIATE` transaction — for example, if an unexpected durable `main.ward_audit_*` view or index appears after the swap but before commit. On that failure path, the caller's explicit `ROLLBACK` @@ -134,12 +139,12 @@ so unrelated shared-store version state is preserved. Executable rusqlite tests cover: 1. schema-state classification for `missing`, exact `legacy_v013`, and exact - `current_v014`, including absent-table reserved durable-namespace collisions, + `current_v020`, including absent-table reserved durable-namespace collisions, exact current/legacy shapes with extra durable reserved-name view/table objects, reserved temp objects, and TEMP `ward_audit` shadows classifying as `unknown`; 2. fresh empty-store initialization classifying `missing`, running - `WARD_AUDIT_SCHEMA_SQL` atomically to `current_v014`, and enforcing + `WARD_AUDIT_SCHEMA_SQL` atomically to `current_v020`, and enforcing append-only UPDATE/DELETE rejection; 3. exact current reruns of `WARD_AUDIT_SCHEMA_SQL` staying idempotent while preserving rows, indexes, triggers, and exact stored SQL fingerprints; @@ -160,8 +165,8 @@ Executable rusqlite tests cover: 9. current drift cases such as a missing append-only trigger, an extra index, a `recorded_at DESC` index, a `COLLATE NOCASE` index, or altered append-only trigger SQL classifying as `unknown`; -10. successful exact-legacy upgrade landing in `current_v014`, with fresh and - migrated current schemas both classifying `current_v014` while matching only +10. successful exact-legacy upgrade landing in `current_v020`, with fresh and + migrated current schemas both classifying `current_v020` while matching only their controlled exact stored SQL variants; 11. a test-only migration variant that injects durable reserved-name drift after the rebuild but before the real postcondition guard, proving the guard fails @@ -179,7 +184,7 @@ Executable rusqlite tests cover: `BEGIN IMMEDIATE` serializes them before guard reads; and 15. file-backed multi-connection legacy migration with two simultaneous callers, `busy_timeout`, and repeated runs proving one migration succeeds while the - second waits, re-runs the guard against exact `current_v014`, and fails only + second waits, re-runs the guard against exact `current_v020`, and fails only at the legacy guard (never with a lock error), while preserving the legacy row data. @@ -191,7 +196,7 @@ This repair does not add daemon migration orchestration or change the migration, treating `main.ward_audit` as the only durable audit contract, accepting only the exact durable whitelist of `main.ward_audit` plus its two indexes and two append-only triggers, running `WARD_AUDIT_SCHEMA_SQL` only -through the allowed `missing`/`current_v014` contract, and failing closed on +through the allowed `missing`/`current_v020` contract, and failing closed on `unknown`. Concurrent callers must also treat a migration guard rejection after waiting as a signal to reclassify: another writer may already have serialized -the schema to exact `current_v014`. +the schema to exact `current_v020`.