Skip to content

Commit 551b78b

Browse files
refactor(dogfood): drop the DB-migration boundary ceremony
With no database migrations in the product, the persisted boundary marker, its checksum and state-machine legality tables, and the forward-recovery gate existed to shepherd schema migrations that no longer happen. The forward-only binary policy stays: identity checks, the strict dogfood-forward-only post-update mode, the backup contract, and a post-install failure path that proves the daemon stopped and prints forward-recovery instructions inline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent cd8febb commit 551b78b

2 files changed

Lines changed: 78 additions & 404 deletions

File tree

docs/DOGFOOD-RECOVERY-RUNBOOK.md

Lines changed: 71 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -1,122 +1,105 @@
1-
# Dogfood Migration-Boundary Recovery Runbook
1+
# Dogfood Recovery Runbook
22

3-
How to recover `cargo dogfood` once it has crossed its forward-only migration
4-
boundary. Documents the boundary state file format, the backup contract, the
3+
How to recover `cargo dogfood` if it fails partway through installing a new
4+
binary. Documents the forward-only binary policy, the backup contract, the
55
recovery ladder, and the failure classes that strand a run. Sourced from
66
`scripts/dogfood.sh`, `.claude/skills/dogfooding-tracedecay/SKILL.md`, and
77
the runtime source cited inline. Do not read `~/.tracedecay` directly to
88
verify this document — every claim below traces to a file in this repo.
99

10-
## 1. The boundary state file
11-
12-
Path: `$HOME/.tracedecay/dogfood-migration-boundary.state` (overridable via
13-
`TRACEDECAY_DOGFOOD_PROFILE_DIR`). Written and read entirely by
14-
`scripts/dogfood.sh`; nothing else touches it.
15-
16-
Structural rules, enforced by `load_boundary_state()`
17-
(`scripts/dogfood.sh:287-380`):
18-
19-
- Must not be a symlink, must be a regular file, must be mode `0600`.
20-
- Currently written as `format=3` (`record_boundary_outcome()`,
21-
`scripts/dogfood.sh:430-470`). `format=2` is still accepted for reading
22-
(no `retained_binary_sha256` line) so a marker left by an older dogfood
23-
build doesn't hard-fail the next run.
24-
- Eight newline-delimited fields for `format=3`:
25-
1. `format=3`
26-
2. `attempt_id=<epoch-pid-random-random>`
27-
3. `outcome=<state>`
28-
4. `attempt_boundary=<reached|not-reached>`
29-
5. `old_binary_policy=<allowed|forbidden>`
30-
6. `managed_daemon=<state>`
31-
7. `retained_binary_sha256=<sha256|none>` — checksum of whatever is
32-
currently at `~/.local/bin/tracedecay` at write time, used to prove a
33-
later reader that the *retained* binary is the one the marker was
34-
written for.
35-
8. `checksum=<sha256 of lines 1-7>` — tamper/corruption check for the
36-
marker itself.
37-
38-
### outcome values and what each means
39-
40-
`marker_transition_is_valid()` (`scripts/dogfood.sh:255-276`) is the
41-
authoritative table of legal `outcome:boundary:policy:daemon` tuples. In
42-
practice:
43-
44-
| outcome | boundary | old_binary_policy | meaning |
45-
|---|---|---|---|
46-
| `preparing` | `not-reached` | either | a run is staging a new candidate binary; nothing has replaced the installed binary yet. |
47-
| `safe-rollback-complete` | `not-reached` | either | a run failed before crossing the boundary; `scripts/dogfood.sh` restored the previous installed/staged binaries (`cleanup_install()`, `scripts/dogfood.sh:610-681`, the non-`boundary_reached` branch). Safe to just rerun `cargo dogfood`. |
48-
| `post-update-starting` | `reached` | `forbidden` | the new binary is installed and `post-update --strict --mode dogfood-forward-only` is running. If the process dies here, the marker is left in this state and the **next** run must recover forward, never re-run an old binary. |
49-
| `forward-recovery-required` | `reached` | `forbidden` | a run crossed the boundary and then failed; `cleanup_install()` recorded this outcome instead of restoring the old binary, because doing so would let an older binary reopen a store a newer one may have already migrated. `managed_daemon` is `inactivity-pending`, `inactive`, or `inactivity-unproven` depending on whether the daemon could be proven stopped. |
50-
| `validated` | `reached` | `forbidden` | the boundary was crossed and `post-update` succeeded; `committed=1`. This is the terminal success state. |
51-
52-
Once `attempt_boundary=reached`, `old_binary_policy` is always forced to
53-
`forbidden` (`scripts/dogfood.sh:532-535`) regardless of what the marker
54-
said before — the script never trusts a "the old binary is fine to run"
55-
claim after the point of no return.
56-
57-
`marker_retained_binary_trusted` (set in `load_boundary_state()`,
58-
`scripts/dogfood.sh:373-379`) is `1` only when the SHA-256 of the file
59-
currently at `~/.local/bin/tracedecay` matches `retained_binary_sha256`
60-
from the marker. If someone replaced the installed binary out from under a
61-
pending marker, this flag drops to `0` and
62-
`require_inactive_recovery_before_preparing()`
63-
(`scripts/dogfood.sh:494-528`) refuses to execute it, falling back to the
64-
freshly built source binary for recovery instead.
10+
## 1. The forward-only binary policy
11+
12+
There is no on-disk state file tracking dogfood's progress across runs.
13+
Each `cargo dogfood` invocation is self-contained: it stages a candidate
14+
binary, atomically installs it, then runs `post-update --strict --mode
15+
dogfood-forward-only` (`scripts/dogfood.sh:454`). In-run bookkeeping lives
16+
entirely in shell variables local to the script process
17+
(`replacement_active`, `boundary_reached`, `committed`,
18+
`scripts/dogfood.sh:310-312`) — nothing is written to `~/.tracedecay` to
19+
remember where a prior attempt left off.
20+
21+
The forward-only guarantee — never let an older binary reopen a store a
22+
newer binary has already touched — is enforced by the binary itself at
23+
open time (a schema/version identity check), not by a marker file. An old
24+
binary either matches what a store expects or is refused there, typed and
25+
observable, rather than by a script-side ceremony.
26+
27+
### What `cleanup_install()` does on failure
28+
29+
`cleanup_install()` (`scripts/dogfood.sh:356-415`) is an `EXIT`/`HUP`/
30+
`INT`/`TERM` trap that only acts if a candidate replaced the installed
31+
binary during this invocation (`replacement_active=1`) and the run did not
32+
reach `committed=1`:
33+
34+
- **Before the new binary was installed** (`boundary_reached=0`): the
35+
script restores the pre-existing `installed_binary` and `staged_binary`
36+
from the copies it took at `scripts/dogfood.sh:436-447`
37+
(`restore_path()`, `scripts/dogfood.sh:316-326`). Safe to just rerun
38+
`cargo dogfood`.
39+
- **After the new binary was installed but before `post-update` finished**
40+
(`boundary_reached=1`, set at `scripts/dogfood.sh:453`): the script does
41+
**not** restore the previous binary — running an older binary against a
42+
store the new one may already have touched is exactly what the
43+
forward-only policy forbids. Instead it runs
44+
`post-update --mode dogfood-recover-inactive` against whichever binary is
45+
available (installed, then staged, then the candidate,
46+
`scripts/dogfood.sh:367-380`) to prove the managed daemon is stopped, then
47+
prints recovery instructions to stderr
48+
(`scripts/dogfood.sh:381-395`). Recover forward: fix or rebuild a newer
49+
binary and rerun `cargo dogfood`.
6550

6651
## 2. The backup contract
6752

6853
Two mutually exclusive modes, gated by `TRACEDECAY_DOGFOOD_BACKUP_PLAIN`
69-
(`scripts/dogfood.sh:538-557`):
54+
(`scripts/dogfood.sh:291-303`):
7055

7156
- **Checksummed backup (default).** `TRACEDECAY_DOGFOOD_BACKUP` must name a
7257
directory containing `backup-manifest.json`
7358
(`tracedecay migrate backup-profile --to <dir> --backup-id <id>`, CLI
74-
definition at `src/cli.rs:1218-1225`). Before crossing the boundary,
75-
`scripts/dogfood.sh:689-697` rehearses it with
59+
definition at `src/cli.rs:1218-1225`). Before installing the new binary,
60+
`scripts/dogfood.sh:423-431` rehearses it with
7661
`tracedecay migrate rehearse-profile-backup --backup <dir> --restore <tmp>`
7762
(`src/cli.rs:1227-1234`) — a full restore-and-verify into a throwaway
7863
directory, deleted immediately after. This is the safe, default path but
7964
re-reads and re-writes the entire profile twice.
8065
- **Plain backup** (`TRACEDECAY_DOGFOOD_BACKUP_PLAIN=1`, the owner-authorized
81-
fast path, `scripts/dogfood.sh:538-557`).
66+
fast path, `scripts/dogfood.sh:291-297`).
8267
`TRACEDECAY_DOGFOOD_BACKUP` must name a directory holding a plain `cp -a`
8368
profile copy at `<dir>/profile`, with `<dir>/profile/global.db` present.
84-
No manifest, no rehearsal (`scripts/dogfood.sh:687-697` skips the
85-
rehearsal step entirely when this flag is set). The script only checks
86-
that the copy *looks like* a profile (directory + `global.db` file
87-
exist) — it does not verify contents. This mode exists for profiles large
88-
enough that the checksummed path's two full read/write passes outlast the
69+
No manifest, no rehearsal (`scripts/dogfood.sh:423` skips the rehearsal
70+
step entirely when this flag is set). The script only checks that the
71+
copy *looks like* a profile (directory + `global.db` file exist) — it
72+
does not verify contents. This mode exists for profiles large enough
73+
that the checksummed path's two full read/write passes outlast the
8974
available maintenance window.
9075

9176
Naming a backup is optional (see below), but when one is named, both modes
9277
require it to already exist; `cargo dogfood` never creates one implicitly.
9378

9479
A backup is optional insurance, not a gate. With `TRACEDECAY_DOGFOOD_BACKUP`
95-
unset, dogfood proceeds and warns on stderr that it is running without one;
96-
the forward-only boundary still recovers forward (rung 1 below), but rungs 2
97-
and 3 have nothing to restore from. Naming a backup that is incomplete is
98-
still refused outright — that is a misconfiguration, not an opt-out.
80+
unset, dogfood proceeds and warns on stderr that it is running without one
81+
(`scripts/dogfood.sh:285-290`); the forward-only policy still recovers
82+
forward (rung 1 below), but rungs 2 and 3 have nothing to restore from.
83+
Naming a backup that is incomplete is still refused outright — that is a
84+
misconfiguration, not an opt-out.
9985

10086
## 3. The recovery ladder
10187

102-
In order, this is what takes a stuck forward-only boundary back to a clean
103-
`validated` state:
88+
In order, this is what takes a stuck forward-only run back to a clean
89+
install:
10490

105-
1. **Zero-writer proof.** Before touching anything, confirm no process
106-
still holds the managed daemon or an authority lease on the live
107-
stores. `scripts/dogfood.sh`'s own recovery path does this
91+
1. **Zero-writer proof.** Before touching anything further, confirm no
92+
process still holds the managed daemon or an authority lease on the
93+
live stores. `scripts/dogfood.sh`'s own failure path does this
10894
automatically via `post-update --mode dogfood-recover-inactive`
109-
(`require_inactive_recovery_before_preparing()`,
110-
`scripts/dogfood.sh:494-528`, and the `cleanup_install()` failure branch
111-
at `scripts/dogfood.sh:620-658`), which runs the retained/staged/source
112-
binary just far enough to prove the managed service is stopped and
113-
record `managed_daemon=inactive` (or `inactivity-unproven` if that
114-
proof itself fails).
95+
(`cleanup_install()`, `scripts/dogfood.sh:376-380`), which runs the
96+
retained/staged/candidate binary just far enough to prove the managed
97+
service is stopped.
11598
2. **Plain backup.** `TRACEDECAY_DOGFOOD_BACKUP_PLAIN=1` with
11699
`TRACEDECAY_DOGFOOD_BACKUP=<dir>` pointing at a `cp -a` copy already
117100
taken of `~/.tracedecay` (`<dir>/profile/global.db` present), per
118101
section 2.
119-
3. **Restore the pre-boundary `global.db`.** The registry — which projects
102+
3. **Restore the pre-failure `global.db`.** The registry — which projects
120103
are enrolled, their storage locations, graph scopes, artifacts — lives
121104
inside `global.db` and does not need to be reconstructed from scratch;
122105
it is *in* the restored file. What actually matters is that enrollment
@@ -145,10 +128,9 @@ In order, this is what takes a stuck forward-only boundary back to a clean
145128
`diff_registry_reconstruction_report`) is the read-only way to confirm
146129
the registry now matches on-disk reality before moving on.
147130
5. **Rerun `cargo dogfood`.** Once the daemon is proven inactive, the
148-
profile is backed up, `global.db` is consistent (restored or
149-
reconstructed), and `tracedecay init` has rebuilt the derived graph,
150-
`cargo dogfood` runs the normal flow end to end and should reach
151-
`outcome=validated`.
131+
profile is backed up, and `global.db` is consistent (restored or
132+
reconstructed), `cargo dogfood` runs the normal flow end to end and
133+
should install and validate cleanly.
152134

153135
## 4. Failure classes that strand a run
154136

@@ -176,8 +158,8 @@ In order, this is what takes a stuck forward-only boundary back to a clean
176158
Authority-invariant audits page with a keyset cursor at
177159
`AUDIT_PAGE_ROWS = 128` (`invariants.rs:49`) and observation scans at
178160
`OBSERVATION_AUDIT_PAGE_ROWS = 48` (`invariants.rs:52-58`) instead of one
179-
unbounded `SELECT`. The migration path uses the same pattern —
180-
`MIGRATION_QUERY_PAGE_ROWS = 256`
161+
unbounded `SELECT`. The profile-consolidation path uses the same
162+
pattern — `MIGRATION_QUERY_PAGE_ROWS = 256`
181163
(`crates/tracedecay-migrate/src/hermes/copy.rs:14`) drives keyset-paged
182164
`SELECT rowid, ... WHERE rowid > ?1 ... ORDER BY rowid LIMIT ?2` queries
183165
(`crates/tracedecay-migrate/src/hermes/resolution.rs:225-252` and

0 commit comments

Comments
 (0)