|
1 | | -# Dogfood Migration-Boundary Recovery Runbook |
| 1 | +# Dogfood Recovery Runbook |
2 | 2 |
|
3 | | -How to recover `cargo dogfood` once it has crossed its forward-only migration |
4 | | -boundary. Documents the boundary state file format, the backup contract, the |
| 3 | +How to recover `cargo dogfood` if it fails partway through installing a new |
| 4 | +binary. Documents the forward-only binary policy, the backup contract, the |
5 | 5 | recovery ladder, and the failure classes that strand a run. Sourced from |
6 | 6 | `scripts/dogfood.sh`, `.claude/skills/dogfooding-tracedecay/SKILL.md`, and |
7 | 7 | the runtime source cited inline. Do not read `~/.tracedecay` directly to |
8 | 8 | verify this document — every claim below traces to a file in this repo. |
9 | 9 |
|
10 | | -## 1. The boundary state file |
11 | | - |
12 | | -Path: `$HOME/.tracedecay/dogfood-migration-boundary.state` (overridable via |
13 | | -`TRACEDECAY_DOGFOOD_PROFILE_DIR`). Written and read entirely by |
14 | | -`scripts/dogfood.sh`; nothing else touches it. |
15 | | - |
16 | | -Structural rules, enforced by `load_boundary_state()` |
17 | | -(`scripts/dogfood.sh:287-380`): |
18 | | - |
19 | | -- Must not be a symlink, must be a regular file, must be mode `0600`. |
20 | | -- Currently written as `format=3` (`record_boundary_outcome()`, |
21 | | - `scripts/dogfood.sh:430-470`). `format=2` is still accepted for reading |
22 | | - (no `retained_binary_sha256` line) so a marker left by an older dogfood |
23 | | - build doesn't hard-fail the next run. |
24 | | -- Eight newline-delimited fields for `format=3`: |
25 | | - 1. `format=3` |
26 | | - 2. `attempt_id=<epoch-pid-random-random>` |
27 | | - 3. `outcome=<state>` |
28 | | - 4. `attempt_boundary=<reached|not-reached>` |
29 | | - 5. `old_binary_policy=<allowed|forbidden>` |
30 | | - 6. `managed_daemon=<state>` |
31 | | - 7. `retained_binary_sha256=<sha256|none>` — checksum of whatever is |
32 | | - currently at `~/.local/bin/tracedecay` at write time, used to prove a |
33 | | - later reader that the *retained* binary is the one the marker was |
34 | | - written for. |
35 | | - 8. `checksum=<sha256 of lines 1-7>` — tamper/corruption check for the |
36 | | - marker itself. |
37 | | - |
38 | | -### outcome values and what each means |
39 | | - |
40 | | -`marker_transition_is_valid()` (`scripts/dogfood.sh:255-276`) is the |
41 | | -authoritative table of legal `outcome:boundary:policy:daemon` tuples. In |
42 | | -practice: |
43 | | - |
44 | | -| outcome | boundary | old_binary_policy | meaning | |
45 | | -|---|---|---|---| |
46 | | -| `preparing` | `not-reached` | either | a run is staging a new candidate binary; nothing has replaced the installed binary yet. | |
47 | | -| `safe-rollback-complete` | `not-reached` | either | a run failed before crossing the boundary; `scripts/dogfood.sh` restored the previous installed/staged binaries (`cleanup_install()`, `scripts/dogfood.sh:610-681`, the non-`boundary_reached` branch). Safe to just rerun `cargo dogfood`. | |
48 | | -| `post-update-starting` | `reached` | `forbidden` | the new binary is installed and `post-update --strict --mode dogfood-forward-only` is running. If the process dies here, the marker is left in this state and the **next** run must recover forward, never re-run an old binary. | |
49 | | -| `forward-recovery-required` | `reached` | `forbidden` | a run crossed the boundary and then failed; `cleanup_install()` recorded this outcome instead of restoring the old binary, because doing so would let an older binary reopen a store a newer one may have already migrated. `managed_daemon` is `inactivity-pending`, `inactive`, or `inactivity-unproven` depending on whether the daemon could be proven stopped. | |
50 | | -| `validated` | `reached` | `forbidden` | the boundary was crossed and `post-update` succeeded; `committed=1`. This is the terminal success state. | |
51 | | - |
52 | | -Once `attempt_boundary=reached`, `old_binary_policy` is always forced to |
53 | | -`forbidden` (`scripts/dogfood.sh:532-535`) regardless of what the marker |
54 | | -said before — the script never trusts a "the old binary is fine to run" |
55 | | -claim after the point of no return. |
56 | | - |
57 | | -`marker_retained_binary_trusted` (set in `load_boundary_state()`, |
58 | | -`scripts/dogfood.sh:373-379`) is `1` only when the SHA-256 of the file |
59 | | -currently at `~/.local/bin/tracedecay` matches `retained_binary_sha256` |
60 | | -from the marker. If someone replaced the installed binary out from under a |
61 | | -pending marker, this flag drops to `0` and |
62 | | -`require_inactive_recovery_before_preparing()` |
63 | | -(`scripts/dogfood.sh:494-528`) refuses to execute it, falling back to the |
64 | | -freshly built source binary for recovery instead. |
| 10 | +## 1. The forward-only binary policy |
| 11 | + |
| 12 | +There is no on-disk state file tracking dogfood's progress across runs. |
| 13 | +Each `cargo dogfood` invocation is self-contained: it stages a candidate |
| 14 | +binary, atomically installs it, then runs `post-update --strict --mode |
| 15 | +dogfood-forward-only` (`scripts/dogfood.sh:454`). In-run bookkeeping lives |
| 16 | +entirely in shell variables local to the script process |
| 17 | +(`replacement_active`, `boundary_reached`, `committed`, |
| 18 | +`scripts/dogfood.sh:310-312`) — nothing is written to `~/.tracedecay` to |
| 19 | +remember where a prior attempt left off. |
| 20 | + |
| 21 | +The forward-only guarantee — never let an older binary reopen a store a |
| 22 | +newer binary has already touched — is enforced by the binary itself at |
| 23 | +open time (a schema/version identity check), not by a marker file. An old |
| 24 | +binary either matches what a store expects or is refused there, typed and |
| 25 | +observable, rather than by a script-side ceremony. |
| 26 | + |
| 27 | +### What `cleanup_install()` does on failure |
| 28 | + |
| 29 | +`cleanup_install()` (`scripts/dogfood.sh:356-415`) is an `EXIT`/`HUP`/ |
| 30 | +`INT`/`TERM` trap that only acts if a candidate replaced the installed |
| 31 | +binary during this invocation (`replacement_active=1`) and the run did not |
| 32 | +reach `committed=1`: |
| 33 | + |
| 34 | +- **Before the new binary was installed** (`boundary_reached=0`): the |
| 35 | + script restores the pre-existing `installed_binary` and `staged_binary` |
| 36 | + from the copies it took at `scripts/dogfood.sh:436-447` |
| 37 | + (`restore_path()`, `scripts/dogfood.sh:316-326`). Safe to just rerun |
| 38 | + `cargo dogfood`. |
| 39 | +- **After the new binary was installed but before `post-update` finished** |
| 40 | + (`boundary_reached=1`, set at `scripts/dogfood.sh:453`): the script does |
| 41 | + **not** restore the previous binary — running an older binary against a |
| 42 | + store the new one may already have touched is exactly what the |
| 43 | + forward-only policy forbids. Instead it runs |
| 44 | + `post-update --mode dogfood-recover-inactive` against whichever binary is |
| 45 | + available (installed, then staged, then the candidate, |
| 46 | + `scripts/dogfood.sh:367-380`) to prove the managed daemon is stopped, then |
| 47 | + prints recovery instructions to stderr |
| 48 | + (`scripts/dogfood.sh:381-395`). Recover forward: fix or rebuild a newer |
| 49 | + binary and rerun `cargo dogfood`. |
65 | 50 |
|
66 | 51 | ## 2. The backup contract |
67 | 52 |
|
68 | 53 | Two mutually exclusive modes, gated by `TRACEDECAY_DOGFOOD_BACKUP_PLAIN` |
69 | | -(`scripts/dogfood.sh:538-557`): |
| 54 | +(`scripts/dogfood.sh:291-303`): |
70 | 55 |
|
71 | 56 | - **Checksummed backup (default).** `TRACEDECAY_DOGFOOD_BACKUP` must name a |
72 | 57 | directory containing `backup-manifest.json` |
73 | 58 | (`tracedecay migrate backup-profile --to <dir> --backup-id <id>`, CLI |
74 | | - definition at `src/cli.rs:1218-1225`). Before crossing the boundary, |
75 | | - `scripts/dogfood.sh:689-697` rehearses it with |
| 59 | + definition at `src/cli.rs:1218-1225`). Before installing the new binary, |
| 60 | + `scripts/dogfood.sh:423-431` rehearses it with |
76 | 61 | `tracedecay migrate rehearse-profile-backup --backup <dir> --restore <tmp>` |
77 | 62 | (`src/cli.rs:1227-1234`) — a full restore-and-verify into a throwaway |
78 | 63 | directory, deleted immediately after. This is the safe, default path but |
79 | 64 | re-reads and re-writes the entire profile twice. |
80 | 65 | - **Plain backup** (`TRACEDECAY_DOGFOOD_BACKUP_PLAIN=1`, the owner-authorized |
81 | | - fast path, `scripts/dogfood.sh:538-557`). |
| 66 | + fast path, `scripts/dogfood.sh:291-297`). |
82 | 67 | `TRACEDECAY_DOGFOOD_BACKUP` must name a directory holding a plain `cp -a` |
83 | 68 | profile copy at `<dir>/profile`, with `<dir>/profile/global.db` present. |
84 | | - No manifest, no rehearsal (`scripts/dogfood.sh:687-697` skips the |
85 | | - rehearsal step entirely when this flag is set). The script only checks |
86 | | - that the copy *looks like* a profile (directory + `global.db` file |
87 | | - exist) — it does not verify contents. This mode exists for profiles large |
88 | | - enough that the checksummed path's two full read/write passes outlast the |
| 69 | + No manifest, no rehearsal (`scripts/dogfood.sh:423` skips the rehearsal |
| 70 | + step entirely when this flag is set). The script only checks that the |
| 71 | + copy *looks like* a profile (directory + `global.db` file exist) — it |
| 72 | + does not verify contents. This mode exists for profiles large enough |
| 73 | + that the checksummed path's two full read/write passes outlast the |
89 | 74 | available maintenance window. |
90 | 75 |
|
91 | 76 | Naming a backup is optional (see below), but when one is named, both modes |
92 | 77 | require it to already exist; `cargo dogfood` never creates one implicitly. |
93 | 78 |
|
94 | 79 | A backup is optional insurance, not a gate. With `TRACEDECAY_DOGFOOD_BACKUP` |
95 | | -unset, dogfood proceeds and warns on stderr that it is running without one; |
96 | | -the forward-only boundary still recovers forward (rung 1 below), but rungs 2 |
97 | | -and 3 have nothing to restore from. Naming a backup that is incomplete is |
98 | | -still refused outright — that is a misconfiguration, not an opt-out. |
| 80 | +unset, dogfood proceeds and warns on stderr that it is running without one |
| 81 | +(`scripts/dogfood.sh:285-290`); the forward-only policy still recovers |
| 82 | +forward (rung 1 below), but rungs 2 and 3 have nothing to restore from. |
| 83 | +Naming a backup that is incomplete is still refused outright — that is a |
| 84 | +misconfiguration, not an opt-out. |
99 | 85 |
|
100 | 86 | ## 3. The recovery ladder |
101 | 87 |
|
102 | | -In order, this is what takes a stuck forward-only boundary back to a clean |
103 | | -`validated` state: |
| 88 | +In order, this is what takes a stuck forward-only run back to a clean |
| 89 | +install: |
104 | 90 |
|
105 | | -1. **Zero-writer proof.** Before touching anything, confirm no process |
106 | | - still holds the managed daemon or an authority lease on the live |
107 | | - stores. `scripts/dogfood.sh`'s own recovery path does this |
| 91 | +1. **Zero-writer proof.** Before touching anything further, confirm no |
| 92 | + process still holds the managed daemon or an authority lease on the |
| 93 | + live stores. `scripts/dogfood.sh`'s own failure path does this |
108 | 94 | automatically via `post-update --mode dogfood-recover-inactive` |
109 | | - (`require_inactive_recovery_before_preparing()`, |
110 | | - `scripts/dogfood.sh:494-528`, and the `cleanup_install()` failure branch |
111 | | - at `scripts/dogfood.sh:620-658`), which runs the retained/staged/source |
112 | | - binary just far enough to prove the managed service is stopped and |
113 | | - record `managed_daemon=inactive` (or `inactivity-unproven` if that |
114 | | - proof itself fails). |
| 95 | + (`cleanup_install()`, `scripts/dogfood.sh:376-380`), which runs the |
| 96 | + retained/staged/candidate binary just far enough to prove the managed |
| 97 | + service is stopped. |
115 | 98 | 2. **Plain backup.** `TRACEDECAY_DOGFOOD_BACKUP_PLAIN=1` with |
116 | 99 | `TRACEDECAY_DOGFOOD_BACKUP=<dir>` pointing at a `cp -a` copy already |
117 | 100 | taken of `~/.tracedecay` (`<dir>/profile/global.db` present), per |
118 | 101 | section 2. |
119 | | -3. **Restore the pre-boundary `global.db`.** The registry — which projects |
| 102 | +3. **Restore the pre-failure `global.db`.** The registry — which projects |
120 | 103 | are enrolled, their storage locations, graph scopes, artifacts — lives |
121 | 104 | inside `global.db` and does not need to be reconstructed from scratch; |
122 | 105 | it is *in* the restored file. What actually matters is that enrollment |
@@ -145,10 +128,9 @@ In order, this is what takes a stuck forward-only boundary back to a clean |
145 | 128 | `diff_registry_reconstruction_report`) is the read-only way to confirm |
146 | 129 | the registry now matches on-disk reality before moving on. |
147 | 130 | 5. **Rerun `cargo dogfood`.** Once the daemon is proven inactive, the |
148 | | - profile is backed up, `global.db` is consistent (restored or |
149 | | - reconstructed), and `tracedecay init` has rebuilt the derived graph, |
150 | | - `cargo dogfood` runs the normal flow end to end and should reach |
151 | | - `outcome=validated`. |
| 131 | + profile is backed up, and `global.db` is consistent (restored or |
| 132 | + reconstructed), `cargo dogfood` runs the normal flow end to end and |
| 133 | + should install and validate cleanly. |
152 | 134 |
|
153 | 135 | ## 4. Failure classes that strand a run |
154 | 136 |
|
@@ -176,8 +158,8 @@ In order, this is what takes a stuck forward-only boundary back to a clean |
176 | 158 | Authority-invariant audits page with a keyset cursor at |
177 | 159 | `AUDIT_PAGE_ROWS = 128` (`invariants.rs:49`) and observation scans at |
178 | 160 | `OBSERVATION_AUDIT_PAGE_ROWS = 48` (`invariants.rs:52-58`) instead of one |
179 | | - unbounded `SELECT`. The migration path uses the same pattern — |
180 | | - `MIGRATION_QUERY_PAGE_ROWS = 256` |
| 161 | + unbounded `SELECT`. The profile-consolidation path uses the same |
| 162 | + pattern — `MIGRATION_QUERY_PAGE_ROWS = 256` |
181 | 163 | (`crates/tracedecay-migrate/src/hermes/copy.rs:14`) drives keyset-paged |
182 | 164 | `SELECT rowid, ... WHERE rowid > ?1 ... ORDER BY rowid LIMIT ?2` queries |
183 | 165 | (`crates/tracedecay-migrate/src/hermes/resolution.rs:225-252` and |
|
0 commit comments