docs(encryption): address PR707 round-4 P1x3 (3-phase rollout, full-DEK resync, index-gated framing)

P1 (codex on 1b8613f line 645): old-binary tombstone test data[0] != 0 reads encrypted live values (encryption_state in bits 1-2 of byte 0) as tombstones and silently drops them. Convert §7.1 from 2-phase to 3-phase: Phase 0 = capability rollout (new binary writes cleartext, advertises encryption_capable in heartbeats); Phase 1 = enable-storage-envelope cluster-flag flip after membership-snapshot capability check; Phase 2 = enable-raft-envelope under the same gate. Per-node --encryption-enabled is now a capability assertion, not a behaviour trigger.

P1 (codex on 1b8613f line 586): §5.5 recovery rewrapped only the active DEK, leaving intermediate unretired DEKs missing on a node that missed multiple rotations and silently breaking historical reads. Recovery now rewraps every unretired DEK (active + retiring) in one apply; resync-sidecar replays all rotation/rewrap-deks entries, not just the most recent.

P1 (codex on 1b8613f line 847): Phase 2 first-byte framing tag (0x00 cleartext / 0x01 envelope) collided with kv/fsm.go existing tag space (0x00 single, 0x01 batch, 0x02 HLC-lease) and would misroute pre-cutover batch entries during WAL replay. Replace with raft-log-index gating: sidecar persists raft_envelope_cutover_index; entries below it use the legacy first-byte tags, entries at-or-above unwrap through the raft envelope. Snapshot headers carry the cutover index so a follower joining mid-Phase-2 reconstructs the same boundary.

Sidecar example expanded with storage_envelope_active and raft_envelope_cutover_index. §9.1 startup refusal list adds ErrEnvelopeCutoverDivergence. §10 self-review lens 2 updated to call out the old-binary tombstone path and the full-DEK resync requirement as load-bearing. §6.6 admin gains enable-storage-envelope.

Author: bootjp

docs/design/2026_04_29_proposed_data_at_rest_encryption.md

@@ -358,6 +358,8 @@ Two-tier hierarchy:
   {
     "version": 1,
     "raft_applied_index": 184273,
+    "storage_envelope_active": true,
+    "raft_envelope_cutover_index": 184201,
     "active": { "storage": 305419896, "raft": 2596069104 },
     "keys": {
       "305419896":  { "purpose": "storage", "wrapped": "<base64>",
@@ -370,6 +372,15 @@ Two-tier hierarchy:
   }
   ```
 
+  - `storage_envelope_active` mirrors the §7.1 Phase 1 cluster
+    flag once the FSM has seen the `enable-storage-envelope`
+    entry.
+  - `raft_envelope_cutover_index` records the apply index at
+    which §4.2 became active for this cluster. `0` means
+    Phase 2 has not started; any non-zero value is the
+    dispatch boundary used by FSM apply and WAL replay to
+    decide which Raft entries are envelopes.
+
   - **`key_id` is a 32-bit unsigned integer**, the same value that
     appears in the §4.1 envelope `key_id` field. It is generated by
     a CSPRNG draw on the leader at the moment a new DEK is created;
@@ -581,16 +592,32 @@ To detect and repair this:
    sidecar's index AND the gap covers any rotation entries, the
    node refuses to start with `ErrSidecarBehindRaftLog`,
    pointing at the operator runbook.
-3. Recovery is **automatic on the leader** (it re-proposes a
-   "rewrap current active DEK" entry that brings every node's
-   sidecar back in sync) and **manual on a stuck follower**
-   (operator runs `elastickv-admin encryption resync-sidecar`
-   which replays the rotation entries from the Raft log into the
-   local sidecar, then exits). Refusing to start is deliberate:
-   silently serving with a stale sidecar would let the node
-   write under an old `key_id` while peers write under a new one,
-   which is exactly the split-brain key state DEK rotation is
-   designed to prevent.
+3. Recovery rewraps **every unretired DEK**, not just the active
+   one. A node that missed multiple rotations needs *every*
+   intermediate `key_id` to decrypt MVCC history that still
+   carries those IDs (per §5.4 retirement criterion #4: a DEK
+   stays loaded until both the rewrite cursor has passed the
+   keyspace AND `minRetainedTS` has advanced past the youngest
+   `commit_ts` written under that DEK). Recovery comes in two
+   shapes:
+   - **Automatic on the leader** — the leader's keystore already
+     holds every unretired DEK in memory. It re-proposes a
+     `rewrap-deks` entry that serialises **all** unretired
+     wrapped DEKs (active + retiring), bringing every node's
+     sidecar to the full set in one apply.
+   - **Manual on a stuck follower** — operator runs
+     `elastickv-admin encryption resync-sidecar`. The command
+     replays *all* rotation and `rewrap-deks` entries between
+     the sidecar's `raft_applied_index` and the FSM's applied
+     index into the local sidecar (not just the most recent
+     one), then exits. Replaying only the active DEK would
+     leave intermediate `key_id`s missing and silently break
+     historical reads on that node — explicitly rejected.
+   Refusing to start until recovery completes is deliberate:
+   silently serving with an incomplete sidecar would let the
+   node write under one `key_id` while failing to decrypt
+   historical reads under another, which is exactly the
+   split-brain key state DEK rotation is designed to prevent.
 4. The reverse case — sidecar ahead of the raft log — cannot
    happen because the sidecar is only written from inside an FSM
    apply; an apply implies the entry is already in the Raft log.
@@ -726,11 +753,16 @@ elastickv-admin encryption rewrap-deks
 elastickv-admin encryption rewrite --rate=10MiB/s
 elastickv-admin encryption retire-dek --key-id=<uint32>
 elastickv-admin encryption resync-sidecar    # §5.5 follower repair
-elastickv-admin encryption enable-raft-envelope
-                                              # §7.1 Phase 2 cutover;
+elastickv-admin encryption enable-storage-envelope
+                                              # §7.1 Phase 1 cutover;
                                               # refuses unless every
                                               # voting member reports
                                               # encryption_capable
+elastickv-admin encryption enable-raft-envelope
+                                              # §7.1 Phase 2 cutover;
+                                              # same capability gate;
+                                              # records cutover index
+                                              # in the sidecar
 elastickv-admin encryption disable           # refuses; documents the
                                               # dump-and-reload path
 elastickv-admin backup verify --backup-dir=...
@@ -784,75 +816,105 @@ envelope) is impossible by construction — the header layout is
 fixed and the encryption state is read before the value bytes are
 ever interpreted.
 
-#### Rolling enablement: two phases, not one
-
-The §4.1 storage envelope and the §4.2 raft envelope have very
-different rollout constraints:
-
-- **§4.1 (storage envelope)** is per-version, dispatched on the
-  per-MVCC-version `encryption_state` bit. A node that has not yet
-  been upgraded continues to serve old cleartext versions; only
-  upgraded nodes write encrypted versions. Mixed-mode is safe.
-- **§4.2 (raft envelope)** wraps **every** committed Raft entry's
-  `Data []byte`. **Every replica must be able to decrypt every
-  committed entry** because Raft apply is deterministic and runs
-  on every node — there is no "skip this entry on this node"
-  escape hatch. If an upgraded leader proposes an encrypted
-  entry while a non-upgraded follower is still in the cluster,
-  the follower's `Apply` fails for every such entry. The
-  follower stops making progress, drifts behind on commit index,
-  and eventually triggers leader-side throttling.
-
-This rules out enabling §4.2 at the same instant as §4.1. The
-rollout is therefore explicitly two-phase, controlled by separate
-flags / Raft-replicated cluster state:
-
-**Phase 1 — Storage envelope only.**
-
-1. Operator provisions the KEK in the KMS.
-2. Operator restarts each node with `--encryption-enabled --kekUri=...`.
-   Restart is rolling; mixed-mode is safe under §4.1 because each
-   MVCC version carries its own `encryption_state` bit and reads
-   dispatch on that bit, not on the value bytes.
-3. Upgraded nodes write encrypted MVCC versions (`encryption_state
-   = 0b01`). Old MVCC versions retain `encryption_state = 0b00`
-   and continue to be returned as cleartext.
-4. Raft proposals during Phase 1 carry **cleartext** `Data []byte`
-   — i.e., §4.2 is not yet active. Lookup keys and operation tags
-   in the WAL are still cleartext during this window.
-5. Operator runs `elastickv-admin encryption rewrite` to convert
-   all `encryption_state = 0b00` MVCC versions in place
-   (per §5.4). When `encryption status --verify` reports zero
-   cleartext versions across every node AND `minRetainedTS` has
-   advanced past the youngest cleartext `commit_ts`, Phase 1 is
-   complete.
-
-**Phase 2 — Raft envelope.**
-
-6. Operator runs `elastickv-admin encryption enable-raft-envelope`.
-   The admin client RPCs into the leader, which:
-   - Verifies via the route catalog / membership snapshot that
-     **every** voting member of every Raft group has reported
-     `encryption_capable = true` in its periodic heartbeat
-     metadata (a new field). Refuses to proceed otherwise.
-   - Proposes a single Raft entry with the cluster-wide flag
-     `raft_envelope_active = true`. This entry itself is sent
-     **cleartext** so non-upgraded replicas (defensive: should
-     not exist after step 6's check, but treated as a safety
-     net) can still apply it.
-7. From the apply index of that entry onward, every leader
+#### Rolling enablement: three phases, not one
+
+Both the §4.1 storage envelope and the §4.2 raft envelope create
+mixed-binary safety problems that have to be ruled out **before**
+any encrypted byte hits disk. The rollout is therefore explicitly
+three-phase:
+
+**Why an "obvious" rolling restart with `--encryption-enabled` is
+unsafe.** A naive plan — flip the flag node-by-node and rely on
+the per-version `encryption_state` bit — is broken at two layers:
+
+- *At the storage layer:* the `lsm_store.go` value header today is
+  9 bytes `[tombstone(1)] [expireAt(8)]`, and `decodeValue` reads
+  the first byte as `tombstone := data[0] != 0`. The §6.2 plan
+  packs `encryption_state` into bits 1–2 of that same byte. An
+  encrypted live value (`tombstone=0`, `encryption_state=0b01`)
+  becomes `data[0] = 0b00000010 = 2`. A node still on the old
+  binary that ingests this value via Raft replication or snapshot
+  catch-up reads `data[0] != 0` → "tombstone" → drops the value
+  from reads. Silent data loss, not a clean refusal.
+- *At the Raft layer:* §4.2 wraps the entire `Data []byte` per
+  Raft entry. Apply is deterministic across replicas; a
+  non-upgraded follower without the KEK fails every
+  raft-envelope entry on apply, drifts behind, and is eventually
+  removed.
+
+Both problems share one fix: **gate the encryption-active state on
+a Raft-replicated cluster flag, not on a per-node startup flag.**
+The cluster-flag flip happens only after a membership-snapshot
+check confirms every voting member is running an
+encryption-capable binary. The per-node `--encryption-enabled`
+flag becomes a *capability* assertion, not a *behaviour* trigger.
+
+**Phase 0 — Capability rollout.**
+
+0. Operator provisions the KEK in the KMS.
+1. Operator restarts each node with `--encryption-enabled
+   --kekUri=...`. Rolling restart is safe in this phase because
+   the new binary still writes cleartext values (storage
+   envelope is gated on the cluster flag below) and proposes
+   cleartext Raft entries (raft envelope is gated on a separate
+   flag in Phase 2).
+2. Each upgraded node advertises `encryption_capable = true` in
+   its periodic Raft heartbeat metadata (a new field on the
+   peer-metadata payload added by `peer_metadata.go`). The flag
+   is purely informational at this point.
+3. When the membership snapshot of every Raft group shows
+   `encryption_capable = true` for every voting member, Phase 0
+   is complete. `encryption status` reports the capability
+   coverage so the operator can wait for it before moving on.
+
+**Phase 1 — Storage envelope cluster-flag flip.**
+
+4. Operator runs `elastickv-admin encryption enable-storage-envelope`.
+   The leader rechecks the membership-snapshot capability gate
+   from step 3 and then proposes a single Raft entry with the
+   cluster-wide flag `storage_envelope_active = true`. This
+   entry's `Data []byte` is the **legacy framing** (i.e., the
+   existing `kv/fsm.go` first-byte tag space; see Phase 2's note
+   below) so every replica — even one that somehow missed the
+   capability advert — can still decode and apply it.
+5. From the apply index of that flag entry onward, every storage
+   layer Put writes `encryption_state = 0b01` and the §4.1
+   envelope. MVCC versions written before that index keep
+   `encryption_state = 0b00` (cleartext) and are read back through
+   the dispatch path; mixing within a single key is safe per §5.4.
+6. Operator runs `elastickv-admin encryption rewrite` to convert
+   the remaining cleartext MVCC versions in place (per §5.4).
+   Phase 1 is complete when `encryption status --verify` reports
+   zero `encryption_state = 0b00` versions across every node
+   (excluding tombstones) AND `minRetainedTS` has advanced past
+   the youngest cleartext `commit_ts`.
+
+**Phase 2 — Raft envelope cluster-flag flip.**
+
+7. Operator runs `elastickv-admin encryption enable-raft-envelope`.
+   The leader rechecks the same capability gate, then proposes a
+   single Raft entry with the cluster-wide flag
+   `raft_envelope_active = true`. This entry itself uses the
+   legacy first-byte tag space so it remains decodable by any
+   replica that has applied entries up to it.
+8. From the apply index of that flag entry onward, every leader
    wraps new proposal `Data []byte` with the raft DEK (§4.2).
-   Replicas dispatch on a 1-byte format tag at the start of the
-   `Data` payload — `0x00` = cleartext (pre-flag entry, found
-   only in WAL/snapshot history), `0x01` = raft envelope. The
-   discriminator lives **at the start of the application-level
-   payload**, never as the first byte of an arbitrary user
-   value, so the §7.1 byte-collision argument does not apply
-   here: every value the storage layer sees is opaque bytes
-   from a known framing, not raw user input.
-8. Snapshots taken during Phase 2 carry the new flag in their
-   metadata header so a fresh follower joining mid-Phase-2 sees
-   the right framing for every entry it will ever receive.
+   **There is no in-band format tag in the proposal payload.**
+   Replicas dispatch on the **Raft log index** of the entry
+   relative to the persisted `raft_envelope_cutover_index` (the
+   apply index of the flag entry, recorded in the local sidecar
+   on apply). Entries below the cutover index are decoded via
+   the existing first-byte tag space (`0x00` single-request,
+   `0x01` batch, `0x02` HLC-lease, etc.); entries at-or-above
+   the cutover index are unwrapped through the raft envelope
+   first. This explicitly avoids re-using `kv/fsm.go`'s existing
+   first-byte tag values for the "is this an envelope?"
+   discriminator — that re-use would make pre-cutover batch
+   entries (`0x01`) ambiguous during WAL replay.
+9. Snapshots taken during Phase 2 carry the cutover index in
+   their metadata header so a fresh follower joining
+   mid-Phase-2 reconstructs the same dispatch boundary on
+   ingest.
 
 #### Why §4.2 cannot be turned off again
 
@@ -996,6 +1058,12 @@ The process refuses to start if any of the following hold:
 - The encryption package's startup membership check fails to
   resolve a stable 16-bit `node_id` (the §4.1 nonce construction
   needs one; without it nonce uniqueness cannot be guaranteed).
+- The local sidecar's `raft_envelope_cutover_index` disagrees
+  with the value carried in the most-recent ingested snapshot
+  header (`ErrEnvelopeCutoverDivergence`). This catches a node
+  that joined mid-Phase-2 from a snapshot taken before Phase 2
+  was enabled, then later replayed Raft entries that crossed
+  the cutover; resolved by `encryption resync-sidecar`.
 
 Each refusal logs a single, unambiguous error pointing at the
 relevant flag and runbook section.
@@ -1081,14 +1149,23 @@ eventual code change.
    unaffected because the bytes are already ciphertext; the
    receiving node's keystore must contain the relevant DEK before
    it can read the ingested data, which is guaranteed by the Raft
-   ordering of the rotation entry. The two-phase rollout in §7.1 is
-   the load-bearing piece for "node not yet upgraded" cases:
-   Phase 1 keeps Raft proposals cleartext so a non-upgraded
-   follower can still apply, and Phase 2 only flips on after a
-   membership-snapshot check confirms every voting member is
-   encryption-capable. Skipping the membership check would let
-   one upgraded leader produce raft envelopes that lock every
-   non-upgraded follower out of apply.
+   ordering of the rotation entry. The three-phase rollout in §7.1
+   is the load-bearing piece for "node not yet upgraded" cases:
+   Phase 0 (capability rollout) gets every node onto the new
+   binary while still writing cleartext so the storage-layer
+   `data[0] != 0` tombstone test on old code can never see a
+   value with `encryption_state` bits set; Phase 1 (storage
+   cluster-flag) only flips after a membership-snapshot check
+   confirms every voting member is `encryption_capable`; Phase 2
+   (raft cluster-flag) reuses the same gate. The membership
+   check is **not** advisory — skipping it would let one
+   upgraded leader produce raft envelopes that lock every
+   non-upgraded follower out of apply, or produce
+   `encryption_state = 0b01` values that an old follower
+   silently drops as tombstones. Sidecar recovery in §5.5 is
+   correspondingly load-bearing for the "missed multiple
+   rotations" case: replaying only the active DEK would strand
+   intermediate `key_id`s and silently break historical reads.
 
 3. **Performance.** AES-NI puts encryption CPU below the existing
    FSM apply CPU. The compress pass recovers most of Pebble's lost