Skip to content

Commit 24e5538

Browse files
bootjpbootjp
authored
feat(encryption): Stage 6D-6c-1 - Applier in-memory accessors for storage envelope state (#821)
## Summary - Add `Applier.ActiveStorageKeyID() (uint32, bool)` and `Applier.StorageEnvelopeActive() bool` backed by `atomic.Uint32` / `atomic.Bool`. These are the per-Put closures main.go will thread into `store.WithEncryption` and `store.WithStorageEnvelopeGate` in 6D-6c-2; a `ReadSidecar`-on-every-Put would serialise the hot path through a JSON parse + fsync barrier. - Coherence kept by **durable write-then-cache** ordering: `NewApplier` primes from `ReadSidecar` on construction, and `writeBootstrapSidecar` / `writeRotationSidecar` / the fresh-success branch of `applyEnableStorageEnvelope` call `refreshActiveStateCache(sc)` AFTER `WriteSidecar` succeeds. The §2.1 #3 stale-DEKID and §2.1 #4 already-active no-op branches intentionally skip the refresh — they don't change the mirrored fields. - Operator-inert until 6D-6c-2 wires the method values into the storage layer; 6D-6c-3 then wires the capability fan-out closure and the e2e integration test. Design doc updated: [`docs/design/2026_05_18_partial_6d_enable_storage_envelope.md`](docs/design/2026_05_18_partial_6d_enable_storage_envelope.md) tracks 6D-6c-1 as shipped and 6D-6c-2 / 6D-6c-3 as open. ## Why an in-memory mirror The §6.2 storage gate and the §4.1 active storage DEK are read on the storage hot path. The sidecar JSON is the durable source of truth, but reading it per Put would dominate latency. `atomic.Uint32` + `atomic.Bool` give a wait-free single-load read with no allocations and no syscalls. The two atomics are deliberately independent (not snapshot-atomic together): the storage layer consults them independently — `ActiveStorageKeyID` gates "do we encrypt at all" and `StorageEnvelopeActive` gates "do we wrap in the §4.1 envelope". The cross-invariant "envelope-active ⇒ DEK exists" is enforced by apply ordering (cutover requires a bootstrapped sidecar), not by the cache. ## Crash recovery Refresh runs AFTER the durable WriteSidecar. A crash between fsync and atomic store is benign — disk-truth wins and the next startup's `NewApplier` prime re-syncs the cache. A corrupt sidecar at construction time surfaces back to the caller so a misconfigured node fails to start instead of silently serving stale-zero state. ## Self-review (5 lenses) 1. **Data loss** — refresh is purely additive after the existing fsync; no write path shortened, no error suppressed. Corrupt-sidecar reads at construction fail the start instead of silently zeroing the cache. 2. **Concurrency / distributed failures** — atomic primitives + a `-race`-clean concurrent-reads stress test (8 readers × 2000 reads against a single applier goroutine). Two independent atomics is fine because the storage layer consults them independently and the cross-invariant is enforced by apply ordering. 3. **Performance** — hot Put path drops from JSON parse + potential fsync to a single atomic load (~ns vs ~10–100µs). 4. **Data consistency** — durable write-then-cache ordering everywhere; no-op apply branches skip the redundant refresh, keeping the invariant "cache changes only when the mirrored sidecar field changes" explicit. §2.1 / §6.4 semantics unchanged. 5. **Test coverage** — 5 functional tests (pre-bootstrap, post-bootstrap, post-rotate-dek, post-cutover, primed-from-existing-sidecar) + 1 race-stress test, all via the public API. ## Test plan - [x] `go test -race ./internal/encryption/...` - [x] `go test -race ./internal/... ./store/...` - [x] `golangci-lint --config=.golangci.yaml run ./internal/encryption/... ./store/...` - [ ] CI green <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added comprehensive test coverage for storage encryption state tracking across initialization, operational phases, startup recovery, and concurrent access scenarios. * **Chores** * Updated design documentation to reflect current development milestone progress. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/bootjp/elastickv/pull/821?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 32d8b93 + d547b32 commit 24e5538

3 files changed

Lines changed: 669 additions & 12 deletions

File tree

‎docs/design/2026_05_18_partial_6d_enable_storage_envelope.md‎

Lines changed: 44 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
| Field | Value |
44
|---|---|
5-
| Status | partial — 6D-1 (doc), 6D-2 (startup guards), 6D-3 (capability fan-out helper), 6D-4 (cutover wire + apply dispatch), 6D-5 (storage-layer toggle), 6D-6a (EnableStorageEnvelope server method), 6D-6b (CLI subcommand) shipped; 6D-6c (main.go wiring + integration test) remain |
5+
| Status | partial — 6D-1 (doc), 6D-2 (startup guards), 6D-3 (capability fan-out helper), 6D-4 (cutover wire + apply dispatch), 6D-5 (storage-layer toggle), 6D-6a (EnableStorageEnvelope server method), 6D-6b (CLI subcommand), 6D-6c-1 (Applier in-memory accessors) shipped; 6D-6c-2 (main.go cipher + gate wiring) and 6D-6c-3 (capability fan-out closure + e2e integration test) remain |
66
| Date | 2026-05-18 |
77
| Parent design | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) |
88
| Blockers (now satisfied) | 6B (KEK plumbing), 6C-1 / 6C-2 (startup guards), 6C-2d (`ErrSidecarBehindRaftLog` wiring) |
@@ -71,14 +71,51 @@
7171
misconfigured shell variable fails fast before the round-trip;
7272
the server re-validates as the source of truth.
7373

74+
- **6D-6c-1** (Applier in-memory accessors + shared StateCache)
75+
— new exported `encryption.StateCache` type backed by
76+
`atomic.Uint32` / `atomic.Bool` mirrors of `sidecar.Active.Storage`
77+
and `sidecar.StorageEnvelopeActive`. The cache is a **process-
78+
wide singleton** (parallel to the shared `*Keystore`) threaded
79+
into every per-shard `Applier` via the new `WithStateCache`
80+
option. Multi-group encryption FSM entries apply on exactly
81+
one shard's leader, so per-Applier-private atomics would leave
82+
the remaining shards stuck with pre-apply values; the shared
83+
StateCache makes every shard's storage layer observe the update
84+
regardless of which shard ran the apply. Coherence with disk
85+
is maintained by durable write-then-cache ordering inside
86+
`writeBootstrapSidecar`, `writeRotationSidecar`, and the
87+
cutover fresh-success branch of `applyEnableStorageEnvelope`.
88+
`NewApplier` primes the cache from the sidecar on construction
89+
so the storage-layer per-Put closures (wired in 6D-6c-2)
90+
observe correct values before the FSM has replayed a single
91+
entry after restart. `Applier.ActiveStorageKeyID` /
92+
`Applier.StorageEnvelopeActive` remain as delegate methods for
93+
tests and single-applier callers; multi-shard wiring in
94+
6D-6c-2 must read via `cache.ActiveStorageKeyID` /
95+
`cache.StorageEnvelopeActive` directly. Operator-inert by
96+
itself — only consumed once main.go threads the cache methods
97+
into `store.WithEncryption` and `WithStorageEnvelopeGate` in
98+
6D-6c-2.
99+
74100
## Open milestones
75101

76-
- **6D-6c** — main.go production wiring: cipher + WithEncryption
77-
+ WithStorageEnvelopeGate threaded from the sidecar, plus the
78-
CapabilityFanout closure bound to the live Raft membership
79-
view. End-to-end integration test exercises a single-node
80-
cluster Bootstrap → EnableStorageEnvelope → Put → read-back-
81-
via-envelope.
102+
- **6D-6c-2** — main.go production wiring: build
103+
`encryption.NewCipher(keystore)` and construct a single
104+
`encryption.StateCache` at startup (parallel to the shared
105+
`*Keystore`). Thread the cache via `WithStateCache` into every
106+
per-shard `Applier` inside `buildShardGroups`, and pass
107+
`cache.ActiveStorageKeyID` / `cache.StorageEnvelopeActive`
108+
(NOT the per-shard `Applier` delegates) into
109+
`store.WithEncryption` + `store.WithStorageEnvelopeGate` for
110+
each shard's PebbleStore. Reading via the StateCache directly
111+
ensures every shard's storage layer sees the post-apply state
112+
regardless of which shard's leader accepted the encryption
113+
proposal.
114+
- **6D-6c-3** — main.go CapabilityFanout closure bound to the
115+
live Raft membership view (etcd engine route snapshot + admin
116+
client DialFunc), and end-to-end integration test exercising
117+
a single-node cluster Bootstrap → EnableStorageEnvelope →
118+
Put → read-back-via-envelope.
82119

83120
## 0. Why this doc exists
84121

‎internal/encryption/applier.go‎

Lines changed: 184 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package encryption
33
import (
44
"bytes"
55
"strconv"
6+
"sync/atomic"
67
"time"
78

89
"github.com/bootjp/elastickv/internal/encryption/fsmwire"
@@ -63,17 +64,127 @@ type WriterRegistryStore interface {
6364
// ErrKEKNotConfigured marker. Stage 6B will swap these for the
6465
// real KEK-unwrap + sidecar mutate + keystore install path.
6566
//
66-
// The Applier carries no in-memory state of its own; all state
67-
// lives in the supplied WriterRegistryStore. This keeps it safe
68-
// to construct once at FSM startup and share across the lifetime
69-
// of the process — no per-apply allocation, no locks, no leak
70-
// path for stale state across snapshot restore.
67+
// Apart from the shared StateCache pointer (see below), the
68+
// Applier carries no in-memory state of its own; durable state
69+
// lives in the supplied WriterRegistryStore and the on-disk
70+
// sidecar. The StateCache mirrors a small subset of sidecar
71+
// fields the storage hot path consults on every Put — kept
72+
// coherent by durable write-then-cache ordering inside each
73+
// apply path.
7174
type Applier struct {
7275
registry WriterRegistryStore
7376
kek KEKUnwrapper
7477
keystore *Keystore
7578
sidecarPath string
7679
now func() time.Time
80+
// stateCache is the process-shared mirror of the sidecar fields
81+
// the storage hot path consults on every Put. See StateCache for
82+
// the full contract; in short, a single instance is owned by
83+
// main.go (parallel to the shared *Keystore) and threaded into
84+
// every per-shard Applier via WithStateCache so that an apply
85+
// landing on shard A's FSM is immediately visible to shard B's
86+
// storage layer. Multi-group encryption applies always land on
87+
// exactly one shard's FSM (the one whose engine accepted the
88+
// proposal), so a per-Applier cache would leave the remaining
89+
// shards stuck with pre-apply atomic values.
90+
//
91+
// Never nil after NewApplier: when WithStateCache is omitted the
92+
// constructor installs a private instance so single-applier
93+
// callers and tests keep working unchanged.
94+
stateCache *StateCache
95+
}
96+
97+
// StateCache mirrors the sidecar fields the storage hot path needs
98+
// to consult on every Put. Two requirements drive its existence:
99+
//
100+
// 1. ReadSidecar-on-every-Put would serialise the hot path through
101+
// a JSON parse + fsync barrier. atomic.Uint32 / atomic.Bool give
102+
// a wait-free single-load read instead.
103+
//
104+
// 2. In a multi-group deployment, encryption FSM entries apply on
105+
// whichever shard's leader accepted the proposal — not on every
106+
// shard. The per-shard storage layers must still observe the
107+
// updated state, so the cache MUST be a process-shared singleton
108+
// rather than a per-Applier field. main.go constructs one
109+
// StateCache at startup (parallel to the shared *Keystore) and
110+
// threads it into every per-shard Applier via WithStateCache.
111+
//
112+
// Coherence with disk is maintained by **durable write-then-cache**
113+
// ordering: NewApplier primes the cache from ReadSidecar, and every
114+
// apply path calls RefreshFromSidecar AFTER WriteSidecar succeeds.
115+
// A crash between fsync and atomic store is benign because the next
116+
// process start re-primes from disk.
117+
//
118+
// Zero values match the pre-bootstrap posture (no active storage
119+
// DEK, envelope gate off) so a freshly-constructed StateCache is
120+
// safe to use before any apply or prime has run.
121+
type StateCache struct {
122+
// activeStorageDEKID mirrors sidecar.Active.Storage. Zero means
123+
// "not bootstrapped"; readers surface (0, false) and the storage
124+
// layer writes cleartext.
125+
activeStorageDEKID atomic.Uint32
126+
// storageEnvelopeActive mirrors sidecar.StorageEnvelopeActive
127+
// for the §6.2 cutover gate. Lifecycle:
128+
// - false at construction (or primed from disk if a previous
129+
// cutover already fired).
130+
// - flipped to true exactly once by applyEnableStorageEnvelope
131+
// on a fresh-success apply, AFTER WriteSidecar succeeds.
132+
// - never flipped back to false (the cutover is one-way per
133+
// §7.1 Phase 1; rotate-dek under the active envelope keeps
134+
// it true).
135+
storageEnvelopeActive atomic.Bool
136+
}
137+
138+
// NewStateCache returns a zero-initialised StateCache. The
139+
// pre-bootstrap posture (Active.Storage=0, StorageEnvelopeActive=false)
140+
// is the correct initial state; RefreshFromSidecar advances it to the
141+
// current sidecar values when one is supplied.
142+
func NewStateCache() *StateCache { return &StateCache{} }
143+
144+
// RefreshFromSidecar copies the relevant fields out of sc into the
145+
// atomic mirrors. Safe to call concurrently with reads; safe to
146+
// call from multiple goroutines (writers race to the same atomic
147+
// CAS path, but the only writer in production is the FSM apply
148+
// goroutine of the shard that accepted the encryption proposal).
149+
//
150+
// nil sc is a no-op: matches the pre-bootstrap posture where
151+
// ReadSidecar returns IsNotExist.
152+
func (c *StateCache) RefreshFromSidecar(sc *Sidecar) {
153+
if c == nil || sc == nil {
154+
return
155+
}
156+
c.activeStorageDEKID.Store(sc.Active.Storage)
157+
c.storageEnvelopeActive.Store(sc.StorageEnvelopeActive)
158+
}
159+
160+
// ActiveStorageKeyID returns the current sidecar.Active.Storage DEK
161+
// id. Signature matches store.ActiveStorageKeyID so main.go can pass
162+
// `cache.ActiveStorageKeyID` directly into `store.WithEncryption(...)`
163+
// as the per-Put activeKeyID closure. A non-zero id with ok=true
164+
// means the cluster has run BootstrapEncryption; zero with ok=false
165+
// means the cluster is still pre-bootstrap and the storage layer
166+
// should write cleartext.
167+
func (c *StateCache) ActiveStorageKeyID() (uint32, bool) {
168+
if c == nil {
169+
return 0, false
170+
}
171+
id := c.activeStorageDEKID.Load()
172+
return id, id != 0
173+
}
174+
175+
// StorageEnvelopeActive returns the in-memory mirror of
176+
// sidecar.StorageEnvelopeActive. Signature matches
177+
// store.StorageEnvelopeActive so main.go can pass
178+
// `cache.StorageEnvelopeActive` directly into
179+
// `store.WithStorageEnvelopeGate(...)` as the per-Put cutover gate.
180+
// Once true, the storage layer wraps every new version in the §4.1
181+
// envelope; flips exactly once per cluster lifetime when the §7.1
182+
// Phase 1 cutover entry applies.
183+
func (c *StateCache) StorageEnvelopeActive() bool {
184+
if c == nil {
185+
return false
186+
}
187+
return c.storageEnvelopeActive.Load()
77188
}
78189

79190
// KEKUnwrapper is the abstraction the Applier uses to recover
@@ -132,6 +243,21 @@ func WithNowFunc(now func() time.Time) ApplierOption {
132243
return func(a *Applier) { a.now = now }
133244
}
134245

246+
// WithStateCache installs a shared StateCache so that an apply
247+
// landing on this Applier (typically the per-shard Applier whose
248+
// FSM accepted the encryption proposal) updates atomics that every
249+
// other Applier in the process reads. main.go owns one StateCache
250+
// for the lifetime of the binary and threads the same pointer into
251+
// every per-shard Applier and into the storage-layer per-Put
252+
// closures.
253+
//
254+
// If WithStateCache is omitted, NewApplier installs a private
255+
// instance — preserves the single-applier ergonomics that tests
256+
// and pre-multi-shard callers rely on.
257+
func WithStateCache(c *StateCache) ApplierOption {
258+
return func(a *Applier) { a.stateCache = c }
259+
}
260+
135261
// NewApplier wires an Applier against the supplied registry store
136262
// plus optional KEK / Keystore / sidecar / clock dependencies.
137263
// Returns an error if registry is nil so misconfiguration is caught
@@ -162,9 +288,59 @@ func NewApplier(registry WriterRegistryStore, opts ...ApplierOption) (*Applier,
162288
if a.now == nil {
163289
return nil, errors.New("encryption: NewApplier: WithNowFunc(nil) overwrote default time.Now")
164290
}
291+
// Install a private StateCache when WithStateCache was not
292+
// supplied so the apply paths and accessors always have a
293+
// non-nil target. Tests rely on this; production main.go is
294+
// expected to thread a shared instance in.
295+
if a.stateCache == nil {
296+
a.stateCache = NewStateCache()
297+
}
298+
// Prime the in-memory accessors from the on-disk sidecar
299+
// (best-effort: a missing sidecar is the pre-bootstrap
300+
// posture and leaves both atomics at their zero values,
301+
// which is correct). The storage-layer closures may query
302+
// these atomics before the FSM has replayed a single entry
303+
// after restart, so the priming must happen at construction
304+
// rather than lazily on first apply. A read error (corrupt
305+
// JSON, bad version) surfaces back to the caller so a
306+
// misconfigured node fails to start instead of silently
307+
// running with stale-zero state.
308+
if a.sidecarPath != "" {
309+
switch sc, err := ReadSidecar(a.sidecarPath); {
310+
case err == nil:
311+
a.stateCache.RefreshFromSidecar(sc)
312+
case IsNotExist(err):
313+
// Pre-bootstrap; leave atomics at zero.
314+
default:
315+
return nil, errors.Wrap(err, "encryption: NewApplier: prime in-memory state from sidecar")
316+
}
317+
}
165318
return a, nil
166319
}
167320

321+
// StateCache returns the shared cache this Applier writes to on
322+
// every apply path. main.go wires one StateCache across all
323+
// per-shard Appliers via WithStateCache, but for callers that
324+
// constructed an Applier without supplying one this accessor
325+
// returns the privately-installed instance so tests can still
326+
// reach the atomics directly.
327+
func (a *Applier) StateCache() *StateCache { return a.stateCache }
328+
329+
// ActiveStorageKeyID delegates to the shared StateCache. Convenience
330+
// for tests and single-applier callers; multi-shard wiring should
331+
// prefer reading StateCache().ActiveStorageKeyID directly so the
332+
// closure target is independent of which shard's Applier received
333+
// the encryption apply.
334+
func (a *Applier) ActiveStorageKeyID() (uint32, bool) {
335+
return a.stateCache.ActiveStorageKeyID()
336+
}
337+
338+
// StorageEnvelopeActive delegates to the shared StateCache. Same
339+
// rationale as ActiveStorageKeyID above.
340+
func (a *Applier) StorageEnvelopeActive() bool {
341+
return a.stateCache.StorageEnvelopeActive()
342+
}
343+
168344
// bootstrapAndRotationConfigured reports whether WithKEK,
169345
// WithKeystore, and WithSidecarPath have all been supplied. The
170346
// three are an indivisible quorum for ApplyBootstrap /
@@ -513,6 +689,7 @@ func (a *Applier) writeBootstrapSidecar(raftIdx uint64, p fsmwire.BootstrapPaylo
513689
if err := WriteSidecar(a.sidecarPath, sc); err != nil {
514690
return errors.Wrap(err, "applier: write sidecar for bootstrap")
515691
}
692+
a.stateCache.RefreshFromSidecar(sc)
516693
return nil
517694
}
518695

@@ -773,6 +950,7 @@ func (a *Applier) applyEnableStorageEnvelope(raftIdx uint64, p fsmwire.RotationP
773950
if err := WriteSidecar(a.sidecarPath, sc); err != nil {
774951
return errors.Wrap(err, "applier: write sidecar for cutover")
775952
}
953+
a.stateCache.RefreshFromSidecar(sc)
776954
return nil
777955
}
778956

@@ -809,6 +987,7 @@ func (a *Applier) writeRotationSidecar(raftIdx uint64, p fsmwire.RotationPayload
809987
if err := WriteSidecar(a.sidecarPath, sc); err != nil {
810988
return errors.Wrap(err, "applier: write sidecar for rotation")
811989
}
990+
a.stateCache.RefreshFromSidecar(sc)
812991
return nil
813992
}
814993

0 commit comments

Comments
 (0)