Skip to content

Commit ca0b4e9

Browse files
bootjpbootjp
authored
feat(snapshot-skip B3): cold-start restoreSnapshotState skip gate + CRC verifier + metrics (#934)
## Summary Implements **Branch 3** of the cold-start snapshot-restore skip optimisation designed in PR #910, building on PR #915 (B2 — `metaAppliedIndex` plumbing). This is the **user-visible perf win** of the series: after this lands, the engine restores the FSM only when the on-disk state is actually behind the snapshot pointer; the common case (local restart with a fresh fsm.db) collapses to a header read + CRC verification, skipping the multi-GiB body restore that PR #909's `HEALTH_TIMEOUT_SECONDS=300` band-aid was sized to absorb. ## Reading order (6 commits, designed for one-at-a-time review) | # | commit | scope | |---|---|---| | 1 | `8312f4ba` | `raftengine.SnapshotHeaderApplier` two-phase interface (`ParseSnapshotHeader` + `ApplySnapshotHeader`) | | 2 | `024eb43c` | `kvFSM` implements both phases + compile-time guard | | 3 | `933dab86` | `applyHeaderStateOnSkip` (3-step CRC, mirrors `openAndRestoreFSMSnapshot`) + `restoreSnapshotState` skip gate | | 4 | (folded into #3) | `fsmAlreadyAtIndex` + skip-gate wiring | | 5 | (next) | metrics + INFO log: `raftengine.ColdStartObserver` + `monitoring.ColdStartMetrics` + `Registry.ColdStartObserver()` | | 6 | `ce277e66` | 9 new tests (skip-gate behaviour ×5 + CRC failure modes ×3 + kvFSM contract ×1) | (Commit 5 was merged into the metrics commit; the actual git log is 5 commits.) ## What this does `restoreSnapshotState` branches on `decideSkipOutcome`: ```go decision, have := decideSkipOutcome(fsm, tok.Index) reportColdStart(obs, logger, decision, tok.Index, have) if decision == coldStartSkip { return applyHeaderStateOnSkip(fsm, fsmSnapPath(fsmSnapDir, tok.Index), tok.CRC32C) } return openAndRestoreFSMSnapshot(...) // unchanged ``` `decideSkipOutcome` returns one of 5 outcomes: | outcome | trigger | action | |---|---|---| | `skip` | `LastAppliedIndex >= snap.Index` | header-only path | | `execute` | `LastAppliedIndex < snap.Index` | full restore | | `fallback_not_reader` | FSM doesn't implement `AppliedIndexReader` | full restore | | `fallback_missing_meta` | meta key absent (pre-upgrade fsm.db) | full restore | | `fallback_read_err` | `LastAppliedIndex` returned an error | full restore | The three fallback paths preserve the strictly-additive guarantee from the design doc §4: over-restoring is always strictly safer than skipping incorrectly. Errors from `LastAppliedIndex` **do not abort cold start** — they collapse to fallback. ## CRC verification (mirrors `openAndRestoreFSMSnapshot`) `applyHeaderStateOnSkip` runs the same three-step safety contract as the full restore path. Failure of any step propagates a typed error WITHOUT mutating FSM state: | Step | Check | Error | |---|---|---| | 1 | `info.Size() < fsmMinFileSize` | `ErrFSMSnapshotTooSmall` | | 2 | `readFSMFooter` vs `tokenCRC` | `ErrFSMSnapshotTokenCRC` | | 3 | full-body `crc32.TeeReader` vs footer | `ErrFSMSnapshotFileCRC` | The FSM's `ApplySnapshotHeader` (HLC ceiling + Stage 8a cutover) only fires after all three pass. ## Metrics + INFO log `raftengine.ColdStartObserver` interface lives in the parent `raftengine` package so monitoring can implement it without circular import. `monitoring.Registry` gains `ColdStartObserver()`. The engine receives it through `OpenConfig.ColdStartObserver` — nil disables metrics, the skip itself still runs. Two Prometheus series: ``` elastickv_fsm_cold_start_restore_total{outcome,fallback_reason} elastickv_fsm_cold_start_applied_index_gap{outcome} ``` Plus a structured zap.Info log on every cold-start with fsm_applied / snapshot_index / gap and (for fallback) the reason enum. ## Design constraints honoured - **§4 strictly-additive fallback**: `decideSkipOutcome` collapses every uncertainty to a fallback variant; `LastAppliedIndex` errors do NOT abort cold start. - **§5 two-phase seam (round-7)**: `ParseSnapshotHeader` reads the v1/v2 header from a caller-supplied reader and drains the rest; `ApplySnapshotHeader` is pure assignment. Splits the parse from the side-effect so the engine can verify the CRC between them. - **§5 CRC mirroring (round-6)**: 3-step verification (size + footer-vs-tokenCRC + full-body-CRC) exactly matches `openAndRestoreFSMSnapshot`. The full-body CRC pass IS the slow part of this PR (~6s for a 6 GiB FSM at 1 GiB/s SSD read) — but still strictly cheaper than the restore it replaces, which also reads the file once AND writes a temp Pebble database via `restoreBatchLoopInto`. - **§9 observability**: three outcomes + gap label; the design's `not_reporter` label is named `not_reader` in the actual impl (matching the round-5 PR #915 rename). - **Non-Goals respected**: `Engine.applySnapshot` (engine.go:1641 InstallSnapshot hot path) is untouched. The TODO sentinel added in PR #915 round-6 still names B3 as the follow-up site for the peer-after-InstallSnapshot bump — that's not in scope here either. ## Test results ```text go vet ./internal/raftengine/etcd/ ./internal/raftengine/ ./kv/ ./store/ ./monitoring/ → exit 0 golangci-lint run ... (same packages) → 0 issues go test ./internal/raftengine/etcd/ ./kv/ ./store/ ./monitoring/ -short → ok ~57s total go test ./internal/raftengine/etcd/ -run 'TestSkipGate|TestApplyHeaderStateOnSkip' → ok 0.028s ``` ## What this does NOT do - **Does NOT change** `HEALTH_TIMEOUT_SECONDS=300`. That's Branch 4, gated on production data showing steady-state skip rate ≥ 90%. - **Does NOT** wire `Engine.applySnapshot` to populate `metaAppliedIndex` for peers receiving InstallSnapshot. The TODO sentinel from PR #915 round-6 still flags this; addressing it requires a separate design pass. - **Does NOT** add an idle-cluster integration test (`ELASTICKV_RAFT_SNAPSHOT_COUNT=10` end-to-end). The unit tests cover the gate decision matrix; an integration test would prove the codex round-3 P2 scenario is closed end-to-end in production-like setup, which is most valuable as a follow-up after B3 has soaked. ## Soak plan Same as B2: deploy and observe `elastickv_fsm_cold_start_restore_total{outcome="skipped"}` ramp up across cold starts. The design target is steady-state skip rate ≥ 90%; the per-outcome label distribution and gap distribution are the primary signals. After one release of soak, Branch 4 can lower `HEALTH_TIMEOUT_SECONDS`. ## Refs - PR #910 — design doc (rounds 1-7) - PR #915 — Branch 2 plumbing (merged); this PR depends on `metaAppliedIndex` being populated - PR #909 — `HEALTH_TIMEOUT_SECONDS` band-aid this series eventually obviates <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Cold-start lifecycle callbacks added and wired through the raft engine to report skip/execute/fallback outcomes and applied-index gaps. * Prometheus metrics for cold-start outcomes and applied-index gap. * Snapshot-header read/apply contract introduced; KV FSM implements header apply and volatile-entry classification to support header-only skips and volatile-only replay on duplicates. * **Tests** * Extensive unit/integration tests covering skip/execute/fallback decisions, header verification, volatile duplicate replay, and updated restore-call signatures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents aa4a7ba + 58db40c commit ca0b4e9

19 files changed

Lines changed: 1618 additions & 69 deletions

‎internal/raftengine/cold_start.go‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
package raftengine
2+
3+
// ColdStartObserver receives cold-start snapshot-restore lifecycle
4+
// events from restoreSnapshotState. Implementations live in the
5+
// monitoring package and wire to Prometheus counters/gauges; the
6+
// engine receives a value through OpenConfig and treats nil as
7+
// "no metrics emitted" (preserves the byte-for-byte cold-start
8+
// behaviour for tests and callers that do not wire monitoring).
9+
//
10+
// Three outcomes match the design's strictly-additive policy
11+
// (docs/design/2026_06_02_idempotent_snapshot_restore.md §9):
12+
//
13+
// - RestoreSkipped: the gate fired. `gap = haveAppliedIndex -
14+
// snapshot.Metadata.Index` (how far ahead the live store was).
15+
// This is the user-visible perf win.
16+
//
17+
// - RestoreExecuted: the gate did NOT fire because the live store
18+
// was genuinely stale (haveAppliedIndex < snapshot.Metadata.Index).
19+
// `gap = snapshot.Metadata.Index - haveAppliedIndex` (the work
20+
// the full restore re-did).
21+
//
22+
// - RestoreFallback: the strictly-additive fallback path — the
23+
// FSM did not expose AppliedIndexReader, LastAppliedIndex
24+
// reported the meta key missing, or it returned an error. The
25+
// full restore runs but the skip was never even attempted.
26+
// `reason` carries a stable short label so Prometheus can
27+
// surface why the optimisation could not engage:
28+
//
29+
// not_reader — FSM does not implement AppliedIndexReader
30+
// missing_meta — meta key absent (pre-upgrade fsm.db)
31+
// read_err — LastAppliedIndex returned an error
32+
//
33+
// Implementations MUST NOT block; the engine calls these on the
34+
// cold-start critical path. Treat all label/string arguments as
35+
// untrusted enum values from the engine's enumeration above.
36+
type ColdStartObserver interface {
37+
RestoreSkipped(snapIndex, haveAppliedIndex uint64)
38+
RestoreExecuted(snapIndex, haveAppliedIndex uint64)
39+
RestoreFallback(snapIndex uint64, reason string)
40+
}

‎internal/raftengine/etcd/cold_start_volatile_replay_test.go‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
package etcd
2+
3+
import (
4+
"bytes"
5+
"io"
6+
"sync/atomic"
7+
"testing"
8+
9+
"github.com/bootjp/elastickv/internal/raftengine"
10+
raftpb "go.etcd.io/raft/v3/raftpb"
11+
)
12+
13+
// volatileTagFakeFSM is a StateMachine that classifies payloads by
14+
// their leading byte: 0x02 (kv.raftEncodeHLCLease) is volatile, every
15+
// other tag is data-mutating. Mirrors the kvFSM contract closely
16+
// enough that the engine's cold-start duplicate guard can be tested
17+
// without pulling in the full kv package.
18+
type volatileTagFakeFSM struct {
19+
calls atomic.Int32
20+
lastPayload []byte
21+
}
22+
23+
func (f *volatileTagFakeFSM) Apply(data []byte) any {
24+
f.calls.Add(1)
25+
cp := make([]byte, len(data))
26+
copy(cp, data)
27+
f.lastPayload = cp
28+
return nil
29+
}
30+
31+
func (f *volatileTagFakeFSM) Snapshot() (Snapshot, error) { return nil, nil }
32+
func (f *volatileTagFakeFSM) Restore(_ io.Reader) error { return nil }
33+
34+
func (f *volatileTagFakeFSM) IsVolatileOnlyPayload(payload []byte) bool {
35+
return len(payload) > 0 && payload[0] == 0x02
36+
}
37+
38+
var _ raftengine.VolatileEntryClassifier = (*volatileTagFakeFSM)(nil)
39+
40+
// TestApplyNormalCommitted_VolatileEntryReplayedOnDuplicate pins the
41+
// codex P1 #934 round 7 fix: after the cold-start skip gate seeds
42+
// e.applied at the WAL committed tail, entries delivered by Raft at
43+
// indices <= e.applied that are volatile-only (HLC lease, tag 0x02)
44+
// MUST still reach fsm.Apply so the post-snapshot ceiling raise is
45+
// reconstructed. Data-mutating duplicates (any other tag) MUST NOT
46+
// reach fsm.Apply (OCC re-validation; ceiling regression). A
47+
// regression would either silently drop the lease (the bug) or
48+
// silently re-execute KV writes (idempotency violation).
49+
func TestApplyNormalCommitted_VolatileEntryReplayedOnDuplicate(t *testing.T) {
50+
t.Parallel()
51+
cases := []struct {
52+
name string
53+
tag byte
54+
wantApply bool
55+
}{
56+
{"volatile HLC lease (tag 0x02) replays past e.applied", 0x02, true},
57+
{"data-mutating single KV (tag 0x00) is dropped", 0x00, false},
58+
{"data-mutating batch KV (tag 0x01) is dropped", 0x01, false},
59+
}
60+
for _, c := range cases {
61+
t.Run(c.name, func(t *testing.T) {
62+
t.Parallel()
63+
fsm := &volatileTagFakeFSM{}
64+
e := newTestEngine(fsm, nil, nil)
65+
e.applied = 200
66+
67+
payload := []byte{c.tag, 0xde, 0xad, 0xbe, 0xef, 0x00, 0x00, 0x00, 0x00}
68+
entry := raftpb.Entry{
69+
Type: raftpb.EntryNormal,
70+
Index: 150, // <= e.applied → duplicate path
71+
Data: encodeProposalEnvelope(42, payload),
72+
}
73+
74+
if err := e.applyNormalCommitted(entry); err != nil {
75+
t.Fatalf("applyNormalCommitted: %v", err)
76+
}
77+
78+
got := fsm.calls.Load()
79+
switch {
80+
case c.wantApply && got != 1:
81+
t.Fatalf("volatile duplicate: fsm.Apply call count = %d, want 1 (lost in-memory effect)", got)
82+
case !c.wantApply && got != 0:
83+
t.Fatalf("data-mutating duplicate: fsm.Apply call count = %d, want 0 (re-applied; OCC will re-validate against post-tail state)", got)
84+
}
85+
86+
// In every case e.applied must NOT advance — the entry's
87+
// index is below the gate-seeded value and resolveProposal
88+
// is intentionally not called either.
89+
if e.applied != 200 {
90+
t.Fatalf("e.applied advanced to %d, want it pinned at 200 for duplicate-path entries", e.applied)
91+
}
92+
})
93+
}
94+
}
95+
96+
// TestApplyNormalCommitted_FreshEntryAlwaysAppliesAndAdvances pins the
97+
// non-duplicate path: entries past e.applied always reach fsm.Apply
98+
// regardless of the volatile/data classification, and e.applied
99+
// advances. Locks in the asymmetry — the classifier ONLY gates the
100+
// duplicate arm.
101+
func TestApplyNormalCommitted_FreshEntryAlwaysAppliesAndAdvances(t *testing.T) {
102+
t.Parallel()
103+
for _, tag := range []byte{0x00, 0x01, 0x02} {
104+
t.Run("tag=0x0"+string("0123"[tag])+"_fresh", func(t *testing.T) {
105+
t.Parallel()
106+
fsm := &volatileTagFakeFSM{}
107+
e := newTestEngine(fsm, nil, nil)
108+
e.applied = 100
109+
110+
entry := raftpb.Entry{
111+
Type: raftpb.EntryNormal,
112+
Index: 150,
113+
Data: encodeProposalEnvelope(7, []byte{tag, 0x99}),
114+
}
115+
116+
if err := e.applyNormalCommitted(entry); err != nil {
117+
t.Fatalf("applyNormalCommitted: %v", err)
118+
}
119+
if got := fsm.calls.Load(); got != 1 {
120+
t.Fatalf("fsm.Apply call count = %d, want 1 (fresh entry, all tags)", got)
121+
}
122+
if e.applied != 150 {
123+
t.Fatalf("e.applied = %d, want 150 (fresh entry must advance)", e.applied)
124+
}
125+
})
126+
}
127+
}
128+
129+
// TestApplyNormalCommitted_VolatileDuplicate_PostCutoverEncrypted pins
130+
// the post-Stage-8a cutover path: encrypted HLC lease entries past
131+
// e.applied MUST be decrypted FIRST, then classified as volatile, then
132+
// replayed for their in-memory effect. The wire-format reality is
133+
// that a post-cutover HLC lease's `payload[0]` is encrypted bytes;
134+
// only the cleartext (after WrapRaftPayload unwrap) carries the 0x02
135+
// tag, so the classifier must see the cleartext or the lease drops.
136+
// Claude #934 round 7 finding R7-F2 — pre-cutover-only coverage was
137+
// insufficient.
138+
func TestApplyNormalCommitted_VolatileDuplicate_PostCutoverEncrypted(t *testing.T) {
139+
t.Parallel()
140+
c, kid := raftCipherFixture(t)
141+
const cutover uint64 = 100
142+
fsm := &volatileTagFakeFSM{}
143+
e := newTestEngine(fsm, c, func() uint64 { return cutover })
144+
e.applied = 200
145+
146+
// HLC lease cleartext: tag 0x02 + 8-byte big-endian ceiling.
147+
plain := []byte{0x02, 0, 0, 0, 0, 0, 0, 0, 0x01}
148+
// Index above cutover → triggers WrapRaftPayload path inside
149+
// applyNormalEntry; index below e.applied → duplicate arm.
150+
entry := envelopeEntry(t, c, kid, 150, plain)
151+
152+
if err := e.applyNormalCommitted(entry); err != nil {
153+
t.Fatalf("applyNormalCommitted: %v", err)
154+
}
155+
if got := fsm.calls.Load(); got != 1 {
156+
t.Fatalf("encrypted volatile duplicate: fsm.Apply call count = %d, want 1 — decryption must run before classification or the lease drops", got)
157+
}
158+
if !bytes.Equal(fsm.lastPayload, plain) {
159+
t.Fatalf("FSM received %x, want cleartext %x — classifier must see post-decrypt bytes", fsm.lastPayload, plain)
160+
}
161+
if e.applied != 200 {
162+
t.Fatalf("e.applied advanced to %d, want pinned at 200 for duplicate-arm replay", e.applied)
163+
}
164+
}
165+
166+
// TestApplyNormalCommitted_DuplicateWithoutClassifier_DropsAll guards
167+
// the FSM-doesn't-opt-in path: a StateMachine that does NOT implement
168+
// VolatileEntryClassifier (existing FSMs, third-party engines) must
169+
// keep the pre-PR behavior — every duplicate is dropped, including
170+
// any that happen to be volatile. The strictly-additive opt-in keeps
171+
// the engine compatible with FSMs that have not been updated.
172+
func TestApplyNormalCommitted_DuplicateWithoutClassifier_DropsAll(t *testing.T) {
173+
t.Parallel()
174+
// fakeStateMachine (from encryption_test.go) does NOT implement
175+
// VolatileEntryClassifier — that absence is the contract under
176+
// test.
177+
fsm := &fakeStateMachine{}
178+
if _, ok := any(fsm).(raftengine.VolatileEntryClassifier); ok {
179+
t.Fatal("fakeStateMachine unexpectedly implements VolatileEntryClassifier; test contract broken")
180+
}
181+
e := newTestEngine(fsm, nil, nil)
182+
e.applied = 200
183+
184+
entry := raftpb.Entry{
185+
Type: raftpb.EntryNormal,
186+
Index: 150,
187+
Data: encodeProposalEnvelope(13, []byte{0x02, 0xff}), // would-be volatile
188+
}
189+
190+
if err := e.applyNormalCommitted(entry); err != nil {
191+
t.Fatalf("applyNormalCommitted: %v", err)
192+
}
193+
if got := fsm.calls.Load(); got != 0 {
194+
t.Fatalf("FSM without classifier: fsm.Apply call count = %d, want 0 (drop-all on duplicate, no opt-in)", got)
195+
}
196+
if e.applied != 200 {
197+
t.Fatalf("e.applied advanced unexpectedly: %d, want 200", e.applied)
198+
}
199+
}

‎internal/raftengine/etcd/encryption_test.go‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ func TestApplyNormalEntry_CutoverActive_NoCipher_FailsClosed(t *testing.T) {
133133
Index: cutover + 1,
134134
Data: encodeProposalEnvelope(99, []byte("would-be wrapped payload")),
135135
}
136-
_, err := e.applyNormalEntry(entry)
136+
_, err := e.applyNormalEntry(entry, false)
137137
if !errors.Is(err, ErrRaftUnwrapFailed) {
138138
t.Fatalf("expected ErrRaftUnwrapFailed for cutover-active+no-cipher misconfig, got %v", err)
139139
}
@@ -151,7 +151,7 @@ func TestApplyNormalEntry_CutoverActive_NoCipher_FailsClosed(t *testing.T) {
151151
Index: cutover,
152152
Data: encodeProposalEnvelope(11, []byte("legacy cleartext")),
153153
}
154-
if _, err := e.applyNormalEntry(belowCutoverEntry); err != nil {
154+
if _, err := e.applyNormalEntry(belowCutoverEntry, false); err != nil {
155155
t.Fatalf("below-cutover should pass through, got %v", err)
156156
}
157157
if got := fsm.calls.Load(); got != 1 {
@@ -169,7 +169,7 @@ func TestApplyNormalEntry_NoCipher_PassThrough(t *testing.T) {
169169
e := newTestEngine(fsm, nil, nil)
170170
plain := []byte("op=put key=k1 v=hello")
171171
entry := raftpb.Entry{Type: raftpb.EntryNormal, Data: encodeProposalEnvelope(42, plain)}
172-
if _, err := e.applyNormalEntry(entry); err != nil {
172+
if _, err := e.applyNormalEntry(entry, false); err != nil {
173173
t.Fatalf("applyNormalEntry: %v", err)
174174
}
175175
if got := fsm.calls.Load(); got != 1 {
@@ -200,7 +200,7 @@ func TestApplyNormalEntry_BelowCutover_PassThrough(t *testing.T) {
200200
Index: idx,
201201
Data: encodeProposalEnvelope(11, cleartextPayload),
202202
}
203-
if _, err := e.applyNormalEntry(entry); err != nil {
203+
if _, err := e.applyNormalEntry(entry, false); err != nil {
204204
t.Fatalf("idx=%d: applyNormalEntry: %v", idx, err)
205205
}
206206
if got := fsm.calls.Load(); got != 1 {
@@ -226,7 +226,7 @@ func TestApplyNormalEntry_AboveCutover_Unwraps(t *testing.T) {
226226
fsm.calls.Store(0)
227227
plaintext := []byte("op=put key=k1 v=secret")
228228
entry := envelopeEntry(t, c, kid, idx, plaintext)
229-
if _, err := e.applyNormalEntry(entry); err != nil {
229+
if _, err := e.applyNormalEntry(entry, false); err != nil {
230230
t.Fatalf("idx=%d: applyNormalEntry: %v", idx, err)
231231
}
232232
if got := fsm.calls.Load(); got != 1 {
@@ -258,7 +258,7 @@ func TestApplyNormalEntry_UnwrapFailure_Halts(t *testing.T) {
258258
// last byte.
259259
entry.Data[len(entry.Data)-1] ^= 0xff
260260

261-
_, err := e.applyNormalEntry(entry)
261+
_, err := e.applyNormalEntry(entry, false)
262262
if !errors.Is(err, ErrRaftUnwrapFailed) {
263263
t.Fatalf("expected ErrRaftUnwrapFailed, got %v", err)
264264
}
@@ -326,7 +326,7 @@ func TestApplyNormalEntry_BoundaryCutover(t *testing.T) {
326326
Index: cutover,
327327
Data: encodeProposalEnvelope(13, cleartext),
328328
}
329-
if _, err := e.applyNormalEntry(atCutover); err != nil {
329+
if _, err := e.applyNormalEntry(atCutover, false); err != nil {
330330
t.Fatalf("at-cutover: %v", err)
331331
}
332332
if string(fsm.last) != string(cleartext) {
@@ -335,7 +335,7 @@ func TestApplyNormalEntry_BoundaryCutover(t *testing.T) {
335335

336336
// cutover+1: wrapped payload — MUST be unwrapped.
337337
above := envelopeEntry(t, c, kid, cutover+1, []byte("first encrypted"))
338-
if _, err := e.applyNormalEntry(above); err != nil {
338+
if _, err := e.applyNormalEntry(above, false); err != nil {
339339
t.Fatalf("above-cutover: %v", err)
340340
}
341341
if string(fsm.last) != "first encrypted" {
@@ -365,7 +365,7 @@ func TestApplyNormalEntry_ProposalIDStillResolvable(t *testing.T) {
365365
}
366366
data := encodeProposalEnvelope(wantID, wrapped)
367367
entry := raftpb.Entry{Type: raftpb.EntryNormal, Index: cutover + 1, Data: data}
368-
if _, err := e.applyNormalEntry(entry); err != nil {
368+
if _, err := e.applyNormalEntry(entry, false); err != nil {
369369
t.Fatalf("applyNormalEntry: %v", err)
370370
}
371371
gotID, _, ok := decodeProposalEnvelope(entry.Data)
@@ -396,7 +396,7 @@ func TestApplyNormalEntry_NoCutoverDefault(t *testing.T) {
396396
Index: idx,
397397
Data: encodeProposalEnvelope(7, cleartext),
398398
}
399-
if _, err := e.applyNormalEntry(entry); err != nil {
399+
if _, err := e.applyNormalEntry(entry, false); err != nil {
400400
t.Fatalf("idx=%d: %v", idx, err)
401401
}
402402
if string(fsm.last) != string(cleartext) {

0 commit comments

Comments
 (0)