Skip to content

Commit bd597b5

Browse files
authored
perf(lease-read): cache LeaseProvider type assertion on Coordinate and ShardGroup (#947)
Closes #555. ## What `Coordinate.LeaseRead` and `groupLeaseRead` performed the interface type assertion `engine.(raftengine.LeaseProvider)` on every call (a hot path: Redis GET and all lease reads). This caches the assertion once at construction: - Added `lp raftengine.LeaseProvider` field to `Coordinate` (`kv/coordinator.go`), set once in `NewCoordinatorWithEngine` in the same block that already asserts for the leader-loss callback. - Added `lp raftengine.LeaseProvider` field to `ShardGroup` (`kv/sharded_coordinator.go`), set once per group in the `NewShardedCoordinator` loop that already asserts for the per-shard leader-loss callback. - `LeaseRead`, `refreshLeaseAfterDispatch`, `groupLeaseRead`, and `leaseRefreshingTxn.maybeRefresh` now test `lp == nil` instead of re-asserting. `groupLeaseRead` keeps a nil-group guard so `engineForGroup`'s nil-safety is preserved. - Added `BenchmarkLeaseRead` / `BenchmarkGroupLeaseRead` (`kv/lease_read_benchmark_test.go`) exercising the fast path; both assert the slow path (`LinearizableRead`) never runs so the benchmark can't silently measure the wrong path. The `engine` field is never reassigned after construction, so the cached value stays valid for the object's lifetime without a lock. ## Behavior change None expected. The assertion result is identical; it is now computed once instead of per call. All existing lease tests pass unchanged. ## Risk Low. The only subtlety is construction ordering in `NewShardedCoordinator`: the `leaseRefreshingTxn` wrapper is created earlier in the same loop iteration than `g.lp` is assigned, but it holds a `*ShardGroup` pointer (not a copy), and `maybeRefresh` only runs at Commit time — long after the loop finishes — so `g.lp` is always populated by then. ## Scope note / deviation `leaseReadEngineCtx` (`kv/raft_engine.go`) also does a per-call `LeaseProvider` assertion, but it takes a bare `raftengine.LeaderView` and is shared across the internal storage read path (`shard_store.go`); it has no `Coordinate`/`ShardGroup` to cache on. The issue's accepted direction scopes the field to `Coordinate` and `ShardGroup`, so `leaseReadEngineCtx` is intentionally left unchanged. ## Test evidence `go test -race ./kv/...`: ``` ok github.com/bootjp/elastickv/kv 10.499s ``` `golangci-lint --config=.golangci.yaml run ./kv/...`: ``` 0 issues. ``` Benchmarks (Apple M1 Max, fast path, 0 allocs, LinearizableRead never invoked): ``` BenchmarkLeaseRead-10 9749521 209.1 ns/op 0 B/op 0 allocs/op BenchmarkGroupLeaseRead-10 11888186 89.07 ns/op 0 B/op 0 allocs/op ``` ## Self-review 1. **Data loss** — No persistence/Raft propose/apply/snapshot path touched. Lease-read semantics (fast vs. slow path selection) are byte-for-byte identical; only the cached assertion replaces the per-call one. No new error-swallowing. No risk. 2. **Concurrency / distributed failures** — `lp` is written once during construction and only read afterward; `engine` is never reassigned, so no lock is needed and there is no data race (confirmed by `go test -race`). Construction ordering of `leaseRefreshingTxn` vs. `g.lp` assignment is safe (pointer aliasing; `maybeRefresh` runs only at Commit time). Leader-loss callback registration is unchanged. 3. **Performance** — Removes an interface type assertion from the lease-read hot path, replacing it with a single nil check on a struct field. Benchmarks show 0 allocs and the fast path is preserved. No extra Raft round-trips, no new lock contention. 4. **Data consistency** — No change to MVCC visibility, OCC commit-ts ordering, HLC ceiling, or the lease freshness bound (`engineLeaseAckValid` and the secondary caller-side lease check are untouched). The fallback to `LinearizableRead` when the engine lacks `LeaseProvider` or the lease is disabled is preserved exactly. 5. **Test coverage** — Existing lease tests (`TestCoordinate_LeaseRead_*`, `TestShardedCoordinator_LeaseRead*`, fallback-when-engine-lacks-LeaseProvider, leader-loss registration) cover both the cached-present and nil paths and pass unchanged. Added two benchmarks that fail if the slow path is taken, locking in the "assertion off the hot path" claim.
2 parents e465b61 + a8efccc commit bd597b5

3 files changed

Lines changed: 113 additions & 14 deletions

File tree

kv/coordinator.go

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,11 @@ func NewCoordinatorWithEngine(txm Transactional, engine raftengine.Engine, opts
120120
for _, opt := range opts {
121121
opt(c)
122122
}
123+
// Resolve the optional LeaseProvider capability once here so the
124+
// LeaseRead / refreshLeaseAfterDispatch hot paths test a cached
125+
// field instead of repeating the interface type assertion per call.
126+
// engine is never reassigned after construction, so the cached value
127+
// stays valid for the Coordinate's lifetime.
123128
// Register a leader-loss hook so the lease is invalidated the instant
124129
// the engine notices a state transition out of the leader role,
125130
// rather than waiting for wall-clock expiry of the current lease.
@@ -128,6 +133,7 @@ func NewCoordinatorWithEngine(txm Transactional, engine raftengine.Engine, opts
128133
// one-shot tools) MUST call Close() to avoid leaking a closure
129134
// pointing into this Coordinate.
130135
if lp, ok := engine.(raftengine.LeaseProvider); ok {
136+
c.lp = lp
131137
c.deregisterLeaseCb = lp.RegisterLeaderLossCallback(c.lease.invalidate)
132138
}
133139
return c
@@ -169,10 +175,18 @@ type CoordinateResponse struct {
169175
type Coordinate struct {
170176
transactionManager Transactional
171177
engine raftengine.Engine
172-
clock *HLC
173-
connCache GRPCConnCache
174-
log *slog.Logger
175-
lease leaseState
178+
// lp caches the engine's optional LeaseProvider capability so the
179+
// LeaseRead hot path (and refreshLeaseAfterDispatch) test a single
180+
// field for nil instead of performing an interface type assertion on
181+
// every call. It is set once in NewCoordinatorWithEngine and is nil
182+
// when the engine does not implement raftengine.LeaseProvider. The
183+
// engine field is never reassigned after construction, so this stays
184+
// in sync without a lock.
185+
lp raftengine.LeaseProvider
186+
clock *HLC
187+
connCache GRPCConnCache
188+
log *slog.Logger
189+
lease leaseState
176190
// deregisterLeaseCb removes the leader-loss callback registered
177191
// against engine at construction. Long-lived Coordinates don't
178192
// need to call it (the engine will be closed after them), but
@@ -599,11 +613,10 @@ func (c *Coordinate) refreshLeaseAfterDispatch(resp *CoordinateResponse, err err
599613
if resp == nil || resp.CommitIndex == 0 {
600614
return
601615
}
602-
lp, ok := c.engine.(raftengine.LeaseProvider)
603-
if !ok {
616+
if c.lp == nil {
604617
return
605618
}
606-
c.lease.extend(dispatchStart.Add(lp.LeaseDuration()), expectedGen)
619+
c.lease.extend(dispatchStart.Add(c.lp.LeaseDuration()), expectedGen)
607620
}
608621

609622
func (c *Coordinate) IsLeader() bool {
@@ -760,8 +773,8 @@ func (c *Coordinate) LinearizableReadForKey(ctx context.Context, _ []byte) (uint
760773
// Callers that resolve timestamps via store.LastCommitTS may discard
761774
// the value.
762775
func (c *Coordinate) LeaseRead(ctx context.Context) (uint64, error) {
763-
lp, ok := c.engine.(raftengine.LeaseProvider)
764-
if !ok {
776+
lp := c.lp
777+
if lp == nil {
765778
return c.LinearizableRead(ctx)
766779
}
767780
leaseDur := lp.LeaseDuration()

kv/lease_read_benchmark_test.go

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
package kv
2+
3+
import (
4+
"context"
5+
"testing"
6+
"time"
7+
8+
"github.com/bootjp/elastickv/distribution"
9+
"github.com/bootjp/elastickv/internal/monoclock"
10+
)
11+
12+
// BenchmarkLeaseRead measures the Coordinate.LeaseRead fast path, where
13+
// the engine-driven lease anchor (LastQuorumAck + State==Leader) is
14+
// fresh so the read is served from the cached AppliedIndex without a
15+
// LinearizableRead round-trip. The LeaseProvider capability is resolved
16+
// once at construction (NewCoordinatorWithEngine) and cached on
17+
// Coordinate.lp, so this hot loop performs a single nil check rather
18+
// than an interface type assertion per call.
19+
func BenchmarkLeaseRead(b *testing.B) {
20+
eng := &fakeLeaseEngine{applied: 4242, leaseDur: time.Hour}
21+
eng.setQuorumAck(monoclock.Now())
22+
c := NewCoordinatorWithEngine(nil, eng)
23+
ctx := context.Background()
24+
25+
b.ReportAllocs()
26+
b.ResetTimer()
27+
for i := 0; i < b.N; i++ {
28+
if _, err := c.LeaseRead(ctx); err != nil {
29+
b.Fatal(err)
30+
}
31+
}
32+
b.StopTimer()
33+
34+
// Guard the benchmark against silently exercising the slow path: a
35+
// single LinearizableRead would invalidate the "assertion is off the
36+
// hot path" claim because the slow path dominates the measurement.
37+
if got := eng.linearizableCalls.Load(); got != 0 {
38+
b.Fatalf("expected the lease fast path on every iteration, but LinearizableRead ran %d times", got)
39+
}
40+
}
41+
42+
// BenchmarkGroupLeaseRead measures the sharded groupLeaseRead fast path
43+
// (via ShardedCoordinator.LeaseRead on the default group). The
44+
// LeaseProvider capability is resolved once per group in
45+
// NewShardedCoordinator and cached on ShardGroup.lp, so the hot loop is
46+
// a single nil check rather than a per-call interface type assertion.
47+
func BenchmarkGroupLeaseRead(b *testing.B) {
48+
eng := newShardedLeaseEngine(7777)
49+
eng.setQuorumAck(monoclock.Now())
50+
51+
distEngine := distribution.NewEngine()
52+
distEngine.UpdateRoute([]byte("a"), nil, 1)
53+
coord := NewShardedCoordinator(distEngine, map[uint64]*ShardGroup{
54+
1: {Engine: eng, Txn: &recordingTransactional{}},
55+
}, 1, NewHLC(), nil)
56+
ctx := context.Background()
57+
58+
b.ReportAllocs()
59+
b.ResetTimer()
60+
for i := 0; i < b.N; i++ {
61+
if _, err := coord.LeaseRead(ctx); err != nil {
62+
b.Fatal(err)
63+
}
64+
}
65+
b.StopTimer()
66+
67+
if got := eng.linearizableCalls.Load(); got != 0 {
68+
b.Fatalf("expected the lease fast path on every iteration, but LinearizableRead ran %d times", got)
69+
}
70+
}

kv/sharded_coordinator.go

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,14 @@ type ShardGroup struct {
2424
Store store.MVCCStore
2525
Txn Transactional
2626
lease leaseState
27+
// lp caches the Engine's optional LeaseProvider capability so the
28+
// groupLeaseRead / maybeRefresh hot paths test a single field for
29+
// nil instead of performing an interface type assertion per call.
30+
// NewShardedCoordinator resolves it once for every group it owns; it
31+
// is nil when the Engine does not implement raftengine.LeaseProvider.
32+
// Engine is not reassigned after the coordinator is constructed, so
33+
// the cached value stays valid.
34+
lp raftengine.LeaseProvider
2735
// raftPayloadWrap is the Stage 6E-2c hot-swap point for the raft
2836
// envelope wrap closure. A nil load means the wrap is inactive
2937
// and proposals pass through cleartext (the Stage 3 default).
@@ -283,11 +291,10 @@ func (t *leaseRefreshingTxn) maybeRefresh(resp *TransactionResponse, start monoc
283291
if resp == nil || resp.CommitIndex == 0 {
284292
return
285293
}
286-
lp, ok := t.g.Engine.(raftengine.LeaseProvider)
287-
if !ok {
294+
if t.g.lp == nil {
288295
return
289296
}
290-
t.g.lease.extend(start.Add(lp.LeaseDuration()), expectedGen)
297+
t.g.lease.extend(start.Add(t.g.lp.LeaseDuration()), expectedGen)
291298
}
292299

293300
// Close forwards to the wrapped Transactional if it implements
@@ -577,10 +584,14 @@ func NewShardedCoordinator(engine *distribution.Engine, groups map[uint64]*Shard
577584
}
578585
}
579586
router.Register(gid, g.Txn, g.Store)
587+
// Resolve the optional LeaseProvider capability once so
588+
// groupLeaseRead / maybeRefresh test g.lp for nil instead of
589+
// re-asserting the interface per call.
580590
// Per-shard leader-loss hook: when this group's engine notices
581591
// a state transition out of leader, drop the lease so the next
582592
// LeaseReadForKey on that shard takes the slow path.
583593
if lp, ok := g.Engine.(raftengine.LeaseProvider); ok {
594+
g.lp = lp
584595
deregisters = append(deregisters, lp.RegisterLeaderLossCallback(g.lease.invalidate))
585596
}
586597
}
@@ -1547,10 +1558,15 @@ func observeLeaseRead(observer LeaseReadObserver, hit bool) {
15471558

15481559
func groupLeaseRead(ctx context.Context, g *ShardGroup, observer LeaseReadObserver) (uint64, error) {
15491560
engine := engineForGroup(g)
1550-
lp, ok := engine.(raftengine.LeaseProvider)
1551-
if !ok {
1561+
// g.lp caches the LeaseProvider assertion done once at construction
1562+
// (NewShardedCoordinator); a nil group or an engine without the
1563+
// capability falls through to the linearizable slow path. The nil-g
1564+
// guard preserves engineForGroup's nil-safety since g.lp would panic
1565+
// on a nil receiver.
1566+
if g == nil || g.lp == nil {
15521567
return linearizableReadEngineCtx(ctx, engine)
15531568
}
1569+
lp := g.lp
15541570
leaseDur := lp.LeaseDuration()
15551571
if leaseDur <= 0 {
15561572
return linearizableReadEngineCtx(ctx, engine)

0 commit comments

Comments
 (0)