Commit 688ba8a
authored
adapter: wrap DynamoDB query/scan/transactGetItems in LeaseRead (#952)
## Summary
Part of #557. Wraps the remaining DynamoDB read handlers in a
quorum-freshness `LeaseRead` check, following the pattern PR #549
established for `getItem`. Previously these handlers read local
`LastCommitTS`/snapshot state with **no quorum check at all**, so a
partitioned-but-not-yet-stepped-down stale leader could serve stale
reads. Each handler now performs one lease check (cheap atomic-load fast
path within the lease window; one `LinearizableRead` on a lease miss,
after which the lease is refreshed for the rest of the window) before
resolving its read timestamp.
The Redis slice of #557 is deliberately left for a follow-up — this PR
touches only `adapter/dynamodb.go`.
### Per-handler placement
- **`query`** — keyless `LeaseRead(ctx)` (`leaseReadKeyless` helper)
before `queryItems` samples `readTS`. A Query scans a key range that can
span shards in a multi-group deployment, and the owning shard can't be
resolved in the handler without duplicating
`prepareReadSchema`/`resolveQueryCondition`, so the keyless check (which
anchors the cluster freshness bound) is the correct, cheaper call.
- **`scan`** — keyless `LeaseRead(ctx)` (same helper) before `scanItems`
samples `readTS`. A Scan reads the whole table and spans every shard
holding its items, so keyless is the right choice (matches the issue's
guidance for keyless/multi-shard ops).
- **`transactGetItems`** — per-key `LeaseReadForKey`, deduplicated by
item key, run in `leaseCheckTransactGetItems` **before**
`nextTxnReadTS()` resolves the single snapshot timestamp. Item keys are
resolved at a tentative schema timestamp purely to route the lease
check; the single-snapshot-ts semantics (one `readTS` shared by all
items) are unchanged. Malformed items are skipped in the pre-pass so the
existing validation/error mapping still surfaces downstream identically.
Within the lease window, same-shard keys hit the cheap fast path, so
per-key checks are effectively free.
### Timeout decision
Every wrapped handler runs the lease check under a bounded
`context.WithTimeout(r.Context(), dynamoLeaseReadTimeout)` with `defer
cancel()` — the exact constant (`5s`) and structure `getItem` uses after
PR #549 — so a stalled Raft cannot hang a handler indefinitely when the
client never cancels.
### Note on `batchGetItem`
The issue lists `batchGetItem` in scope, but **there is no
`BatchGetItem` handler in this codebase** (no `batchGetItemTarget`, no
registration, no implementation). There was nothing to wrap; the
multi-key precedent it asked about is implemented in `transactGetItems`
(per-key `LeaseReadForKey`, deduped). If `BatchGetItem` is added later,
it should reuse the same `leaseCheckTransactGetItems`-style per-key
dedup pattern.
## Behavior change
Adds a quorum-freshness check to `query`/`scan`/`transactGetItems`. Read
results and error mapping are otherwise identical; lease-read failure
surfaces as the same `500 InternalServerError` (`dynamoErrInternal`)
that `getItem` produces.
## Risk
Low. The only new failure mode is a `500` when the lease check fails
(quorum loss), which is strictly more correct than serving a stale read.
No write paths, FSM, or snapshot semantics touched. The added
per-request `LeaseRead` is amortized by the lease window.
## Test evidence
- `go test -race -run TestDynamoDB ./adapter/` → ok (15s)
- Targeted: `TestDynamoDB_Query_LeaseRead`,
`TestDynamoDB_Scan_LeaseRead`, `TestDynamoDB_TransactGetItems_LeaseRead`
(each asserts (a) success with a healthy lease and (b) lease failure →
500 `InternalServerError`, mirroring `getItem`'s error class) plus
existing `TestDynamoDB_TransactGetItems*` → ok with `-race`.
- `golangci-lint --config=.golangci.yaml run ./adapter/...` → 0 issues.
- `go vet ./adapter/`, `gofmt -l` → clean.
- Full `go test -race ./adapter/` exceeds the default 10-minute package
timeout (the suite spans Redis/S3/SQS/gRPC/etc., unrelated to this
change); it passes under an extended `-timeout`.
New tests inject lease-read failures via a `failLeaseReads` toggle added
to the existing `testCoordinatorWrapper`, and drive the failure path
over raw HTTP so the AWS SDK does not retry the `500`.
## Self-review
1. **Data loss** — None. Read-only handlers; no propose/apply, FSM,
snapshot, TTL, or compaction paths touched. No `return nil`-after-error
introduced; lease failure returns an explicit `500`.
2. **Concurrency / distributed failures** — Lease check is bounded by
`dynamoLeaseReadTimeout` with `defer cancel()`, so a leader change /
partition can't hang a handler. `transactGetItems` keeps its single
shared `readTS`; the lease pre-pass runs before the timestamp is
sampled, so a leadership flip mid-pass only causes a lease miss (→
`LinearizableRead`) or a `500`, never a torn snapshot. Ran `go test
-race` on the dynamo suite.
3. **Performance** — One `LeaseRead` per request, amortized by the lease
window (atomic-load fast path; one `LinearizableRead` per lease miss,
then refreshed). `transactGetItems` dedups by item key and same-shard
keys hit the fast path, so per-key checks add negligible cost; no extra
Raft round-trips per HLC tick. The tentative schema load in the transact
pre-pass is a cheap, cached Pebble read.
4. **Data consistency** — `readTS` is sampled **after** the lease
confirmation in every handler, so any commit that landed before
confirmation is visible; this tightens the freshness bound without
bypassing the leader-issued read pipeline. `transactGetItems`
single-snapshot-ts semantics are preserved.
5. **Test coverage** — Added a success + lease-failure test for each
wrapped handler, reusing the existing
`testCoordinatorWrapper`/`createNode` harness and mirroring `getItem`'s
error class on failure.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Read operations perform pre-read freshness checks to ensure
consistent, up-to-date responses; queries and scans fallback to broader
fencing when needed.
* Transactional reads now confirm freshness across all affected groups
before resolving a transaction snapshot.
* Coordinators support multi-group lease fencing and per-group read
collapsing for more robust multi-shard reads.
* **Tests**
* Extensive tests added covering lease/fencing, validation, skip-paths,
and multi-group behaviors for query, scan, and transactional reads.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->10 files changed
Lines changed: 2522 additions & 0 deletions
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
0 commit comments