@@ -8,6 +8,34 @@ All notable changes to HyperCache are recorded here. The format follows
88
99### Added
1010
11+ - ** Async read-repair batching (Phase 4) + unconditional ` ForwardSet ` -only repair.** Two composing changes
12+ in the same PR that together cut the wire-call cost of read-repair under quorum reads. (1) The defensive
13+ ` ForwardGet ` probe in ` repairRemoteReplica ` is gone — every repair is now exactly one ` ForwardSet ` ,
14+ because the receiver's ` applySet ` already version-compares and noops downgrades, so the probe was pure
15+ duplication. ~ 50% wire-call reduction per repair regardless of batching. (2) New opt-in
16+ [ ` backend.WithDistReadRepairBatch(interval, maxBatchSize) ` ] ( pkg/backend/dist_memory.go ) option queues
17+ repairs by destination peer + key (last-write-wins by ` (version, origin) ` ) and dispatches per-peer batches
18+ on the interval or when a peer's pending count hits ` maxBatchSize ` . Concurrent reads of the same hot key
19+ produce ONE repair through the queue, not N — the coalescer collapses duplicate ` (peer, key) ` entries
20+ and bumps the new ` dist.read_repair.coalesced ` counter per collapsed enqueue. Disabled by default
21+ (` interval == 0 ` = current synchronous behavior preserved, so ` TestDistMemoryReadRepair ` and
22+ ` TestDistMemoryRemoveReplication ` pass byte-identical). Clean shutdown drains the queue inside ` Stop() ` ;
23+ crash exit loses queued repairs by design, with merkle anti-entropy as the convergence safety net.
24+ New [ ` pkg/backend/dist_read_repair.go ` ] ( pkg/backend/dist_read_repair.go ) hosts the ` repairQueue ` type
25+ with errgroup-driven per-peer parallel ` ForwardSet ` dispatch. Eight unit tests in
26+ [ ` pkg/backend/dist_read_repair_test.go ` ] ( pkg/backend/dist_read_repair_test.go ) cover the coalesce rule
27+ (same ` (peer, key) ` keeps the higher version, distinct peers stay independent), the size-threshold
28+ inline flush, the nil-transport noop path, the ` Stop() ` drain semantics, the ` (version, origin) `
29+ tie-break rule, and concurrent-enqueue race-safety. Three integration tests in
30+ [ ` tests/hypercache_distmemory_readrepair_batch_test.go ` ] ( tests/hypercache_distmemory_readrepair_batch_test.go )
31+ drive the end-to-end shape — a 3-node RF=3 ConsistencyQuorum cluster, one node's local copy dropped,
32+ N concurrent Gets from a third node — and assert the batched flush heals the dropped node, parallel
33+ reads coalesce to ≤2 dispatches (one per remote owner) regardless of N, and ` Stop() ` drains queued
34+ repairs before returning. Two new OTel metrics:
35+ ` dist.read_repair.batched ` (per actual ` ForwardSet ` dispatched by the queue's flusher) and
36+ ` dist.read_repair.coalesced ` (per duplicate-enqueue collapsed). New "Tuning — read-repair batching"
37+ section in [ ` docs/operations.md ` ] ( docs/operations.md ) covers the option shape, the divergence-window
38+ trade-off, the two metrics, and when to enable it (high read-amplification with stable hot keys).
1139- ** Token-refresh visibility for the OIDC source.** Closes RFC 0003 open question 6: the
1240 ` WithOIDCClientCredentials ` source now wraps its ` oauth2.TokenSource ` with a logger that emits one
1341 ` "oidc token rotated" ` Info line per real rotation (expiry change), staying silent on cached returns.
@@ -258,6 +286,23 @@ All notable changes to HyperCache are recorded here. The format follows
258286
259287# ## Fixed
260288
289+ - **Set-forward promotion no longer requires the in-process `ErrBackendNotFound` sentinel.**
290+ [`handleForwardPrimary`](pkg/backend/dist_memory.go) used to gate "primary unreachable → promote to
291+ replica" on `errors.Is(errFwd, sentinel.ErrBackendNotFound)`, the error the in-process transport returns
292+ for an unregistered peer. HTTP/gRPC transports against a stopped container surface
293+ ` net.OpError` / `io.EOF` / `context.DeadlineExceeded` instead — none of which matched the condition.
294+ Result : when a cluster node was killed (e.g. `docker stop` in
295+ [`scripts/tests/20-test-cluster-resilience.sh`](scripts/tests/20-test-cluster-resilience.sh)), writes for
296+ keys whose primary was the dead node failed immediately at the forwarding hop, no hint was queued, and
297+ the data never landed anywhere — the same 7 of 50 "during-*" writes failed reproducibly in CI's cluster
298+ workflow. Promotion now triggers on **any** non-nil forward error when the local node is in `owners[1:]`,
299+ matching the in-process and production transport behavior under the same resilience contract. Spurious
300+ promotion on a transient blip is benign — `applySet` version-compares on the receiver, and merkle
301+ anti-entropy / `chooseNewer` reconcile any divergent `(version, origin)` pair via the existing
302+ last-write-wins rule. New test [`TestDistSet_PromotesOnGenericForwardError`](tests/hypercache_distmemory_forward_primary_promotion_test.go)
303+ uses the chaos hooks at `DropRate=1.0` to deterministically force a generic forward error and asserts the
304+ Set succeeds via promotion; the existing `TestDistFailureRecovery` continues to pass byte-identical (the
305+ change widens the promotion gate, doesn't narrow it).
261306- **`TestDistRebalanceReplicaDiffThrottle` no longer flakes under `make test-race`.** The test's 900ms hard
262307 sleep wasn't enough wall-clock budget for the rebalancer's 80ms-tick loop to actually fire 11 ticks under
263308 ` -race` + `-shuffle=on`'s scheduler pressure. Replaced the sleep with a 5-second polling loop that exits as
0 commit comments