fix(dist-memory): broaden set-forward promotion to any transport error

Previously, `handleForwardPrimary` only promoted to a replica owner
when the forward error matched the in-process transport's
`ErrBackendNotFound` sentinel. HTTP/gRPC transports against a stopped
node surface `net.OpError`, `io.EOF`, or `context.DeadlineExceeded`
instead — causing writes to fail silently for keys whose primary had
just been killed, rather than falling through to a replica.

Promotion now triggers on any non-nil forward error when the local
node is listed in `owners[1:]` and still owns the key locally
(defensive against a stale ring snapshot). Spurious promotion on a
transient blip is benign: `applySet` version-compares on the receiver
and `chooseNewer` / merkle anti-entropy reconcile divergent
`(version, origin)` pairs via the existing LWW rule.

Add `TestDistSet_PromotesOnGenericForwardError` which uses chaos hooks
at `DropRate=1.0` to deterministically force a generic forward error
and asserts the Set succeeds via promotion; `TestDistFailureRecovery`
continues to pass unchanged (the change widens the gate, doesn't
narrow it).

Also add a `check_command_exists` Makefile macro and CHANGELOG entry.

Author: hyp3rd

CHANGELOG.md

@@ -286,6 +286,23 @@ All notable changes to HyperCache are recorded here. The format follows
 
 ### Fixed
 
+- **Set-forward promotion no longer requires the in-process `ErrBackendNotFound` sentinel.**
+  [`handleForwardPrimary`](pkg/backend/dist_memory.go) used to gate "primary unreachable → promote to
+  replica" on `errors.Is(errFwd, sentinel.ErrBackendNotFound)`, the error the in-process transport returns
+  for an unregistered peer. HTTP/gRPC transports against a stopped container surface
+  `net.OpError` / `io.EOF` / `context.DeadlineExceeded` instead — none of which matched the condition.
+  Result: when a cluster node was killed (e.g. `docker stop` in
+  [`scripts/tests/20-test-cluster-resilience.sh`](scripts/tests/20-test-cluster-resilience.sh)), writes for
+  keys whose primary was the dead node failed immediately at the forwarding hop, no hint was queued, and
+  the data never landed anywhere — the same 7 of 50 "during-*" writes failed reproducibly in CI's cluster
+  workflow. Promotion now triggers on **any** non-nil forward error when the local node is in `owners[1:]`,
+  matching the in-process and production transport behavior under the same resilience contract. Spurious
+  promotion on a transient blip is benign — `applySet` version-compares on the receiver, and merkle
+  anti-entropy / `chooseNewer` reconcile any divergent `(version, origin)` pair via the existing
+  last-write-wins rule. New test [`TestDistSet_PromotesOnGenericForwardError`](tests/hypercache_distmemory_forward_primary_promotion_test.go)
+  uses the chaos hooks at `DropRate=1.0` to deterministically force a generic forward error and asserts the
+  Set succeeds via promotion; the existing `TestDistFailureRecovery` continues to pass byte-identical (the
+  change widens the promotion gate, doesn't narrow it).
 - **`TestDistRebalanceReplicaDiffThrottle` no longer flakes under `make test-race`.** The test's 900ms hard
   sleep wasn't enough wall-clock budget for the rebalancer's 80ms-tick loop to actually fire 11 ticks under
   `-race` + `-shuffle=on`'s scheduler pressure. Replaced the sleep with a 5-second polling loop that exits as

Makefile

@@ -99,6 +99,16 @@ run-example:
 update-deps:
 	go get -u -t ./... && go mod tidy -v && go mod verify
 
+# check_command_exists expands to a recipe line that succeeds if the
+# given command resolves on PATH, otherwise prints "<cmd> command not
+# found" and exits non-zero. Used by prepare-base-tools' chain of
+# `$(call check_command_exists,X) || install-fallback` lines: the
+# first failure message above is the developer-facing hint; the
+# install fallback fires when the tool itself is genuinely missing.
+define check_command_exists
+@which $(1) > /dev/null 2>&1 || (echo "$(1) command not found" && exit 1)
+endef
+
 prepare-toolchain: prepare-base-tools
 
 prepare-base-tools:

pkg/backend/dist_memory.go

@@ -3622,7 +3622,21 @@ func (dm *DistMemory) repairRemoteReplica(
 	dm.metrics.readRepair.Add(1)
 }
 
-// handleForwardPrimary tries to forward a Set to the primary; returns (proceedAsPrimary,false) if promotion required.
+// handleForwardPrimary tries to forward a Set to the primary; returns
+// (proceedAsPrimary, err). On any non-nil forward error — not just
+// ErrBackendNotFound — the function attempts to promote to a replica
+// owner if the local node is in the replica list. This is the correct
+// resilience contract: the in-process transport surfaces unreachable
+// peers as ErrBackendNotFound, but HTTP/gRPC transports against a
+// stopped container surface net.OpError / io.EOF /
+// context.DeadlineExceeded. Gating promotion on the in-process-only
+// sentinel meant production forwarding failures dropped writes to keys
+// whose primary had just been killed, until the next merkle tick.
+//
+// A spurious promotion (primary was healthy but the call hit a
+// transient blip) is benign: applySet on the receiver version-compares,
+// and `chooseNewer` / merkle anti-entropy reconcile any divergent
+// `(version, origin)` pair through the existing last-write-wins rule.
 func (dm *DistMemory) handleForwardPrimary(ctx context.Context, owners []cluster.NodeID, item *cache.Item) (bool, error) {
 	transport := dm.loadTransport()
 	if transport == nil {
@@ -3632,26 +3646,23 @@ func (dm *DistMemory) handleForwardPrimary(ctx context.Context, owners []cluster
 	dm.metrics.forwardSet.Add(1)
 
 	errFwd := transport.ForwardSet(ctx, string(owners[0]), item, true)
-	switch {
-	case errFwd == nil:
-		return false, nil // forwarded successfully
-	case errors.Is(errFwd, sentinel.ErrBackendNotFound) && len(owners) > 1:
-		// primary missing: promote if this node is a listed replica
-		for _, oid := range owners[1:] {
-			if oid == dm.localNode.ID { // we can promote
-				if !dm.ownsKeyInternal(item.Key) { // still not recognized locally (ring maybe outdated)
-					return false, errFwd
-				}
+	if errFwd == nil {
+		return false, nil
+	}
 
-				return true, nil // proceed as primary path
+	// Primary unreachable for any reason — try to promote to a replica
+	// owner. The local node must be in owners[1:] AND still recognize
+	// itself as an owner (defensive against a stale ring snapshot
+	// mid-membership-change).
+	if len(owners) > 1 {
+		for _, oid := range owners[1:] {
+			if oid == dm.localNode.ID && dm.ownsKeyInternal(item.Key) {
+				return true, nil
 			}
 		}
-
-		return false, errFwd // not promotable
-
-	default:
-		return false, errFwd
 	}
+
+	return false, errFwd
 }
 
 // initStandaloneMembership initializes membership & ring for standalone mode with optional seeds.

tests/hypercache_distmemory_forward_primary_promotion_test.go

@@ -0,0 +1,74 @@
+package tests
+
+import (
+	"context"
+	"testing"
+
+	"github.com/hyp3rd/hypercache/internal/cluster"
+	"github.com/hyp3rd/hypercache/pkg/backend"
+	cache "github.com/hyp3rd/hypercache/pkg/cache/v2"
+)
+
+// TestDistSet_PromotesOnGenericForwardError pins the resilience
+// contract that `handleForwardPrimary` must promote to a replica
+// owner on ANY non-nil forward error, not just the in-process
+// transport's `ErrBackendNotFound` sentinel. Production HTTP/gRPC
+// transports against a stopped container surface `net.OpError`,
+// `io.EOF`, or `context.DeadlineExceeded` — none of which match
+// the pre-fix promotion condition.
+//
+// Pre-fix behavior on this test: Set returns ErrChaosDrop (the
+// forward fails, the default switch arm returns the error
+// verbatim). Post-fix: Set returns nil and the key lands on the
+// promoting node's local shard.
+//
+// Uses chaos hooks at DropRate=1.0 to deterministically force every
+// outbound forward call to return ErrChaosDrop. Combined with
+// ConsistencyOne writes so the local primary path satisfies quorum
+// without needing any successful replica fan-out (chaos breaks the
+// replica calls too — that's covered by TestDistChaos_*
+// in tests/integration/, not what this test pins).
+func TestDistSet_PromotesOnGenericForwardError(t *testing.T) {
+	t.Parallel()
+
+	chaos := backend.NewChaos()
+
+	// 3 nodes, RF=3, ConsistencyOne. Chaos is wired onto every
+	// node's transport; we only ever Set from one node, so only
+	// that node's forwards exercise the promotion path.
+	dc := SetupInProcessClusterRF(
+		t, 3, 3,
+		backend.WithDistChaos(chaos),
+		backend.WithDistWriteConsistency(backend.ConsistencyOne),
+	)
+
+	a := dc.Nodes[0]
+	b := dc.Nodes[1]
+	c := dc.Nodes[2]
+
+	// Pick a key whose primary is `b` and where `a` is a replica
+	// owner. From `a`, the Set will forward to `b`; chaos drops
+	// that call; promotion should fire because `a` is in
+	// owners[1:].
+	desired := []cluster.NodeID{b.LocalNodeID(), a.LocalNodeID(), c.LocalNodeID()}
+
+	key, ok := FindOwnerKey(a, "promote-on-net-err-", desired, 5000)
+	if !ok {
+		t.Fatalf("could not find key with owner ordering [B, A, C]")
+	}
+
+	chaos.SetDropRate(1.0)
+
+	err := a.Set(context.Background(), &cache.Item{Key: key, Value: "v1"})
+	if err != nil {
+		t.Fatalf("Set: got %v, want nil (promotion should have succeeded)", err)
+	}
+
+	if !a.LocalContains(key) {
+		t.Errorf("LocalContains(%q) on promoting node: got false, want true", key)
+	}
+
+	if chaos.Drops() == 0 {
+		t.Errorf("chaos.Drops: got 0, want > 0 (chaos didn't see the forward attempt)")
+	}
+}