feat(relayenv): re-anchor mechanism

.agent-docs/concurrent-keys/phase1-design.md

@@ -253,6 +253,25 @@ This is the highest-risk piece of Phase 1. The **T0 PoC** validated the swap mec
 
 This order is *necessary* — the PoC found that flipping the pointer too early leaves `GetClient()` nil mid-swap (H6) and breaks the env on init failure (H7) — and *sufficient* only with the store-handling approach below.
 
+### Two re-anchor cases — Case A (new key) vs Case B (already-accepted key)
+
+The sequence above is **Case A**: the new anchor is a key relay has not previously accepted, so no SDK client exists for it. Relay must build one, hand over the store, wait for `Initialized()`, then flip and re-wire.
+
+**Case B** is the abbreviated path taken when the new anchor's key **already has a live SDK client** — most commonly a *former* anchor that is still inside its grace period (it was demoted on an earlier rotation, kept alive to serve downstream traffic, and is now being promoted back). In that situation:
+
+1. **No `Build`.** The existing client is reused as-is — there is no second upstream connection to stand up.
+2. **No store handover.** The store the existing client created is already populated and initialized; nothing is handed over because nothing new is constructed.
+3. **Atomically flip the rotator's anchor pointer** (`CommitAnchor`) — identical to Case A step 3.
+4. **Call `ReplaceCredential`** on the event dispatcher + metrics publisher — identical to Case A step 4.
+5. Big-segment sync is re-wired (T2.d), identical to Case A.
+6. The retiring anchor's client is closed by the existing `removeCredential` path when its own grace period ends — identical to Case A step 6.
+
+The two paths **converge after the flip**: steps 3–6 are the same. The only difference is the front of the sequence — Case A builds + initializes + hands over the store; Case B reuses what is already there and does none of that. The caller branches on whether a client already exists for the new anchor (`c.clients[newAnchor] != nil`).
+
+Because Case B does no client build, there is no init-failure rollback to consider for it — the client it reuses was already initialized and serving. Rollback handling (preserve previous anchor, log a structured error) applies to **Case A only**.
+
+**Reconcile/additions interaction:** so the synchronous re-anchor owns the new anchor's setup end-to-end, `Rotator.Reconcile` does not flip the anchor itself — it returns a `ReconcileResult.AnchorChange` and the caller invokes `CommitAnchor` at the right moment. In **Case A** the new anchor would otherwise appear in the reconcile's `additions` list and `addCredential` would fire an *async* `startSDKClient` that races the synchronous build — so Reconcile strips the new anchor from `additions` in Case A and the synchronous path installs the peripherals (envStreams, handlers, connection mapping) itself. In **Case B** the new anchor was already accepted, so it was never going to appear in `additions` — no stripping is needed there.
+
 ### The data store: hand the existing store over to the new client
 
 An earlier version of this design assumed two SDK clients pointed at the same env would feed the *same* data store as a side-effect. The PoC (H1, H5) showed this is **wrong for the in-memory store**: each SDK client construction calls `storeAdapter.Build()`, which atomically swaps in a *new, empty* store, so the new client would otherwise have to re-sync from scratch (an empty-store window). This affects only the in-memory case; with a persistent store (Redis, DynamoDB) the data lives outside the wrapper and survives the swap.
@@ -276,17 +295,17 @@ This is the concrete form of decoupling the store's lifecycle from the client's.
 
 ### Failure handling
 
-If the new client fails to initialize, the swap **rolls back**: the rotator's anchor pointer stays on the old key, the previous accepted set is preserved, a structured error is logged, and an alarm is raised. The old anchor's client (still alive in its grace period) continues to serve. This is the §8 atomicity principle applied to re-anchor.
+If the new client fails to initialize, the swap **rolls back**: the rotator's anchor pointer stays on the old key (the caller simply does not call `CommitAnchor`), the previous accepted set is preserved, and a structured error is logged. The old anchor's client (still alive in its grace period) continues to serve. This is the §8 atomicity principle applied to re-anchor, and it applies to **Case A only** — Case B reuses an already-initialized client and has nothing to fail. Relay has no dedicated alarm infrastructure today; an `Error`-level structured log (`globalLoggers.Errorf`) is the strongest signal available and is sufficient.
 
 ### Consolidated specification for T2.c / T2.d
 
 | # | Requirement | Source | Owner |
 |---|---|---|---|
-| 1 | Build + initialize the new anchor client *before* flipping the pointer; flip atomically. | H5, H6 | T2.c |
-| 2 | On init failure, roll back to old anchor; preserve previous accepted set; log + alarm. | H7 | T2.c |
-| 3 | Hand the existing store over to the new client (adapter reuses its store); ensure the retiring client's `Close()` does not tear down the shared store. | H1, H5 | T2.c |
-| 4 | Re-wire big-segment sync on re-anchor (recreate or replace-credential). | H3 | T2.d |
-| 5 | Call `ReplaceCredential` on event dispatcher + metrics publisher. | §7 | T2.c (already wired in `addCredential`) |
+| 1 | **Case A**: build + initialize the new anchor client *before* flipping the pointer; flip atomically via `CommitAnchor`. **Case B** (new anchor already has a live client): skip the build, reuse it, then flip. | H5, H6 | T2.c |
+| 2 | **Case A** init failure: roll back (do not `CommitAnchor`); preserve previous accepted set; log a structured error. Not applicable to Case B. | H7 | T2.c |
+| 3 | **Case A**: hand the existing store over to the new client (adapter reuses its store, refcounted so the retiring client's `Close()` does not tear it down). **Case B**: no handover — the store is already populated. | H1, H5 | T2.c |
+| 4 | Re-wire big-segment sync on re-anchor (recreate or replace-credential). Same for both cases. | H3 | T2.d |
+| 5 | Call `ReplaceCredential` on event dispatcher + metrics publisher (synchronously, in the re-anchor sequence — not via `addCredential`, since Reconcile strips the new anchor from `additions`). Same for both cases. | §7 | T2.c |
 | 6 | Expect duplicate downstream `put`; retain connections for credentials still in the accepted set. | H2 | T2.c (awareness) |
 | 7 | No `httpconfig` change. | H4 | n/a |
 

internal/credential/rotator.go

@@ -2,6 +2,7 @@ package credential
 
 import (
 	"maps"
+	"slices"
 	"sync"
 	"time"
 
@@ -209,6 +210,42 @@ func (r *Rotator) StepTime(now time.Time) (additions []SDKCredential, expiration
 	return additions, expirations
 }
 
+// ReconcileResult signals state changes that the caller must apply synchronously rather than rely
+// on the normal addCredential / removeCredential flow driven by StepTime.
+//
+// AnchorChange is non-nil when the SDK anchor changed during Reconcile. The rotator does NOT flip
+// its anchor pointer in that case — the caller must drive the synchronous re-anchor sequence
+// (build the new anchor's SDK client if one does not exist, wait for Initialized, then invoke
+// CommitAnchor to atomically move the pointer, then call ReplaceCredential on the event dispatcher
+// and metrics publisher, then re-wire big-segment sync).
+//
+// MobilePrimaryRepoint is non-nil when the primary mobile key changed AND the new primary was
+// already in the accepted set. In that case it does not appear in StepTime's additions list and
+// addCredential's primary-mobile gate will not fire for it, so the caller must invoke
+// eventDispatcher.ReplaceCredential synchronously. When nil, either the primary mobile key did not
+// change, or it changed to a newly-accepted key — in which case the normal addCredential path
+// handles the ReplaceCredential call via the existing gate.
+type ReconcileResult struct {
+	AnchorChange         *AnchorChange
+	MobilePrimaryRepoint *config.MobileKey
+}
+
+// AnchorChange describes an SDK anchor transition produced by Reconcile.
+//
+// NewAnchorPreviouslyAccepted distinguishes the two re-anchor paths:
+//   - false (the anchor is a new key): the new anchor was not previously in the accepted set. The
+//     synchronous re-anchor must register the credential mappings (envStreams, handlers, connection
+//     mapping), construct and initialize a new SDK client, then invoke CommitAnchor + ReplaceCredential.
+//   - true (the anchor is a previously-accepted key): the new anchor was already accepted (typically
+//     a former anchor still in its grace period). Its credential mappings are already registered and a
+//     client may already exist; the synchronous re-anchor reuses it (or constructs one only if missing
+//     — see the re-anchor sequence in env_context_impl.go), then invokes CommitAnchor + ReplaceCredential.
+type AnchorChange struct {
+	PreviousAnchor              config.SDKKey
+	NewAnchor                   config.SDKKey
+	NewAnchorPreviouslyAccepted bool
+}
+
 // Reconcile updates the rotator to match set. The set names its own anchor (the primary SDK key) and
 // primary mobile key. It diffs the desired accepted set against the current one and queues additions
 // and expirations (drained by the next StepTime call); keys newly present are accepted, and keys no
@@ -220,13 +257,101 @@ func (r *Rotator) StepTime(now time.Time) (additions []SDKCredential, expiration
 // The set is assumed well-formed: AcceptedSetBuilder.Build validates that an anchor was designated
 // (and, because WithAnchor adds the key as it designates it, that the anchor is among the SDK
 // keys), so Reconcile trusts what it is handed rather than re-validating.
-func (r *Rotator) Reconcile(set AcceptedSet, now time.Time) {
+//
+// Reconcile does NOT flip the SDK anchor pointer when the anchor changes — the returned
+// ReconcileResult.AnchorChange signals the change so the caller can drive the synchronous re-anchor
+// sequence, then call CommitAnchor to atomically move the pointer. When the new anchor is a new key
+// (NewAnchorPreviouslyAccepted == false) it is also stripped from additions so that the async
+// startSDKClient invocation in addCredential does not race the synchronous client build.
+func (r *Rotator) Reconcile(set AcceptedSet, now time.Time) ReconcileResult {
 	r.mu.Lock()
 	defer r.mu.Unlock()
 
-	r.reconcileSDKKeys(set, set.anchor, now)
+	var result ReconcileResult
+
+	previousAnchor := r.anchorKey
+	newAnchor := set.anchor
+	if previousAnchor != newAnchor && newAnchor.Defined() {
+		_, alreadyAccepted := r.acceptedSDKKeys[newAnchor]
+		result.AnchorChange = &AnchorChange{
+			PreviousAnchor:              previousAnchor,
+			NewAnchor:                   newAnchor,
+			NewAnchorPreviouslyAccepted: alreadyAccepted,
+		}
+	}
+
+	r.reconcileSDKKeys(set, now)
+
+	if result.AnchorChange != nil && !result.AnchorChange.NewAnchorPreviouslyAccepted {
+		// The anchor is a new key: reconcileAcceptedKeys just appended it to r.additions. Strip it —
+		// the synchronous re-anchor sequence in env_context_impl owns the new anchor's setup
+		// (credential mappings + client build + flip + ReplaceCredential). If addCredential drained this
+		// addition normally, its async startSDKClient would race the synchronous build. When the anchor
+		// is a previously-accepted key it was already in acceptedSDKKeys, so reconcileAcceptedKeys did
+		// not add it — no strip needed.
+		r.additions = slices.DeleteFunc(r.additions, func(c SDKCredential) bool { return c == newAnchor })
+	}
+
+	previousMobile := r.primaryMobileKey
+	newMobile := set.primaryMobileKey
+	var newMobileAlreadyAccepted bool
+	if newMobile.Defined() {
+		_, newMobileAlreadyAccepted = r.acceptedMobileKeys[newMobile]
+	}
 	r.reconcileMobileKeys(set, now)
+	if previousMobile != newMobile && newMobile.Defined() && newMobileAlreadyAccepted {
+		// Primary mobile key changed to a key already in the accepted set: addCredential's gate will
+		// not fire for it (it's not in additions), so the caller must call ReplaceCredential itself.
+		m := newMobile
+		result.MobilePrimaryRepoint = &m
+	}
+
 	r.reconcileEnvironmentID(set)
+
+	return result
+}
+
+// CommitAnchor atomically moves the rotator's SDK anchor pointer to the given key. The caller
+// invokes this once the synchronous re-anchor sequence is ready to flip — i.e. after the new
+// anchor's client is built and reports Initialized (when the anchor is a new key) or after confirming
+// the existing client will be reused (when the anchor is a previously-accepted key). Until CommitAnchor
+// is called, the rotator's anchor stays on the previous key so GetClient() returns the still-serving
+// old client and the gate in addCredential does not fire for the pending new anchor.
+//
+// Aside from Initialize (which establishes the initial anchor), CommitAnchor is the only path that
+// moves the anchor pointer: Reconcile deliberately does not flip it (see reconcileSDKKeys).
+func (r *Rotator) CommitAnchor(key config.SDKKey) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.anchorKey = key
+}
+
+// RevertAnchorChange undoes the accepted-set effects of an AnchorChange whose synchronous re-anchor
+// failed and rolled back. Because CommitAnchor was never called, the anchor pointer still names the
+// previous anchor; this realigns the accepted set with it so the two don't disagree.
+//
+//   - If the previous anchor is a defined key that was revoked in the same reconcile (it is no longer
+//     accepted — an immediate revocation rather than a grace demotion), re-admit it as a permanent
+//     key, since it remains the anchor and keeps serving. If it is still accepted (grace demotion),
+//     leave it and its expiry untouched. An undefined previous anchor (the env's first SDK key) is
+//     never admitted.
+//   - Drop the failed new anchor, but only if it was brand new; a previously-accepted key that was
+//     promoted and failed stays accepted as the non-anchor key it already was.
+func (r *Rotator) RevertAnchorChange(change AnchorChange) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	// Only re-admit a defined previous anchor. When an env gains its first SDK key, the previous anchor
+	// is the empty (undefined) key — there is nothing to re-admit, and inserting "" would put an
+	// undefined credential into the accepted set (the rotator otherwise only holds defined keys).
+	if change.PreviousAnchor.Defined() {
+		if _, stillAccepted := r.acceptedSDKKeys[change.PreviousAnchor]; !stillAccepted {
+			r.acceptedSDKKeys[change.PreviousAnchor] = AcceptedKey{}
+		}
+	}
+	if !change.NewAnchorPreviouslyAccepted {
+		delete(r.acceptedSDKKeys, change.NewAnchor)
+	}
 }
 
 // reconcilableKey constrains the generic reconcile helper to a comparable credential (so it can key a
@@ -276,7 +401,11 @@ func reconcileAcceptedKeys[K reconcilableKey](
 // reconcileAcceptedKeys. The set is trusted as well-formed: BuildAcceptedSet / the builder guarantee
 // the anchor is present and permanent (WithAnchor forces a nil expiry), so no special handling is
 // needed here. The caller must hold the write lock.
-func (r *Rotator) reconcileSDKKeys(set AcceptedSet, anchor config.SDKKey, now time.Time) {
+//
+// NOTE: reconcileSDKKeys does NOT flip r.anchorKey when the anchor changes. The Reconcile caller
+// signals the anchor change via ReconcileResult.AnchorChange and invokes CommitAnchor to move the
+// pointer once the synchronous re-anchor sequence is ready (see Reconcile + CommitAnchor).
+func (r *Rotator) reconcileSDKKeys(set AcceptedSet, now time.Time) {
 	desired := make(map[config.SDKKey]AcceptedKey, len(set.sdkKeys))
 	for key, info := range set.sdkKeys {
 		if info.Expiry != nil && !now.Before(*info.Expiry) {
@@ -285,7 +414,6 @@ func (r *Rotator) reconcileSDKKeys(set AcceptedSet, anchor config.SDKKey, now ti
 		desired[key] = info
 	}
 	reconcileAcceptedKeys(desired, r.acceptedSDKKeys, &r.additions, &r.expirations, r.loggers, "SDK key")
-	r.anchorKey = anchor
 }
 
 // reconcileMobileKeys mirrors reconcileSDKKeys for mobile keys. The set is trusted as well-formed: