fix: replace the blocking StatefulSet-ready wait with reconcile-loop gates

[xrl]

What

reconcileStatefulSet slept up to 45s inside the reconcile worker (waitForStatefulSetReady, exponential backoff polling ReadyReplicas == Replicas). That blocks the worker for the whole rollout, and with a single worker it head-of-line blocks every other EtcdCluster. This PR removes the blocking wait and re-plumbs the same predicate into the reconcile loop as requeue gates:

• reconcileStatefulSet now returns the CreateOrPatch response directly — fresher than the post-write cache read it replaces, which could still hold the pre-write object.
• Observed-generation gate: reconcileClusterState requeues before reading Spec.Replicas if the StatefulSet controller hasn't observed the latest spec write (own-write cache staleness), so the operator never acts on a spec the workload controller hasn't seen.
• Settled gate: membership mutations (learner promotion, scale-out member add) require ObservedGeneration caught up and ReadyReplicas == Replicas — the old wait's predicate, now non-blocking. The Owns(StatefulSet) watch plus the existing 10s requeue backstop drive convergence.

Gate placement is deliberate:

• The settled gate sits after the replica/member-count mismatch recovery, so an interrupted scale-in — whose orphaned pod has exited etcd and can never become ready — still unblocks.
• Spec-driven scale-in is exempt from the settled gate: reaching it means every member passed health checks (quorum-safe), and the member removed first is the last ordinal — often the very pod whose unreadiness prevents settling. Gating it would wedge a shrink away from a broken pod.
• While unsettled, performHealthChecks failures requeue instead of erroring: after bootstrap scales 0→1 the endpoint is unreachable until the pod starts, a window the old blocking wait absorbed — without this every new cluster inflates reconcile error metrics on the happy path. Once settled, health failures surface as errors exactly as before.
• A persistent settled-gate stall (etcd healthy but pod readiness lagging, e.g. dead kubelet) emits a StatefulSetNotSettled warning event, so it's visible on the CR instead of only in operator logs.

Files

• internal/controller/utils.go — delete waitForStatefulSetReady; createOrPatchStatefulSet returns the written object; add statefulSetSpecObserved / isStatefulSetSettled predicates.
• internal/controller/etcdcluster_controller.go — the two gates in reconcileClusterState, the unsettled health-check requeue in Reconcile, and the StatefulSetNotSettled event.

Review

An adversarial review pass produced 3 findings; all blocking/important ones resolved (second commit): the settled gate wedged spec-driven scale-in whenever a pod was NotReady but its etcd still answered — exactly the pod an admin shrinks away from (blocking — scale-in exempted, quorum-safety argued above); every new cluster erred through health checks during the bootstrap 0→1 window the old wait used to absorb (important — requeue while unsettled); and a persistent gate stall was invisible outside operator logs (resolved with the warning event).

Testing

• TestReconcileClusterStateGates: stale generation requeues without writes, unsettled replicas requeue before scale-out, spec-driven scale-in is exempt from the settled gate, interrupted scale-in recovery is not gated.
• TestReconcileClusterStateStaleGenerationEnvtest: envtest-backed own-write staleness scenario through the real cache.
• TestStatefulSetSpecObserved / TestIsStatefulSetSettled: predicate table tests (nil replicas, generation skew, partial readiness).
• internal/controller/utils_test.go reworked for createOrPatchStatefulSet returning the object; waitForStatefulSetReady tests deleted with the function.
• Full make test and make verify pass.

Descoped follow-ups

(1) In-memory expectations mechanism (ECK-style per-cluster expected-Generation map recorded after each write): descoped — the ObservedGeneration gate plus reconcileClusterState's existing replica/memberCnt cross-check (etcdcluster_controller.go:284, which turns own-write cache staleness into an idempotent re-patch) already neutralize the race once the blocking wait is gone; an expectations store adds a stateful component and restart semantics for marginal gain. Revisit with the MaxConcurrentReconciles follow-up if soak shows residual double-acts. (2) MaxConcurrentReconciles bump: explicitly excluded from this PR; noted as follow-up (see #379). (3) UpdatedReplicas/CurrentRevision==UpdateRevision checks in the settled predicate: descoped — the old wait only checked ReadyReplicas==Replicas; adding rollout-revision checks would be a behavior change beyond re-plumbing. (4) Gating bootstrapStatefulSet's 0→1 scale on the gate: unnecessary (idempotent re-patch, no pods yet) and it would add bootstrap latency visible to e2e. (5) New CRD status fields / conditions for "waiting on StatefulSet": descoped — no API change keeps the PR marker-free and generated-file-free. (6) Reworking performHealthChecks error noise during pod startup in general (e.g. treating connection-refused as requeue-not-error even when settled): descoped, separate concern.

🤖 Generated with Claude Code

---

PR series — operability fixes, TLS & EtcdMirror

Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.

	PR	Lands	After
	Reviewable now — small, any order
🟢	#374	Requeue instead of swallowing the client-cert provisioning error	—
🟢	#391	Grant events.k8s.io RBAC so operator Events are actually recorded	—
🟢	#392	Correlate members[]/leaderID from one health snapshot (consistent leader)	—
🟢	#393	Propagate user-supplied altNames.ipAddresses into certificates	—
🟢	#394	Accept day-suffix validityDuration (365d, 100d12h) as documented	—
🟢	#395	Surface early reconcile errors as a Degraded condition (was empty status)	—
🟢	#379	Configurable reconcile worker pool (--max-concurrent-reconciles)	—
🟢	#369	kind-based stress/scale e2e harness (1/3/7 members, churn, quorum watcher)	—
🟢	#398	podTemplate.spec scheduling fields: topologySpread, resources, priorityClass, schedulerName	—
🟢	#397	First-class Helm chart covering the full config/ surface	—
🟢	#399	Quorum-aware PodDisruptionBudget per EtcdCluster	—
🟢	#400	Scale-in removes the right member and transfers leadership first	—
🟢	#401	Backend quota + auto-compaction spec fields, NOSPACE alarm surfacing	—
🟢→	#402	Reconcile-loop gates replace the blocking StatefulSet-ready wait	—
🟢	#403	Quorum-gated one-pod-at-a-time version upgrades via OnDelete	—
	TLS stack — in order
🟢	#376	Independent spec.tls.{peer,client} surfaces (breaking alpha API)	—
⚪	#377	TLSReady condition + TLS lifecycle Events	#376
⚪	#378	Multi-member TLS quorum e2e + PeerCANotShared	#377
	EtcdMirror — in order
🟢	#406	EtcdMirror CRD: one-way cross-cluster replication API (types + CEL)	—
🟢	#407	pkg/mirroragent replication engine (fenced checkpoint-in-target)	#406
🟢	#408	Periodic reconciliation pass wired into the engine's steady-state loop	#407
🟢	#412	mirror-agent binary: config/TLS-reload/statusz/metrics + kind e2e	#408
🟢	#413	EtcdMirror controller: Deployment rendering, statusz-driven conditions, guards, fenced finalizer	#412
🟢	#414	CRD/sample/editor+viewer-role kustomize wiring	#413
🟢	#415	Controller-side phase/condition gauges, shipped PrometheusRule + dashboard	#414
🟢	#416	Full-lifecycle kind e2e: fence overlap, compaction+prune, restart resume, cert rotation, cutover/reversal	#415
	Parked as drafts pending #363
⚪	#382	Per-cluster domain metrics on the operator /metrics endpoint	—
⚪	#384	EtcdCluster admission webhooks (consolidating with #328)	#363
⚪	#386	EtcdBackup CR → object storage (S3/GCS)	#363
⚪	#387	Automatic quorum-loss disaster recovery (bootstrap-latch guarded)	#363

🟢 ready · ⚪ draft

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign ahrtr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.