feat: quorum-gated one-pod-at-a-time version upgrades via OnDelete

[xrl]

What

A pure .spec.version change validated the upgrade path but never re-rendered the StatefulSet — the cluster silently stayed on the old image (repro test in the first commit). This PR makes version changes actually roll out, one pod at a time, gated on cluster health:

• OnDelete update strategy. A spec edit re-renders the pod template without restarting anything; the controller then deletes one outdated pod per reconcile, and only while every up-to-date pod is Ready, a leader exists, and every member's revision is within 90% of the leader's. The leader's pod is replaced last.
• Outdated pods detected by controller revision, not image string: each pod's controller-revision-hash label is compared against status.updateRevision. Image comparison broke under pod-mutating admission webhooks (digest/mirror rewrites made every pod permanently "outdated", deleting a healthy member in a loop) and missed non-image template drift, which OnDelete no longer converges on its own. The template is also re-rendered whenever the cluster generation is unobserved, so etcdOptions/podTemplate/TLS edits roll out too.
• Pods read via an uncached API reader (mgr.GetAPIReader()): the cached client's first pod Get lazily starts an informer that lists/watches all pods cluster-wide, but the operator only ever reads its own N pods by name. RBAC gains pods get;delete only — no list/watch.
• Failed upgrades recover instead of wedging. A pod replaced with a broken image (phantom patch tag, flag removed in the new minor) failed every subsequent health check, which blocked template re-rendering, made a spec.version rollback look like a downgrade of the never-run template tag, and left the cluster degraded at N-1 members with no spec-level way out. Upgrade-path validation now runs against status.currentVersion (what members actually run) so rollback is accepted; on health-check failure the template is re-synced and, while the remaining members still hold quorum, one not-Ready outdated pod is replaced so the fixed template can land. Not-Ready outdated pods are exempt from the all-Ready gate — requiring the victim Ready deadlocked its own replacement.
• docs/upgrade.md documents the mechanics, rollback, and the manual step needed after quorum loss.

Files

• internal/controller/upgrade.go — reconcileVersionUpgrade (gates + victim ordering: not-Ready outdated first, then lowest ordinal, leader last), recoverDegradedUpgrade, isPodOutdated, ordinalFromPeerEp (memberHealth sorts lexically by endpoint, so slice index != ordinal once i >= 10).
• internal/controller/utils.go — OnDelete strategy on the StatefulSet; syncStatefulSet split out of reconcileStatefulSet so the recovery path can apply the template while pods are down, when waitForStatefulSetReady could never succeed.
• internal/controller/etcdcluster_controller.go — PodReader field, the upgrade call where the up-to-date branch used to no-op, the degraded-recovery hook on health-check failure, and the status.currentVersion preference in upgrade-path validation.
• config/rbac/role.yaml — regenerated via make manifests.

Review

An adversarial review pass produced 3 findings, all blocking/important, each resolved in its own commit: cached-client pod reads silently starting a cluster-wide pod informer (fixed with the uncached reader), image-string outdated detection deleting healthy members in a loop under pod-mutating webhooks and missing non-image drift (fixed with controller-revision detection), and failed upgrades wedging the cluster with no spec-level rollback (fixed with the recovery path).

Testing

• upgrade_test.go (envtest): re-render on version bump and on unobserved spec change; one pod deleted per reconcile; leader pod last; gates hold on a lagging member, a not-Ready or terminating pod, and a pending learner; no-op when all pods are updated; requeue on a missing pod; webhook-mutated pod image ignored; non-image template change rolls pods; not-Ready outdated pod exempt from the Ready gate and replaced; recoverDegradedUpgrade table (quorum lost, victim on current template, replacement in flight); ordinalFromPeerEp parsing including ordinals >= 10.
• TestFetchAndValidateState extended for rollback via status.currentVersion.
• Full make test and make verify pass.

Descoped follow-ups

(1) No readiness probe on the etcd container: TLS listener flags arrive only via free-form spec.etcdOptions (defaultArgs hard-codes http, utils.go:80-88), so the probe scheme and client-cert requirements cannot be derived reliably; the gate uses etcd revision checks, which are stronger than pod readiness. Follow-up issue. (2) No MoveLeader before deleting the leader pod — the helper is absent from origin/main (it lives on the unmerged scale-in branch, #400); leader-last ordering is used instead, to be deduped at merge time. (3) No upgrade e2e: the harness requires kind, which this environment cannot run; a test sketch is documented instead. (4) No Progressing condition with Reason=UpgradeInProgress in updateConditions — cosmetic, follow-up. (5) No Warning Event when rolling a cluster without persistent storage (r.Recorder is wired but currently unused anywhere, and wiring events adds test surface) — documented in docs/upgrade.md instead. (6) Upgrades are not refused/paused for spec.storageSpec == nil clusters — documented caveat only.

🤖 Generated with Claude Code

---

PR series — operability fixes, TLS & EtcdMirror

Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.

	PR	Lands	After
	Reviewable now — small, any order
🟢	#374	Requeue instead of swallowing the client-cert provisioning error	—
🟢	#391	Grant events.k8s.io RBAC so operator Events are actually recorded	—
🟢	#392	Correlate members[]/leaderID from one health snapshot (consistent leader)	—
🟢	#393	Propagate user-supplied altNames.ipAddresses into certificates	—
🟢	#394	Accept day-suffix validityDuration (365d, 100d12h) as documented	—
🟢	#395	Surface early reconcile errors as a Degraded condition (was empty status)	—
🟢	#379	Configurable reconcile worker pool (--max-concurrent-reconciles)	—
🟢	#369	kind-based stress/scale e2e harness (1/3/7 members, churn, quorum watcher)	—
🟢	#398	podTemplate.spec scheduling fields: topologySpread, resources, priorityClass, schedulerName	—
🟢	#397	First-class Helm chart covering the full config/ surface	—
🟢	#399	Quorum-aware PodDisruptionBudget per EtcdCluster	—
🟢	#400	Scale-in removes the right member and transfers leadership first	—
🟢	#401	Backend quota + auto-compaction spec fields, NOSPACE alarm surfacing	—
🟢	#402	Reconcile-loop gates replace the blocking StatefulSet-ready wait	—
🟢→	#403	Quorum-gated one-pod-at-a-time version upgrades via OnDelete	—
	TLS stack — in order
🟢	#376	Independent spec.tls.{peer,client} surfaces (breaking alpha API)	—
⚪	#377	TLSReady condition + TLS lifecycle Events	#376
⚪	#378	Multi-member TLS quorum e2e + PeerCANotShared	#377
	EtcdMirror — in order
🟢	#406	EtcdMirror CRD: one-way cross-cluster replication API (types + CEL)	—
🟢	#407	pkg/mirroragent replication engine (fenced checkpoint-in-target)	#406
🟢	#408	Periodic reconciliation pass wired into the engine's steady-state loop	#407
🟢	#412	mirror-agent binary: config/TLS-reload/statusz/metrics + kind e2e	#408
🟢	#413	EtcdMirror controller: Deployment rendering, statusz-driven conditions, guards, fenced finalizer	#412
🟢	#414	CRD/sample/editor+viewer-role kustomize wiring	#413
🟢	#415	Controller-side phase/condition gauges, shipped PrometheusRule + dashboard	#414
🟢	#416	Full-lifecycle kind e2e: fence overlap, compaction+prune, restart resume, cert rotation, cutover/reversal	#415
	Parked as drafts pending #363
⚪	#382	Per-cluster domain metrics on the operator /metrics endpoint	—
⚪	#384	EtcdCluster admission webhooks (consolidating with #328)	#363
⚪	#386	EtcdBackup CR → object storage (S3/GCS)	#363
⚪	#387	Automatic quorum-loss disaster recovery (bootstrap-latch guarded)	#363

🟢 ready · ⚪ draft

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign hakman for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.