async mode: mysql pod stays `1/2 Running` indefinitely after restart — chicken-and-egg between `reconcileReplication` and `HeadlessService.PublishNotReadyAddresses=false`

[vvyushmanov]

Report

In async clusters, any single mysql pod restart (helm upgrade rolling restart, kubectl delete pod, node drain, eviction) can leave the restarted pod stuck 1/2 Running indefinitely. MySQL is up, replication is healthy and applying binlog events, but the startupProbe never returns success and the pod's started flag stays false. The cluster stays in state: initializing forever; manual kubectl delete pod --force recovers (the second attempt wins the race).

This is the "fresh issue with the latest details" requested when #626 was closed (#626 (comment)).

More about the problem

Three concurrent log signatures confirm the pattern (all from a real reproduction on a 3-node async cluster, mysql-0 was the affected pod):

1. Operator — every reconcile loop (~5 s):

   INFO reconcileReplication mysql is not ready, host not found. skip
        {"controllerKind": "PerconaServerMySQL", "name": "mcx-ps-db", "namespace": "mcx"}
   

2. Headless service Endpoints — affected pod is missing from the ready service but present in the -unready sibling:

   $ kubectl -n mcx get endpoints mcx-ps-db-mysql mcx-ps-db-mysql-unready
   NAME                      ENDPOINTS
   mcx-ps-db-mysql           10.42.2.227:3306, 10.42.3.102:3306                    # only mysql-1 and mysql-2
   mcx-ps-db-mysql-unready   10.42.1.143:3306, 10.42.2.227:3306, 10.42.3.102:3306  # all three
   

3. Orchestrator (<release>-ps-db-orc-0 orchestrator container) sees the unready pod as a phantom one-node cluster it cannot reach:

   WARNING DiscoverInstance(mcx-ps-db-mysql-0.mcx-ps-db-mysql.mcx:3306) instance is nil
           error=dial tcp: lookup mcx-ps-db-mysql-0.mcx-ps-db-mysql.mcx on 10.43.0.10:53: no such host
   WARNING executeCheckAndRecoverFunction: ignoring analysisEntry that has no action plan:
           DeadMasterWithoutReplicas; key: mcx-ps-db-mysql-0.mcx-ps-db-mysql.mcx:3306
   

   Meanwhile, on the affected pod's own mysql container, replication is fully operational:

   [System] Replica receiver thread for channel '': connected to source
            'replication@mcx-ps-db-mysql-1.mcx-ps-db-mysql.mcx:3306' ... starting GTID-based replication.
   [Note]   Multi-threaded replica statistics for channel '':
            seconds elapsed = 159; events assigned = 6145; ...
   

Root cause — chicken-and-egg between two design choices on main:

• pkg/mysql/mysql.go#L440-L458 — HeadlessService sets PublishNotReadyAddresses: cr.Spec.MySQL.IsGR(). In async mode this is false, so the ready service <release>-mysql deliberately omits unready pods. The per-pod FQDN <release>-mysql-N.<release>-mysql.<ns> only resolves once the pod is in the ready endpoints.
• pkg/controller/ps/controller.go#L905 — reconcileReplication calls orchestrator.Discover(ctx, ..., mysql.ServiceName(cr), DefaultPort). Orchestrator's discover endpoint resolves each pod by its per-pod FQDN against the ready service, which NXDOMAINs for the unready pod. The error maps to orchestrator.ErrNoSuchHost → log line in 2 above → return nil. No fallback to pod IP, no fallback to the -unready service.

The deadlock:

pod unready  →  ready-service Endpoints omits pod  →  per-pod FQDN NXDOMAINs
        ↑                                                       ↓
        ↑                          operator's orchestrator.Discover returns ErrNoSuchHost
        ↑                                                       ↓
        ↑                  reconcileReplication never registers pod in topology
        ↑                                                       ↓
        ↑                          startupProbe `/opt/percona/bootstrap` never confirmed ←

Group replication is unaffected because HeadlessService flips PublishNotReadyAddresses to true when IsGR() — the per-pod FQDN resolves regardless of readiness, and the operator's discover succeeds on the first attempt.

Steps to reproduce

1. Deploy a healthy 3-replica async cluster (clusterType: async, mysql.size: 3) and let it reach state: ready.
2. Restart any single mysql pod: kubectl delete pod <release>-ps-db-mysql-0 (or trigger via any spec change, helm upgrade, node drain, etc.).
3. Observe the pod come back, replicate from the primary, but stay 1/2 Running indefinitely. The PerconaServerMySQL resource stays in state: initializing.
4. Confirm by grepping the operator log for reconcileReplication ... host not found. skip and by checking that the pod's IP is in <release>-ps-db-mysql-unready Endpoints but NOT in <release>-ps-db-mysql Endpoints.
5. Recover with kubectl delete pod <release>-ps-db-mysql-0 --force --grace-period=0. The second attempt usually wins the race because the new pod's IP races into the ready service before the operator's next reconcile pass classifies it as unready.

The bug is timing-sensitive — sometimes the first restart self-heals, sometimes it deadlocks. Helm upgrade rolling restarts hit it most often (we see it almost every release).

Versions

1. Kubernetes: 1.34.5+k3s1 (also reported on 1.23 in MySQL StatefulSet can't rollout to ready state after cluster spec update #626).
2. Operator: 1.0.0 (verified the same code paths persist on main and v1.1.0).
3. Database: percona/percona-server:8.0.44

Anything else?

Affected versions. The relevant code in pkg/mysql/mysql.go and pkg/controller/ps/controller.go is unchanged between v1.0.0 and main (as of this filing). PR #722 / K8SPS-372 (v0.10.0/v0.11.0) fixed an adjacent async bootstrap race in 2-pod clusters but does not touch this code path.

Suggested fix directions (ranked from least to most invasive):

1. reconcileReplication fall back to the -unready service. Change the mysql.ServiceName(cr) argument at controller.go:905 to mysql.UnreadyServiceName(cr) in async mode. The unready service already exists and is already used by the bootstrap container itself (per PR #16) — extending it to the operator's reconcile path is internally consistent.
2. reconcileReplication retry on ErrNoSuchHost with pod IP. When the FQDN lookup fails, fall back to querying orchestrator with pod.Status.PodIP (the operator already has each mysql pod object via r.Get). Slightly larger change but doesn't depend on the secondary service.
3. Flip HeadlessService.PublishNotReadyAddresses=true for async. Most direct fix, single line at mysql.go:458. The downside: applications connecting to the headless service by name (rare — almost everyone uses the -haproxy or -router services) would briefly see unready pods in DNS responses. Probably worth the tradeoff but the other two options sidestep it entirely.

Workaround until a fix lands. Force-delete the stuck pod:

kubectl -n <namespace> delete pod <release>-ps-db-mysql-N --force --grace-period=0

Force-delete is safe because the affected pod is a replica that was never confirmed into orchestrator's topology — there's no clean-shutdown state to lose. The new pod boots from the existing PVC, reconnects to the primary, and on this second attempt the operator's reconcile loop wins the race against the startup-probe deadline.

Happy to test a candidate fix on our staging cluster if it'd help.

Related

• MySQL StatefulSet can't rollout to ready state after cluster spec update #626 — original report, closed for inactivity; collaborator @hors explicitly invited a fresh issue.
• K8SPS-372: fix bootstrap for cluster with 2 mysql pods #722 — adjacent async bootstrap fix, merged 2024-08-12.
• K8SPS-13 improve orchestrator discovery MySQL pods #16 — historical PR that introduced the -unready service for the bootstrap container's peer-list discovery.

[egegunes]

@vvyushmanov thank you for detailed report. We need to reproduce this but your analysis seems legit to me. I created https://perconadev.atlassian.net/browse/K8SPS-732 to fix this.

[vvyushmanov]

@egegunes thank you!
One reproduction note I think I forgot to include - this mainly happen when mysql-0 pod is the current master, and I do the rolling update.
Mysql pods restart in reverse ordinal sequence (2 -> 1 -> 0), 2 and 1 come back and healthily become active replicas again, then mysql-0 restarts, Orchestrator executes the failover, mysql-1 becomes the new master, and mysql-0 becomes deadlocked as described.