`mongo.Dial` returns immediately on transient errors (EOF, server selection timeout), causing entire reconcile to fail

[mve720]

Report

In pkg/psmdb/mongo/mongo.go, the Dial function does a single
mongo.Connect + Ping and returns the error directly:

client, err := mongo.Connect(ctx, opts)
if err != nil {
    return nil, errors.Wrap(err, "connect mongo")
}
if err := client.Ping(ctx, readpref.Primary()); err != nil {
    return nil, errors.Wrap(err, "ping mongo")
}
return client, nil

There is no retry. Any transient error — EOF on socket close, server
selection timeout, context deadline — bubbles up immediately and fails
the entire reconcile.

More about the problem

The operator logs Dial failures like:

connect mongo: ... EOF
ping mongo: server selection error: server selection timeout

and the reconcile aborts. The operator relies entirely on the next
reconcile to retry, which restarts the whole flow from scratch.

This is particularly painful in service-mesh deployments (Istio/ASM):
after a new mongo pod is created, the client-side proxy may take
seconds-to-minutes to receive the necessary CDS/EDS config from istiod.
During this window the proxy falls back to PassthroughCluster and
sends plain TCP, which the mongo-side proxy (STRICT mTLS) rejects with
filter_chain_not_found — manifesting as EOF on the client.

Combined with the non-idempotent handleReplsetInit (see related
issue #2374), this single transient failure can permanently break a
freshly-deployed cluster: the Dial failure aborts the reconcile right
after createUser consumed the localhost-exception, and the retry can
never finish the bootstrap.

Steps to reproduce

1. Kubernetes cluster with cluster-wide STRICT PeerAuthentication
   (Istio/ASM)
2. Apply a PerconaServerMongoDB CR
3. Observe Dial failing with EOF / "server selection timeout" while
   the sidecar of the new mongo pod waits for EDS config
4. The operator does not retry within the reconcile; the reconcile
   fails, cluster init stays incomplete

Versions

• Operator: 1.22.x (tested); Dial in pkg/psmdb/mongo/mongo.go is
  unchanged in current main
• Kubernetes: GKE with managed ASM (Istio data plane, Traffic Director
  control plane), cluster-wide STRICT mTLS
• EDS-sync windows measured in production: up to ~2.5 min

Anything else?

Suggested fix — wrap Connect + Ping in
wait.ExponentialBackoff with a transient-error filter. Only retry on
connection-level errors (EOF, timeout) — NOT on auth errors or other
definitive failures:

import "k8s.io/apimachinery/pkg/util/wait"

var client *mongo.Client
err := wait.ExponentialBackoff(wait.Backoff{
    Steps:    10,
    Duration: 2 * time.Second,
    Factor:   1.5,
    Cap:      30 * time.Second,
}, func() (bool, error) {
    c, err := mongo.Connect(ctx, opts)
    if err != nil {
        if isTransientError(err) { return false, nil }
        return false, err
    }
    if pingErr := c.Ping(ctx, readpref.Primary()); pingErr != nil {
        c.Disconnect(ctx)
        if isTransientError(pingErr) { return false, nil }
        return false, pingErr
    }
    client = c
    return true, nil
})

Total retry budget ~5 min with exponential backoff — covers the Istio
EDS-sync windows we've measured in production (up to ~2.5 min).

[mve720]

I couldn't create a task in Jira because the dialog keeps loading forever.

@egegunes Can I open a PR for this? I'd love to get this bugfix into version 1.23 if possible.

[egegunes]

@mve720 I created https://perconadev.atlassian.net/browse/K8SPSMDB-1677 to address this.

[egegunes]

Can I open a PR for this? I'd love to get this bugfix into version 1.23 if possible.

@mve720 of course you can, let me know if you need any help.