MachineDeployment controller creates duplicate MachineSets when informer cache sync times out during initial deployment

[frajamomo]

What steps did you take and what happened?

During initial spoke cluster deployment via CAPI with a Metal3 infrastructure provider, the MachineDeployment controller created 4 MachineSets (all with identical template specs) for a MachineDeployment with replicas: 2. This resulted in 8 worker Machines being created instead of 2, wasting bare-metal host resources and triggering an unnecessary scale-down.

Timeline from CAPI controller logs:

07:57:53 MachineSet spoke-0-worker-mlwjz created, scaled to 2 replicas
07:58:03 ERROR: "Failed to wait for cache to be up-to-date: timed out: MachineSet spoke-0-worker-mlwjz not found" (10s timeout)
07:58:03 MachineSet spoke-0-worker-zrgkk created, scaled to 2 replicas
07:58:13 ERROR: "Failed to wait for cache to be up-to-date: timed out: MachineSet spoke-0-worker-zrgkk not found" (10s timeout)
07:58:14 MachineSet spoke-0-worker-5qz58 created, scaled to 2 replicas
07:58:24 ERROR: "Failed to wait for cache to be up-to-date: timed out: MachineSet spoke-0-worker-5qz58 not found" (10s timeout)
07:58:25 MachineSet spoke-0-worker-vxl8q created, scaled to 2 replicas
07:58:29 Cache finally populates — all 4 MachineSets scale up simultaneously (8 Machines total)
Root cause: After creating a MachineSet, the controller waits up to 10 seconds for the informer cache to reflect it (internal/util/client/client.go). When this times out, the reconcile errors out and re-enters. On re-entry, computeDesiredMachineSet cannot find the previously created MachineSet in the cache, so it computes a new "desired" MachineSet with a fresh machine-template-hash random suffix and creates it. This repeats until the cache eventually syncs.

The machine-template-hash label includes a random suffix (by design, since #8585), so each new MachineSet gets a unique label value even though the template spec is identical. This prevents the controller from recognizing the previously created MachineSet as matching the desired state.

Evidence that all 4 MachineSets have identical specs:

spoke-0-worker-mlwjz: hash=694949418-mlwjz, revision=2, replicas=2
spoke-0-worker-zrgkk: hash=694949418-zrgkk, revision=1, replicas=0
spoke-0-worker-5qz58: hash=694949418-5qz58, revision=1, replicas=0
spoke-0-worker-vxl8q: hash=694949418-vxl8q, revision=1, replicas=2
All share the same base hash (694949418) — the template specs are identical. Only the random suffixes differ.

What did you expect to happen?

A MachineDeployment with replicas: 2 should create exactly 1 MachineSet with 2 replicas during initial deployment, even if the informer cache is slow to sync.

Cluster API version

v1beta2 (shipped via MCE 2.17, based on CAPI v1.12.x)

Kubernetes version

OpenShift 4.21 (k8s 1.34)

Anything else you would like to add?

Suggested fixes

1. Guard against duplicate creation: Before calling computeDesiredMachineSet, query the API server directly (not just the cache) for existing MachineSets owned by this MachineDeployment. If one already exists with generation == 1, skip creation.
2. Increase cache wait timeout: The 10-second timeout in internal/util/client/client.go may be too short for clusters under load. Consider making it configurable or increasing the default.
3. Use MachineDeployment generation as guard: Track which generation created the current MachineSet. If the MachineDeployment observedGeneration hasn't changed since the last MachineSet creation, don't create a new one.

[sbueringer]

v1beta2 (shipped via MCE 2.17, based on CAPI v1.9.x)

CAPI v1.9 does not support v1beta2

[frajamomo]

The MCE-shipped CAPI is not upstream v1.9.x — it's the OpenShift fork (openshift/cluster-api) which is based on a much newer upstream version. The main.version=1.25 and the presence of v1beta2 CRDs + controller-runtime v0.22.5 indicate this is based on upstream CAPI v1.12.x (the release branch that introduced v1beta2 support and uses controller-runtime 0.22.x).

[sbueringer]

Is this issue reproducible with upstream Cluster API? It seems strange to me that the informer does not sync after 10s

(we didn't get any reports from anyone else about this issue, so maybe it's OpenShift specific?)

[frajamomo]

The cache wait mechanism in internal/util/client/client.go (introduced in PR #12957) exists in the same form in upstream v1.12.x. The WaitForObjectsToBeAddedToTheCache() uses the same exponential backoff capping at ~10 seconds. When this timeout is exceeded, the reconciler re-enters, computeDesiredMachineSet doesn't find the previously created MachineSet in the cache, and creates a duplicate with a new random suffix in the machine-template-hash label.

I haven't reproduced this with a vanilla upstream CAPI deployment yet, but the code path is identical — the issue is not in any OpenShift-specific fork changes. It manifests when the informer cache is slow to sync (e.g., cluster under load during initial spoke deployment with multiple providers starting simultaneously, which was my case).

[frajamomo]

Reproduction with upstream CAPI v1.13.2

I cloned kubernetes-sigs/cluster-api at tag v1.13.2 and reduced the waitBackoff in internal/util/client/client.go to force the cache timeout to trigger deterministically:

-       Duration: 25 * time.Microsecond,
+       Duration: 1 * time.Nanosecond,
        Cap:      2 * time.Second,
        Factor:   1.2,
-       Steps:    63,
+       Steps:    1,

Then ran the existing MachineDeployment reconciler test:

go test ./internal/controllers/machinedeployment/ \ 
  -run 'TestMachineDeploymentReconciler/Should_reconcile_a_MachineDeployment$' \ 
  -v -count=1 -timeout 300s

Results

The test passes (the controller eventually self-corrects), but 3 distinct MachineSets were created for a single MachineDeployment with replicas: 2:

16:44:19 MachineSet md-x5c8m-sb7rn created (2 replicas)
16:44:25 MachineSet md-x5c8m-f95tg created (2 replicas) ← duplicate
16:44:27 MachineSet md-x5c8m-h54qz created (1 replica) ← duplicate

At the point h54qz was created, two MachineSets were active simultaneously with a combined 4 replicas:

machineSets={"md-x5c8m-f95tg":"0/3 replicas","md-x5c8m-h54qz":"0/1 replicas"}

The cache timeout error fires immediately after creating h54qz:

E0514 16:44:27 "Failed to wait for cache to be up-to-date"
err="failed to wait for up-to-date MachineSet objects in the cache after MachineSet creation:
timed out: MachineSet.cluster.x-k8s.io "md-x5c8m-h54qz" not found"
waitDuration="8.461µs"

Why this matters in production

In the envtest environment the controller self-corrects within seconds because the in-process cache catches up quickly. In a real cluster under load (our case: MCE hub deploying spoke clusters with multiple CAPI providers starting simultaneously), the cache can lag for 10+ seconds, which is enough to exceed the default waitBackoff (~10s) and trigger the same duplicate creation path — except recovery takes much longer because each duplicate MachineSet provisions real bare-metal hosts.

In our production case (reported in the original issue), 4 MachineSets were created for a replicas: 2 MachineDeployment, resulting in 8 worker Machines being provisioned on bare-metal before the controller scaled back down.

Root cause

The duplicate creation path is:

1. createOrUpdateMachineSetsAndSyncMachineDeploymentRevision() creates a MachineSet via SSA and calls WaitForObjectsToBeAddedToTheCache()
2. The wait times out → returns error → Reconcile() returns error → requeue
3. On re-entry, getAndAdoptMachineSetsForDeployment() reads from the informer cache — the newly created MachineSet is not yet visible
4. rolloutPlanner.init() → computeDesiredNewMS() generates a new MachineSet name via computeNewMachineSetName() (which appends a fresh apirand.String(5) suffix)
5. A duplicate MachineSet is created, and the cycle repeats until the cache syncs

[sbueringer]

Got it.

If I got it right this issue existed since CAPI v1.6?

I think this one should solve it: #13720

[kunalworldwide]

The timing here makes sense — the cache-wait timeout is a best-effort guard, not a guarantee, and when it fires the reconciler re-enters without any idempotency key to recognize the already-created MachineSet. The random suffix in machine-template-hash means each loop iteration produces something that looks "new" to the controller.

One thing worth documenting in the fix: this only manifests under slow informer sync (high API server load, large etcd response, slow control plane). In normal clusters the 10s window is more than enough. We hit this on a Metal3 cluster where the bootstrap etcd was under significant I/O pressure during the initial spoke deployment — informer sync consistently took 15–20s. Worth capturing that context in the issue or PR so future operators don't assume something is broken with their CAPI version.

If PR #13720 addresses it, would be good to confirm whether it's also targeted for the 1.9.x backport since that's still in support.

[sbueringer]

@kunalworldwide Why do you think 1.9 is still in support?

[kunalworldwide]

@sbueringer my bad, didn't check properly — I'm still running an older setup and got tunnel vision. 1.9's clearly EOL, the backport question doesn't apply here. Sorry for the noise!