fix(proxmoxmachine): tolerate owner Machine NotFound to unblock stuck deletes

[raul-cq]

What this fixes

The ProxmoxMachine reconciler currently deadlocks any ProxmoxMachine whose owner Machine has been removed before the InfraMachine was finalized. Reproduction: take any cluster, remove a Machine out-of-band (manual finalizer strip, cluster.x-k8s.io/force-delete-machine annotation, or any other actor) before its ProxmoxMachine finishes finalizing. The ProxmoxMachine then sits in Terminating forever and blocks the parent ProxmoxCluster and Cluster from finalizing.

Root cause

internal/controller/proxmoxmachine_controller.go#L97-L105:

machine, err := util.GetOwnerMachine(ctx, r.Client, proxmoxMachine.ObjectMeta)
if err != nil {
    return ctrl.Result{}, err
}

util.GetOwnerMachine returns nil, nil only when there is no Machine OwnerRef at all. When the OwnerRef is present but the Machine object is gone, it calls client.Get and returns apierrors.NotFound. The reconciler returns that error verbatim, controller-runtime backs off exponentially, and the delete branch (if !proxmoxMachine.ObjectMeta.DeletionTimestamp.IsZero()) is never reached. The MachineFinalizer never comes off.

The log signature observed in production:

"Reconciler error" err="Machine.cluster.x-k8s.io \"<name>\" not found"

repeating every backoff cycle for hours.

This is a known class of issue: docs/Troubleshooting.md under "Machine deletion deadlock" already documents the manual workaround (strip the proxmoxmachine finalizer by hand). This PR addresses the underlying controller behaviour so the manual workaround is no longer needed.

What this PR does

In Reconcile, when GetOwnerMachine returns IsNotFound:

• If the ProxmoxMachine has a DeletionTimestamp: remove the MachineFinalizer and return cleanly. A log line tells the operator to verify the underlying Proxmox VM, since a full machineScope cannot be constructed without the owner Machine and vmservice.DeleteVM requires the scope. This is the conservative choice: unblock the controller-level deadlock without taking destructive action against the hypervisor under partial state.
• If the ProxmoxMachine is not being deleted: log and return nil instead of error-looping. Either the OwnerRef will be corrected (which triggers another reconcile via the watch) or the ProxmoxMachine will be deleted (which triggers another reconcile via the delete event).

All other failure modes still propagate as before.

Tests

Adds an envtest in proxmoxmachine_controller_test.go covering the deletion-with-missing-owner case: the test creates a ProxmoxMachine with a Machine OwnerRef pointing to a non-existent Machine and the MachineFinalizer set, deletes it (the API server records a deletionTimestamp since the finalizer is held), runs Reconcile, and asserts the object is reaped (i.e. the finalizer was removed).

$ make test WHAT=./internal/controller/...
ok  github.com/ionos-cloud/cluster-api-provider-proxmox/internal/controller  30.534s  coverage: 50.6% of statements

Compatibility

• No CRD changes.
• No public API changes.
• The IONOS Troubleshooting doc workaround still works as a runtime escape hatch for clusters that were stuck before this fix is deployed.

Why I'm sending this

We hit this in production (CAPMOX v0.8.1) when a workload cluster's apiserver became unreachable mid-delete and a Machine ended up reaped before its InfraMachine finalized — the exact deadlock above. Walking the diagnosis backward led here. Happy to iterate on the patch shape, the log wording, or the test scaffolding.

[mcbenjemaa]

I wonder how did you reproduce this?
as the InfraMachines will be deleted as long as the parent Machine is being deleted which is why it has an ownerReference.
in fact, the CAPI machine controller waits for the infraMachine to be deleted first, then cleanup the finalizer and deletes the Machine.

Do you have something specific that lets the parent machine gone, and the inframachine remain?

[raul-cq]

Hi there... this seems to be a corner case. When creating a cluster one worker vm creation got stuck in cloud-init, i just destroyed the vm in proxmox, and then proceeded to delete the cluster. This is how i hit this behaviour.

[wikkyk]

This comment (that looks AI-generated btw, please attribute it) is excessive. Convert the comments in your PR to one-liners and instead elaborate in the commit message.