[Feature Request] [occm] Controller to clean-up OCCM orphaned openstack ressources

[Banh-Canh]

/kind feature

What happened:

If we provision a cluster with OCCM controlled resources, then delete it directly, OCCM can't and will never remove openstack resources.
This is because the cluster and the controller are removed before any garbage collection and proper cascading deletion can happen.

What you expected to happen:

This is not something we can solve in OCCM itself as it is getting removed.
We need another controller running outside of the said provisioned cluster to manage and detect orphaned resources.

How to reproduce it:

• Provision a cluster
• Deploy OCCM on it
• Create some OCCM CRs (loadbalancer?, network?)
• OCCM provision thoses resources in Openstack
• Delete the cluster directly
• The openstack resources are now orphaned

Anything else we need to know?:

This is a common use case when using Cluster API openstack provider and OCCM.
I think such feature would help keeping openstack free from orphaned resources and the need to clean that up manually.

[jihyun-huh]

Hi, @Banh-Canh Thanks for you report,

We've run into the same problem in production level before, particularly using with Cluster-API.
when Cluster-API delete the cluster, it also removes OCCM before clean up the OpenStack resources.

In our case we ended up solving it outside of OCCM, with a separate cleanup controller logic, so I definitely think this is a valuable use case.

and since OCCM itself cannot perform the cleanup once the cluster is gone,
this feels more like a lifecycle/architecture problem than an OCCM bug.
A dedicated external controller such as Cluster-api-proivder-openstack (or another component responsible for cluster teardown) sounds like a reasonable direction to explore.

I think this is worth discussing further as a feature proposal.