Add `spec.kubeconfig.metadata` to allow setting annotations/labels on generated kubeconfig secrets

[jonathanrainer]

What would you like to be added (User Story)?

As a user I would like to be able to set custom annotations and labels on the secrets that are created that hold the KubeConfig data. I could then use these custom annotations to allow Cluster API to integrate better with tools like reflector (https://github.com/emberstack/kubernetes-reflector) which require you to annotate source secrets.

Detailed Description

CAPI creates a kubeconfig Secret for each managed cluster but provides no way to set
custom annotations or labels on that secret. Tools that gate on annotations (e.g.
Reflector, which requires reflector.v1.k8s.emberstack.com/reflection-allowed: "true")
cannot be used without some kind of workaround, like a CronJob that continuously re-patches the annotation
after CAPI reconciliation removes it.

This issue proposes adding spec.kubeconfig.metadata.{annotations,labels} to ClusterSpec
so that CAPI sets and maintains these values on the generated kubeconfig secret.

Use case

Reflector (https://github.com/emberstack/kubernetes-reflector) mirrors Kubernetes secrets
across namespaces. It requires the source secret to carry
reflector.v1.k8s.emberstack.com/reflection-allowed: "true". Any tool with similar
annotation-based admission gating has the same problem.

Prior art

This was previously proposed in #9651 and closed for lack of a concrete use case.

Proposed API

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
spec:
  kubeconfig:
    metadata:
      annotations:
        reflector.v1.k8s.emberstack.com/reflection-allowed: "true"

Anything else you would like to add?

Already have a PR in mind and will submit that shortly!

Label(s) to be applied

/kind feature
/area cluster

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[fabriziopandini]

In CAPI, we already have https://cluster-api.sigs.k8s.io/reference/api/metadata-propagation (+ corresponding proposal) for metadata propagation, so please look into this first. If this does not cover your use case, we can discuss options to expand this feature, but we should not introduce alternative paths for metadata propagation.

[jonathanrainer]

In CAPI, we already have https://cluster-api.sigs.k8s.io/reference/api/metadata-propagation (+ corresponding proposal) for metadata propagation, so please look into this first. If this does not cover your use case, we can discuss options to expand this feature, but we should not introduce alternative paths for metadata propagation.

Hi @fabriziopandini, this might be misunderstanding where the boundary of responsibility is for KubeConfigs, reading the documentation it’s not super clear whether it’s the responsibility of CAPI or the providers to generate the KubeConfigs that are used. This section of the docs https://cluster-api.sigs.k8s.io/developer/providers/contracts/control-plane.html?highlight=Kubeconfig#cluster-kubeconfig-management, implies it’s the provider, but I couldn’t be 100% sure. If it’s a core responsibility then I think expanding metadata propagation makes sense but I’m not sure what should propagate to the KubeConfig, it doesn’t seem to really fit the model because a consequent artefact unlike the other examples where it makes perfect sense.

On the other hand if it’s specifically a provider responsibility then I’m more than happy to close this out and re-open it with specific providers as it’s required.