From 817288cd5d1787d980ef04bbeb974c2f8edbc762 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 10:37:20 +0200 Subject: [PATCH 01/14] :seedling: keep Syself CCM for bare metal until upstream release --- .../03-creating-workload-cluster.md | 48 ++--- docs/caph/03-reference/08-annotations.md | 2 +- .../bases/hcloud-hetznerCluster-network.yaml | 2 - .../v1beta1/bases/hcloud-hetznerCluster.yaml | 2 - .../v1beta1/bases/hetzner-hetznerCluster.yaml | 2 - .../v1beta1/cluster-class.yaml | 3 - test/e2e/Makefile | 21 +- test/e2e/config/hetzner.yaml | 2 +- test/e2e/data/ccm/hcloud-ccm-hetzner.yaml | 183 ------------------ test/e2e/data/ccm/syself-ccm-hetzner.yaml | 135 +++++++++++++ 10 files changed, 168 insertions(+), 232 deletions(-) delete mode 100644 test/e2e/data/ccm/hcloud-ccm-hetzner.yaml create mode 100644 test/e2e/data/ccm/syself-ccm-hetzner.yaml diff --git a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md index 3b842e67e..9ff450e33 100644 --- a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md +++ b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md @@ -13,7 +13,7 @@ Secrets as of now are hardcoded given we are using a flavor which is essentially {% /callout %} -First you need to decide if you want to use the Syself CCM or the upstream HCloud CCM. +For bare metal, use the Syself CCM on this branch. The CCM is the "Cloud Controller" which runs in the workload-cluster. The most important tasks of the CCM are: @@ -21,8 +21,10 @@ The CCM is the "Cloud Controller" which runs in the workload-cluster. The most i machine (in mgt-cluster) is which Node (in wl-cluster). - Creates LoadBalancers -If you are unsure, use the HCloud CCM. In the long run we (Syself) want to switch from our fork to -the upstream CCM. +The upstream fix for stale Robot cache misses on bare-metal name lookups merged in +[hetznercloud/hcloud-cloud-controller-manager#1204](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204) +on April 9, 2026, but the latest upstream release is still `v1.30.1` from February 20, 2026. Until +an upstream release includes that fix, keep using the Syself CCM for bare metal. The CCM calls the Hetzner APIs. To authenticate, it reads the credentials from a secret. This secret has to be in the workload cluster, when the CCM runs in the workload cluster. CAPH creates the @@ -36,10 +38,10 @@ Important: CAPH and the CCM must both use the same ProviderID format for bare me - old: `hcloud://bm-NNNN` - new: `hrobot://NNNN` -The Syself CCM uses the old format by default. The HCloud CCM always uses the new format. +The Syself CCM uses the old format by default. The upstream HCloud CCM uses the new format. If you use the new format, set the annotation `capi.syself.com/use-hrobot-provider-id-for-baremetal` -to `"true"` on the `HetznerCluster`. Our default templates have this annotation set. +to `"true"` on the `HetznerCluster`. If CAPH and the CCM do not agree on the ProviderID format, then new nodes will not be able to join the cluster, because CAPI waits for the wrong ProviderID. @@ -108,7 +110,25 @@ This requires a secret containing access credentials to both Hetzner Robot and H {% /callout %} -If you want to use the HCloud CCM: +Use the Syself CCM: + +```shell +helm repo add syself https://charts.syself.com +helm repo update syself + +helm upgrade --install ccm syself/ccm-hetzner --version 2.0.6 \ + --namespace kube-system \ + --kubeconfig workload-kubeconfig +``` + +Be sure that the HetznerCluster does not have the annotation +`capi.syself.com/use-hrobot-provider-id-for-baremetal`. + +--- + +Once an upstream release includes PR +[#1204](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204), you can use the +upstream HCloud CCM for bare metal with the following values: ```shell helm repo add hcloud https://charts.hetzner.cloud @@ -133,22 +153,6 @@ Be sure that the HetznerCluster has the annotation If you use a different `hetznerSecretRef.name` or different keys, then adjust the chart values above accordingly. ---- - -If you want to use the Syself CCM (not recommended for new clusters): - -```shell -helm repo add syself https://charts.syself.com -helm repo update syself - -$ helm upgrade --install ccm syself/ccm-hetzner --version 2.0.6 \ - --namespace kube-system \ - --kubeconfig workload-kubeconfig -``` - -Be sure that the HetznerCluster does not have the annotation -`capi.syself.com/use-hrobot-provider-id-for-baremetal`. - ### Installing CNI For CNI, let's deploy cilium in the workload cluster that will facilitate the networking in the cluster. diff --git a/docs/caph/03-reference/08-annotations.md b/docs/caph/03-reference/08-annotations.md index e9e64c774..9e8302cfc 100644 --- a/docs/caph/03-reference/08-annotations.md +++ b/docs/caph/03-reference/08-annotations.md @@ -67,6 +67,6 @@ You can set [Kubernetes Annotations](https://kubernetes.io/docs/concepts/overvie | **Resource** | [HetznerCluster](/docs/caph/03-reference/02-hetzner-cluster.md) | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Description** | When set to `"true"`, then the ProviderID format `hrobot://NNNN` will be used. If not set or empty `hcloud://bm-NNNN` will be used. | +| **Description** | When set to `"true"`, CAPH uses the ProviderID format `hrobot://NNNN` for bare metal. Leave it unset to use `hcloud://bm-NNNN`, which matches the Syself CCM. Set it only when you run an upstream bare-metal-capable `hcloud-cloud-controller-manager` release. | | **Value** | `"true"` enables this feature. | | **Auto-Remove** | Disabled: The annotation remains on the resource. | diff --git a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml index 5ba015e67..10237e4b4 100644 --- a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml +++ b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml @@ -2,8 +2,6 @@ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerCluster metadata: name: "${CLUSTER_NAME}" - annotations: - capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: hcloudNetwork: enabled: true diff --git a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml index 1855e39f0..85999bae4 100644 --- a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml +++ b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml @@ -2,8 +2,6 @@ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerCluster metadata: name: "${CLUSTER_NAME}" - annotations: - capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: hcloudNetwork: enabled: false diff --git a/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml b/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml index 67b98d9a1..40b6014c2 100644 --- a/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml +++ b/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml @@ -2,8 +2,6 @@ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerCluster metadata: name: "${CLUSTER_NAME}" - annotations: - capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: hcloudNetwork: enabled: false diff --git a/templates/cluster-templates/v1beta1/cluster-class.yaml b/templates/cluster-templates/v1beta1/cluster-class.yaml index c3f69036b..3ba900599 100644 --- a/templates/cluster-templates/v1beta1/cluster-class.yaml +++ b/templates/cluster-templates/v1beta1/cluster-class.yaml @@ -347,9 +347,6 @@ metadata: name: quick-start-cluster spec: template: - metadata: - annotations: - capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: controlPlaneEndpoint: host: "" diff --git a/test/e2e/Makefile b/test/e2e/Makefile index 56415c68a..956800bfc 100644 --- a/test/e2e/Makefile +++ b/test/e2e/Makefile @@ -89,27 +89,16 @@ e2e-ccm-templates: --set networking.network.valueFrom.secretKeyRef.name=hetzner \ --set networking.network.valueFrom.secretKeyRef.key=network > $(REPO_ROOT)/test/e2e/data/ccm/hcloud-ccm-network.yaml - helm template ccm hetzner-cloud/hcloud-cloud-controller-manager --version 1.30.1 \ + helm template ccm syself/ccm-hetzner --version 2.0.6 \ --namespace kube-system \ - --set image.repository=ghcr.io/syself/hcloud-cloud-controller-manager \ - --set-string env.HCLOUD_DEBUG.value=true \ - --set env.HCLOUD_TOKEN.valueFrom.secretKeyRef.name=hetzner \ - --set env.HCLOUD_TOKEN.valueFrom.secretKeyRef.key=hcloud \ - --set env.ROBOT_USER.valueFrom.secretKeyRef.name=hetzner \ - --set env.ROBOT_USER.valueFrom.secretKeyRef.key=robot-user \ - --set env.ROBOT_PASSWORD.valueFrom.secretKeyRef.name=hetzner \ - --set env.ROBOT_PASSWORD.valueFrom.secretKeyRef.key=robot-password \ - --set-json 'additionalTolerations=[{"key":"node.cluster.x-k8s.io/uninitialized","operator":"Exists","effect":"NoSchedule"},{"key":"node.cilium.io/agent-not-ready","operator":"Exists","effect":"NoSchedule"}]' \ - --set args.route-reconciliation-period=null \ - --set args.feature-gates=null \ - --set robot.enabled=true \ - --set nameOverride=ccm-ccm-hetzner \ - --set networking.enabled=false > $(REPO_ROOT)/test/e2e/data/ccm/hcloud-ccm-hetzner.yaml + --set env.debug=true \ + --set-json 'tolerations=[{"key":"node.cluster.x-k8s.io/uninitialized","operator":"Exists","effect":"NoSchedule"},{"key":"node.cilium.io/agent-not-ready","operator":"Exists","effect":"NoSchedule"}]' \ + > $(REPO_ROOT)/test/e2e/data/ccm/syself-ccm-hetzner.yaml for file in \ $(REPO_ROOT)/test/e2e/data/ccm/hcloud-ccm.yaml \ $(REPO_ROOT)/test/e2e/data/ccm/hcloud-ccm-network.yaml \ - $(REPO_ROOT)/test/e2e/data/ccm/hcloud-ccm-hetzner.yaml; do \ + $(REPO_ROOT)/test/e2e/data/ccm/syself-ccm-hetzner.yaml; do \ tmp_file="$${file}.tmp"; \ { \ # Keep the generated-file marker split so boilerplate verification does not classify this Makefile as generated. \ diff --git a/test/e2e/config/hetzner.yaml b/test/e2e/config/hetzner.yaml index 4e7e44130..7e8bd1d76 100644 --- a/test/e2e/config/hetzner.yaml +++ b/test/e2e/config/hetzner.yaml @@ -158,7 +158,7 @@ variables: CILIUM: "./data/cni/cilium/values.yaml" CCM: "./data/ccm/hcloud-ccm.yaml" CCM_NETWORK: "./data/ccm/hcloud-ccm-network.yaml" - CCM_HETZNER: "./data/ccm/hcloud-ccm-hetzner.yaml" + CCM_HETZNER: "./data/ccm/syself-ccm-hetzner.yaml" # Conformance testing variables CONFORMANCE_WORKER_MACHINE_COUNT: 3 diff --git a/test/e2e/data/ccm/hcloud-ccm-hetzner.yaml b/test/e2e/data/ccm/hcloud-ccm-hetzner.yaml deleted file mode 100644 index 1a11d4b62..000000000 --- a/test/e2e/data/ccm/hcloud-ccm-hetzner.yaml +++ /dev/null @@ -1,183 +0,0 @@ -# Code generated by `make -C test/e2e e2e-ccm-templates`. DO NOT EDIT. ---- -# Source: hcloud-cloud-controller-manager/templates/serviceaccount.yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: ccm-ccm-hetzner - namespace: kube-system ---- -# Source: hcloud-cloud-controller-manager/templates/clusterrole.yml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: "system:ccm-ccm-hetzner" -rules: - - apiGroups: - - "" - resources: - - configmaps - verbs: - - get - - list - - watch - - apiGroups: - - "" - resources: - - events - verbs: - - create - - patch - - update - - apiGroups: - - "" - resources: - - nodes - verbs: - - "*" - - apiGroups: - - "" - resources: - - nodes/status - verbs: - - patch - - apiGroups: - - "" - resources: - - services - verbs: - - list - - watch - - apiGroups: - - "" - resources: - - services/status - verbs: - - patch - - update - - apiGroups: - - "" - resources: - - serviceaccounts - verbs: - - create - - apiGroups: - - "" - resources: - - persistentvolumes - verbs: - - get - - list - - update - - watch - - apiGroups: - - coordination.k8s.io - resources: - - leases - verbs: - - create - - get - - list - - watch - - update ---- -# Source: hcloud-cloud-controller-manager/templates/clusterrolebinding.yaml -kind: ClusterRoleBinding -apiVersion: rbac.authorization.k8s.io/v1 -metadata: - # The prefix ":restricted" originates from removing the cluster-admin role from HCCM. - # Renaming the ClusterRoleBinding makes the migration easier for users. - name: "system:ccm-ccm-hetzner:restricted" -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: "system:ccm-ccm-hetzner" -subjects: - - kind: ServiceAccount - name: ccm-ccm-hetzner - namespace: kube-system ---- -# Source: hcloud-cloud-controller-manager/templates/deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: ccm-ccm-hetzner - namespace: kube-system -spec: - replicas: 1 - revisionHistoryLimit: 2 - selector: - matchLabels: - app.kubernetes.io/instance: 'ccm' - app.kubernetes.io/name: 'ccm-ccm-hetzner' - template: - metadata: - labels: - app.kubernetes.io/instance: 'ccm' - app.kubernetes.io/name: 'ccm-ccm-hetzner' - spec: - serviceAccountName: ccm-ccm-hetzner - dnsPolicy: Default - tolerations: - # Allow HCCM itself to schedule on nodes that have not yet been initialized by HCCM. - - key: "node.cloudprovider.kubernetes.io/uninitialized" - value: "true" - effect: "NoSchedule" - - key: "CriticalAddonsOnly" - operator: "Exists" - - # Allow HCCM to schedule on control plane nodes. - - key: "node-role.kubernetes.io/master" - effect: NoSchedule - operator: Exists - - key: "node-role.kubernetes.io/control-plane" - effect: NoSchedule - operator: Exists - - - key: "node.kubernetes.io/not-ready" - effect: "NoExecute" - - - effect: NoSchedule - key: node.cluster.x-k8s.io/uninitialized - operator: Exists - - effect: NoSchedule - key: node.cilium.io/agent-not-ready - operator: Exists - containers: - - name: hcloud-cloud-controller-manager - args: - - "--allow-untagged-cloud" - - "--cloud-provider=hcloud" - - "--webhook-secure-port=0" - - "--leader-elect=false" - env: - - name: HCLOUD_DEBUG - value: "true" - - name: HCLOUD_TOKEN - valueFrom: - secretKeyRef: - key: hcloud - name: hetzner - - name: ROBOT_PASSWORD - valueFrom: - secretKeyRef: - key: robot-password - name: hetzner - optional: true - - name: ROBOT_USER - valueFrom: - secretKeyRef: - key: robot-user - name: hetzner - optional: true - - name: ROBOT_ENABLED - value: "true" - image: ghcr.io/syself/hcloud-cloud-controller-manager:v1.30.1 # x-releaser-pleaser-version - ports: - - name: metrics - containerPort: 8233 - resources: - requests: - cpu: 100m - memory: 50Mi - priorityClassName: "system-cluster-critical" diff --git a/test/e2e/data/ccm/syself-ccm-hetzner.yaml b/test/e2e/data/ccm/syself-ccm-hetzner.yaml new file mode 100644 index 000000000..fa44af0f4 --- /dev/null +++ b/test/e2e/data/ccm/syself-ccm-hetzner.yaml @@ -0,0 +1,135 @@ +# Code generated by `make -C test/e2e e2e-ccm-templates`. DO NOT EDIT. +--- +# Source: ccm-hetzner/templates/pdb.yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: ccm-ccm-hetzner + namespace: kube-system +spec: + minAvailable: 1 + maxUnavailable: + selector: + matchLabels: + app.kubernetes.io/name: ccm-hetzner + app.kubernetes.io/instance: ccm +--- +# Source: ccm-hetzner/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: ccm-ccm-hetzner + namespace: kube-system + labels: + helm.sh/chart: ccm-hetzner-2.0.6 + app: ccm + app.kubernetes.io/name: ccm-hetzner + app.kubernetes.io/instance: ccm + app.kubernetes.io/version: "v2.0.6" + app.kubernetes.io/managed-by: Helm +--- +# Source: ccm-hetzner/templates/serviceaccount.yaml +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ccm-ccm-hetzner + namespace: kube-system + labels: + helm.sh/chart: ccm-hetzner-2.0.6 + app: ccm + app.kubernetes.io/name: ccm-hetzner + app.kubernetes.io/instance: ccm + app.kubernetes.io/version: "v2.0.6" + app.kubernetes.io/managed-by: Helm +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +subjects: + - kind: ServiceAccount + name: ccm-ccm-hetzner + namespace: kube-system +--- +# Source: ccm-hetzner/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ccm-ccm-hetzner + namespace: kube-system + labels: + helm.sh/chart: ccm-hetzner-2.0.6 + app: ccm + app.kubernetes.io/name: ccm-hetzner + app.kubernetes.io/instance: ccm + app.kubernetes.io/version: "v2.0.6" + app.kubernetes.io/managed-by: Helm +spec: + replicas: 1 + revisionHistoryLimit: 5 + selector: + matchLabels: + app.kubernetes.io/name: ccm-hetzner + app.kubernetes.io/instance: ccm + template: + metadata: + labels: + app.kubernetes.io/name: ccm-hetzner + app.kubernetes.io/instance: ccm + spec: + dnsPolicy: Default + serviceAccountName: ccm-ccm-hetzner + securityContext: + {} + volumes: + - name: hetzner-secret + secret: + secretName: hetzner + tolerations: + # Introduced with CAPI v1.4, more info: https://cluster-api.sigs.k8s.io/developer/providers/bootstrap.html#taint-nodes-at-creation + - key: "node.cluster.x-k8s.io/uninitialized" + effect: "NoSchedule" + # Allow HCCM itself to schedule on nodes that have not yet been initialized by HCCM. + - key: "node.cloudprovider.kubernetes.io/uninitialized" + value: "true" + effect: "NoSchedule" + - key: "CriticalAddonsOnly" + operator: "Exists" + # cloud controller manages should be able to run on masters + - key: "node-role.kubernetes.io/master" + effect: NoSchedule + operator: Exists + - key: "node-role.kubernetes.io/control-plane" + effect: NoSchedule + operator: Exists + - key: "node.kubernetes.io/not-ready" + effect: "NoSchedule" + - effect: NoSchedule + key: node.cluster.x-k8s.io/uninitialized + operator: Exists + - effect: NoSchedule + key: node.cilium.io/agent-not-ready + operator: Exists + containers: + - name: ccm-hetzner + securityContext: + {} + image: "ghcr.io/syself/hetzner-cloud-controller-manager:v2.0.6" + imagePullPolicy: Always + command: + - "/bin/hetzner-cloud-controller-manager" + - "--cloud-provider=hcloud" + - "--leader-elect=true" + - "--allow-untagged-cloud" + resources: + requests: + cpu: 100m + memory: 50Mi + volumeMounts: + - mountPath: /etc/hetzner-secret + name: hetzner-secret + readOnly: true + env: + - name: HCLOUD_DEBUG + value: "true" + - name: HCLOUD_LOAD_BALANCERS_ENABLED + value: "true" From 6b312edc36d1116f0628b58fd37d20be3ebe04a7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 11:12:17 +0200 Subject: [PATCH 02/14] :seedling: restore bare-metal provider ID annotation defaults --- .../03-creating-workload-cluster.md | 20 +++++++++---------- docs/caph/03-reference/08-annotations.md | 2 +- .../bases/hcloud-hetznerCluster-network.yaml | 2 ++ .../v1beta1/bases/hcloud-hetznerCluster.yaml | 2 ++ .../v1beta1/bases/hetzner-hetznerCluster.yaml | 2 ++ .../v1beta1/cluster-class.yaml | 3 +++ 6 files changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md index 9ff450e33..4a0c6b320 100644 --- a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md +++ b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md @@ -13,7 +13,11 @@ Secrets as of now are hardcoded given we are using a flavor which is essentially {% /callout %} -For bare metal, use the Syself CCM on this branch. +For bare metal, use the Syself CCM until an upstream release includes [PR fix robot name lookup +after stale cache miss](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204), +which merged on April 9, 2026. + +In the long run we (Syself) want to switch from our fork to the upstream CCM. The CCM is the "Cloud Controller" which runs in the workload-cluster. The most important tasks of the CCM are: @@ -21,11 +25,6 @@ The CCM is the "Cloud Controller" which runs in the workload-cluster. The most i machine (in mgt-cluster) is which Node (in wl-cluster). - Creates LoadBalancers -The upstream fix for stale Robot cache misses on bare-metal name lookups merged in -[hetznercloud/hcloud-cloud-controller-manager#1204](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204) -on April 9, 2026, but the latest upstream release is still `v1.30.1` from February 20, 2026. Until -an upstream release includes that fix, keep using the Syself CCM for bare metal. - The CCM calls the Hetzner APIs. To authenticate, it reads the credentials from a secret. This secret has to be in the workload cluster, when the CCM runs in the workload cluster. CAPH creates the secret and syncs the credentials specified in the management cluster to the workload cluster. In our @@ -38,10 +37,11 @@ Important: CAPH and the CCM must both use the same ProviderID format for bare me - old: `hcloud://bm-NNNN` - new: `hrobot://NNNN` -The Syself CCM uses the old format by default. The upstream HCloud CCM uses the new format. +The upstream HCloud CCM uses the new format. Our default templates keep the annotation enabled and +therefore use the new format for bare metal. If you use the new format, set the annotation `capi.syself.com/use-hrobot-provider-id-for-baremetal` -to `"true"` on the `HetznerCluster`. +to `"true"` on the `HetznerCluster`. Our default templates have this annotation set. If CAPH and the CCM do not agree on the ProviderID format, then new nodes will not be able to join the cluster, because CAPI waits for the wrong ProviderID. @@ -121,8 +121,8 @@ helm upgrade --install ccm syself/ccm-hetzner --version 2.0.6 \ --kubeconfig workload-kubeconfig ``` -Be sure that the HetznerCluster does not have the annotation -`capi.syself.com/use-hrobot-provider-id-for-baremetal`. +Be sure that the HetznerCluster has the annotation +`capi.syself.com/use-hrobot-provider-id-for-baremetal: "true"`. --- diff --git a/docs/caph/03-reference/08-annotations.md b/docs/caph/03-reference/08-annotations.md index 9e8302cfc..e9e64c774 100644 --- a/docs/caph/03-reference/08-annotations.md +++ b/docs/caph/03-reference/08-annotations.md @@ -67,6 +67,6 @@ You can set [Kubernetes Annotations](https://kubernetes.io/docs/concepts/overvie | **Resource** | [HetznerCluster](/docs/caph/03-reference/02-hetzner-cluster.md) | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Description** | When set to `"true"`, CAPH uses the ProviderID format `hrobot://NNNN` for bare metal. Leave it unset to use `hcloud://bm-NNNN`, which matches the Syself CCM. Set it only when you run an upstream bare-metal-capable `hcloud-cloud-controller-manager` release. | +| **Description** | When set to `"true"`, then the ProviderID format `hrobot://NNNN` will be used. If not set or empty `hcloud://bm-NNNN` will be used. | | **Value** | `"true"` enables this feature. | | **Auto-Remove** | Disabled: The annotation remains on the resource. | diff --git a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml index 10237e4b4..5ba015e67 100644 --- a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml +++ b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster-network.yaml @@ -2,6 +2,8 @@ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerCluster metadata: name: "${CLUSTER_NAME}" + annotations: + capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: hcloudNetwork: enabled: true diff --git a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml index 85999bae4..1855e39f0 100644 --- a/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml +++ b/templates/cluster-templates/v1beta1/bases/hcloud-hetznerCluster.yaml @@ -2,6 +2,8 @@ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerCluster metadata: name: "${CLUSTER_NAME}" + annotations: + capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: hcloudNetwork: enabled: false diff --git a/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml b/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml index 40b6014c2..67b98d9a1 100644 --- a/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml +++ b/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml @@ -2,6 +2,8 @@ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerCluster metadata: name: "${CLUSTER_NAME}" + annotations: + capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: hcloudNetwork: enabled: false diff --git a/templates/cluster-templates/v1beta1/cluster-class.yaml b/templates/cluster-templates/v1beta1/cluster-class.yaml index 3ba900599..c3f69036b 100644 --- a/templates/cluster-templates/v1beta1/cluster-class.yaml +++ b/templates/cluster-templates/v1beta1/cluster-class.yaml @@ -347,6 +347,9 @@ metadata: name: quick-start-cluster spec: template: + metadata: + annotations: + capi.syself.com/use-hrobot-provider-id-for-baremetal: "true" spec: controlPlaneEndpoint: host: "" From 62ddcec0cff5d4242e264cb9660033a1a2d1a650 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 12:17:19 +0200 Subject: [PATCH 03/14] :seedling: align Syself CCM e2e values with prior setup --- test/e2e/Makefile | 2 ++ test/e2e/data/ccm/syself-ccm-hetzner.yaml | 14 -------------- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/test/e2e/Makefile b/test/e2e/Makefile index 956800bfc..1c09bfe66 100644 --- a/test/e2e/Makefile +++ b/test/e2e/Makefile @@ -92,6 +92,8 @@ e2e-ccm-templates: helm template ccm syself/ccm-hetzner --version 2.0.6 \ --namespace kube-system \ --set env.debug=true \ + --set pdb.enabled=false \ + --set privateNetwork.enabled=false \ --set-json 'tolerations=[{"key":"node.cluster.x-k8s.io/uninitialized","operator":"Exists","effect":"NoSchedule"},{"key":"node.cilium.io/agent-not-ready","operator":"Exists","effect":"NoSchedule"}]' \ > $(REPO_ROOT)/test/e2e/data/ccm/syself-ccm-hetzner.yaml diff --git a/test/e2e/data/ccm/syself-ccm-hetzner.yaml b/test/e2e/data/ccm/syself-ccm-hetzner.yaml index fa44af0f4..07a55b1d3 100644 --- a/test/e2e/data/ccm/syself-ccm-hetzner.yaml +++ b/test/e2e/data/ccm/syself-ccm-hetzner.yaml @@ -1,19 +1,5 @@ # Code generated by `make -C test/e2e e2e-ccm-templates`. DO NOT EDIT. --- -# Source: ccm-hetzner/templates/pdb.yaml -apiVersion: policy/v1 -kind: PodDisruptionBudget -metadata: - name: ccm-ccm-hetzner - namespace: kube-system -spec: - minAvailable: 1 - maxUnavailable: - selector: - matchLabels: - app.kubernetes.io/name: ccm-hetzner - app.kubernetes.io/instance: ccm ---- # Source: ccm-hetzner/templates/serviceaccount.yaml apiVersion: v1 kind: ServiceAccount From dc9ae4ef1c9e58f25716a5e24f28f7e7389549a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 14:18:56 +0200 Subject: [PATCH 04/14] :seedling: simplify bare-metal CCM guidance --- .../03-creating-workload-cluster.md | 38 ++----------------- 1 file changed, 4 insertions(+), 34 deletions(-) diff --git a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md index 4a0c6b320..0f054b042 100644 --- a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md +++ b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md @@ -14,8 +14,7 @@ Secrets as of now are hardcoded given we are using a flavor which is essentially {% /callout %} For bare metal, use the Syself CCM until an upstream release includes [PR fix robot name lookup -after stale cache miss](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204), -which merged on April 9, 2026. +after stale cache miss](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204). In the long run we (Syself) want to switch from our fork to the upstream CCM. @@ -37,8 +36,7 @@ Important: CAPH and the CCM must both use the same ProviderID format for bare me - old: `hcloud://bm-NNNN` - new: `hrobot://NNNN` -The upstream HCloud CCM uses the new format. Our default templates keep the annotation enabled and -therefore use the new format for bare metal. +The upstream HCloud CCM uses the new format. If you use the new format, set the annotation `capi.syself.com/use-hrobot-provider-id-for-baremetal` to `"true"` on the `HetznerCluster`. Our default templates have this annotation set. @@ -110,7 +108,8 @@ This requires a secret containing access credentials to both Hetzner Robot and H {% /callout %} -Use the Syself CCM: +For bare metal, use the Syself CCM until an upstream release includes [PR fix robot name lookup +after stale cache miss](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204). ```shell helm repo add syself https://charts.syself.com @@ -124,35 +123,6 @@ helm upgrade --install ccm syself/ccm-hetzner --version 2.0.6 \ Be sure that the HetznerCluster has the annotation `capi.syself.com/use-hrobot-provider-id-for-baremetal: "true"`. ---- - -Once an upstream release includes PR -[#1204](https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/1204), you can use the -upstream HCloud CCM for bare metal with the following values: - -```shell -helm repo add hcloud https://charts.hetzner.cloud -helm repo update hcloud - -helm upgrade --install ccm hcloud/hcloud-cloud-controller-manager \ - --namespace kube-system \ - --set env.HCLOUD_TOKEN.valueFrom.secretKeyRef.name=hetzner \ - --set env.HCLOUD_TOKEN.valueFrom.secretKeyRef.key=hcloud \ - --set env.ROBOT_USER.valueFrom.secretKeyRef.name=hetzner \ - --set env.ROBOT_USER.valueFrom.secretKeyRef.key=robot-user \ - --set env.ROBOT_PASSWORD.valueFrom.secretKeyRef.name=hetzner \ - --set env.ROBOT_PASSWORD.valueFrom.secretKeyRef.key=robot-password \ - --set-json 'additionalTolerations=[{"key":"node.cluster.x-k8s.io/uninitialized","operator":"Exists","effect":"NoSchedule"},{"key":"node.cilium.io/agent-not-ready","operator":"Exists","effect":"NoSchedule"}]' \ - --set robot.enabled=true \ - --kubeconfig workload-kubeconfig -``` - -Be sure that the HetznerCluster has the annotation -`capi.syself.com/use-hrobot-provider-id-for-baremetal: "true"`. - -If you use a different `hetznerSecretRef.name` or different keys, then adjust the chart values -above accordingly. - ### Installing CNI For CNI, let's deploy cilium in the workload cluster that will facilitate the networking in the cluster. From b2e1f861aa767b1e5334d88d2a210f07410a423f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 15:45:54 +0200 Subject: [PATCH 05/14] Disable bare-metal host 1763731 --- .../v1beta1/bases/hetznerbaremetalhosts-raid.yaml | 3 ++- .../v1beta1/bases/hetznerbaremetalhosts.yaml | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml index 49e7d352a..6ffe0981f 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml @@ -12,6 +12,7 @@ spec: maintenanceMode: false description: Test Machine 2 --- +# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -23,5 +24,5 @@ spec: wwn: - "0x500a07511756c36d" - "0x500a075117101996" - maintenanceMode: false + maintenanceMode: true description: Test Machine 6 diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml index ef3d12fe7..680989f32 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml @@ -71,7 +71,7 @@ spec: maintenanceMode: true #### < --- set to false, when ok. description: Test Machine 1751550 --- -# 2026-03-12: Support fixed hardware issue. Now working. +# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -83,5 +83,5 @@ spec: serverID: 1763731 # ip: 144.76.101.50 rootDeviceHints: wwn: "0x500a07511756c36d" - maintenanceMode: false + maintenanceMode: true description: Test Machine 1751550 From eb40bdf772aace5dc8687a9d7a09e461afb7dcfd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 16:31:33 +0200 Subject: [PATCH 06/14] Fix Syself CCM workload secret compatibility --- api/v1beta1/types.go | 8 ++--- controllers/hetznercluster_controller.go | 31 ++++++++++++------- controllers/hetznercluster_controller_test.go | 20 +++++++----- test/e2e/Makefile | 1 + test/e2e/data/ccm/syself-ccm-hetzner.yaml | 2 +- 5 files changed, 39 insertions(+), 23 deletions(-) diff --git a/api/v1beta1/types.go b/api/v1beta1/types.go index 593a985c3..1a0b950c8 100644 --- a/api/v1beta1/types.go +++ b/api/v1beta1/types.go @@ -130,10 +130,10 @@ type HetznerSecretKeyRef struct { // The controller reads the token from the management-cluster secret using this key and writes // it to workload-cluster secrets using the same key. Upstream hcloud-ccm expects the key // "token", while the legacy Syself ccm fork uses "hcloud". The configured workload-cluster - // secret remains untouched. CAPH only adds the compatibility key "token" inside the - // workload-cluster secret named "hcloud", and only if this field is not already "token". If - // the configured secret name is already "hcloud", the same token value is stored under both - // keys in that single secret. We recommend to use "token". + // secret remains untouched. CAPH only adds the compatibility keys "token" and "hcloud" + // inside the workload-cluster secret named "hcloud", unless one of them already matches this + // field. If the configured secret name is already "hcloud", the same token value is stored + // under all required keys in that single secret. We recommend to use "token". // // +optional +kubebuilder:default=hcloud-token HCloudToken string `json:"hcloudToken"` diff --git a/controllers/hetznercluster_controller.go b/controllers/hetznercluster_controller.go index 59e9d202a..7bf9aa0ea 100644 --- a/controllers/hetznercluster_controller.go +++ b/controllers/hetznercluster_controller.go @@ -548,21 +548,30 @@ func workloadClusterSecretNames(secretName string) []string { } // workloadClusterCompatibilityKeys returns the configured key for every secret. -// When reconciling the secret named "hcloud", it also returns the compatibility -// key unless it matches the configured key. -func workloadClusterCompatibilityKeys(secretName, configuredKey, compatibilityKey string) []string { +// When reconciling the secret named "hcloud", it also returns any compatibility +// keys unless they already match the configured key. +func workloadClusterCompatibilityKeys(secretName, configuredKey string, compatibilityKeys ...string) []string { + keys := []string{configuredKey} if secretName != "hcloud" { - // Only the secret named "hcloud" gets the compatibility key. - return []string{configuredKey} + // Only the secret named "hcloud" gets compatibility keys. + return keys } - if configuredKey == compatibilityKey { - // Avoid duplicating the key when it already matches the compatibility key. - return []string{configuredKey} + for _, compatibilityKey := range compatibilityKeys { + duplicate := false + for _, key := range keys { + if key == compatibilityKey { + duplicate = true + break + } + } + if duplicate { + continue + } + keys = append(keys, compatibilityKey) } - // The compatibility secret exposes both the configured key and the compatibility key. - return []string{configuredKey, compatibilityKey} + return keys } func reconcileAllWorkloadClusterSecrets(ctx context.Context, clusterScope *scope.ClusterScope, wlClient client.Client) error { @@ -606,7 +615,7 @@ func reconcileOneWorkloadClusterSecret(ctx context.Context, clusterScope *scope. wlSecret.Data = make(map[string][]byte) } - for _, key := range workloadClusterCompatibilityKeys(name, clusterScope.HetznerCluster.Spec.HetznerSecret.Key.HCloudToken, "token") { + for _, key := range workloadClusterCompatibilityKeys(name, clusterScope.HetznerCluster.Spec.HetznerSecret.Key.HCloudToken, "token", "hcloud") { wlSecret.Data[key] = hcloudToken } diff --git a/controllers/hetznercluster_controller_test.go b/controllers/hetznercluster_controller_test.go index ebcf52674..403731ff2 100644 --- a/controllers/hetznercluster_controller_test.go +++ b/controllers/hetznercluster_controller_test.go @@ -287,8 +287,8 @@ func TestWorkloadClusterSecretNames(t *testing.T) { } } -// TestWorkloadClusterHCloudTokenKeys verifies when CAPH adds the upstream -// compatibility key alongside the configured management-cluster key. +// TestWorkloadClusterHCloudTokenKeys verifies when CAPH adds compatibility +// keys alongside the configured management-cluster key. func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { testCases := []struct { name string @@ -306,7 +306,7 @@ func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { name: "hcloud secret adds token compatibility key", secretName: "hcloud", configuredKey: "custom-token", - want: []string{"custom-token", "token"}, + want: []string{"custom-token", "token", "hcloud"}, }, { name: "non-hcloud secrets keep hcloud configured key without duplication", @@ -315,10 +315,16 @@ func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { want: []string{"hcloud"}, }, { - name: "existing token key is not duplicated", + name: "existing token key still adds legacy hcloud compatibility key", secretName: "hcloud", configuredKey: "token", - want: []string{"token"}, + want: []string{"token", "hcloud"}, + }, + { + name: "existing hcloud key still adds upstream token compatibility key", + secretName: "hcloud", + configuredKey: "hcloud", + want: []string{"hcloud", "token"}, }, { name: "other secret names only keep configured key", @@ -330,7 +336,7 @@ func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { - got := workloadClusterCompatibilityKeys(tc.secretName, tc.configuredKey, "token") + got := workloadClusterCompatibilityKeys(tc.secretName, tc.configuredKey, "token", "hcloud") require.Equal(t, tc.want, got) }) } @@ -469,7 +475,7 @@ func TestReconcileOneWorkloadClusterSecretHCloud(t *testing.T) { require.NoError(t, wlClient.Get(ctx, client.ObjectKey{Namespace: metav1.NamespaceSystem, Name: "hcloud"}, secret)) require.Equal(t, "my-token", string(secret.Data["custom-token"])) require.Equal(t, "my-token", string(secret.Data["token"])) - require.NotContains(t, secret.Data, "hcloud") + require.Equal(t, "my-token", string(secret.Data["hcloud"])) require.Equal(t, "my-user", string(secret.Data["custom-robot-user"])) require.Equal(t, "my-password", string(secret.Data["custom-robot-password"])) require.Equal(t, "my-user", string(secret.Data["robot-user"])) diff --git a/test/e2e/Makefile b/test/e2e/Makefile index 914a33f9f..9b9a70e30 100644 --- a/test/e2e/Makefile +++ b/test/e2e/Makefile @@ -94,6 +94,7 @@ e2e-ccm-templates: --set env.debug=true \ --set pdb.enabled=false \ --set privateNetwork.enabled=false \ + --set secret.name=hcloud \ --set-json 'tolerations=[{"key":"node.cluster.x-k8s.io/uninitialized","operator":"Exists","effect":"NoSchedule"},{"key":"node.cilium.io/agent-not-ready","operator":"Exists","effect":"NoSchedule"}]' \ > $(REPO_ROOT)/test/e2e/data/ccm/syself-ccm-hetzner.yaml diff --git a/test/e2e/data/ccm/syself-ccm-hetzner.yaml b/test/e2e/data/ccm/syself-ccm-hetzner.yaml index 07a55b1d3..684ac29a7 100644 --- a/test/e2e/data/ccm/syself-ccm-hetzner.yaml +++ b/test/e2e/data/ccm/syself-ccm-hetzner.yaml @@ -69,7 +69,7 @@ spec: volumes: - name: hetzner-secret secret: - secretName: hetzner + secretName: hcloud tolerations: # Introduced with CAPI v1.4, more info: https://cluster-api.sigs.k8s.io/developer/providers/bootstrap.html#taint-nodes-at-creation - key: "node.cluster.x-k8s.io/uninitialized" From ee78c9237cb306f88b9fac352b20810ea77cea31 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 16:38:32 +0200 Subject: [PATCH 07/14] Regenerate CRDs for secret compatibility docs --- .../infrastructure.cluster.x-k8s.io_hetznerclusters.yaml | 8 ++++---- ...tructure.cluster.x-k8s.io_hetznerclustertemplates.yaml | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml index 05026e419..aa010ce55 100644 --- a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml +++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml @@ -251,10 +251,10 @@ spec: The controller reads the token from the management-cluster secret using this key and writes it to workload-cluster secrets using the same key. Upstream hcloud-ccm expects the key "token", while the legacy Syself ccm fork uses "hcloud". The configured workload-cluster - secret remains untouched. CAPH only adds the compatibility key "token" inside the - workload-cluster secret named "hcloud", and only if this field is not already "token". If - the configured secret name is already "hcloud", the same token value is stored under both - keys in that single secret. We recommend to use "token". + secret remains untouched. CAPH only adds the compatibility keys "token" and "hcloud" + inside the workload-cluster secret named "hcloud", unless one of them already matches this + field. If the configured secret name is already "hcloud", the same token value is stored + under all required keys in that single secret. We recommend to use "token". type: string hetznerRobotPassword: default: hetzner-robot-password diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml index 9f85022bd..77d3d79ec 100644 --- a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml +++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml @@ -280,10 +280,10 @@ spec: The controller reads the token from the management-cluster secret using this key and writes it to workload-cluster secrets using the same key. Upstream hcloud-ccm expects the key "token", while the legacy Syself ccm fork uses "hcloud". The configured workload-cluster - secret remains untouched. CAPH only adds the compatibility key "token" inside the - workload-cluster secret named "hcloud", and only if this field is not already "token". If - the configured secret name is already "hcloud", the same token value is stored under both - keys in that single secret. We recommend to use "token". + secret remains untouched. CAPH only adds the compatibility keys "token" and "hcloud" + inside the workload-cluster secret named "hcloud", unless one of them already matches this + field. If the configured secret name is already "hcloud", the same token value is stored + under all required keys in that single secret. We recommend to use "token". type: string hetznerRobotPassword: default: hetzner-robot-password From 57bab59f4e521851e24b0805a9a3c03b2319e52f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 16:45:13 +0200 Subject: [PATCH 08/14] Provide bare-metal Syself CCM secret values --- api/v1beta1/types.go | 8 ++--- ...ture.cluster.x-k8s.io_hetznerclusters.yaml | 8 ++--- ...ster.x-k8s.io_hetznerclustertemplates.yaml | 8 ++--- controllers/hetznercluster_controller.go | 31 +++++++------------ controllers/hetznercluster_controller_test.go | 20 +++++------- .../v1beta1/bases/hetzner-hetznerCluster.yaml | 2 +- .../v1beta1/bases/hetzner-secret.yaml | 2 ++ 7 files changed, 33 insertions(+), 46 deletions(-) diff --git a/api/v1beta1/types.go b/api/v1beta1/types.go index 1a0b950c8..593a985c3 100644 --- a/api/v1beta1/types.go +++ b/api/v1beta1/types.go @@ -130,10 +130,10 @@ type HetznerSecretKeyRef struct { // The controller reads the token from the management-cluster secret using this key and writes // it to workload-cluster secrets using the same key. Upstream hcloud-ccm expects the key // "token", while the legacy Syself ccm fork uses "hcloud". The configured workload-cluster - // secret remains untouched. CAPH only adds the compatibility keys "token" and "hcloud" - // inside the workload-cluster secret named "hcloud", unless one of them already matches this - // field. If the configured secret name is already "hcloud", the same token value is stored - // under all required keys in that single secret. We recommend to use "token". + // secret remains untouched. CAPH only adds the compatibility key "token" inside the + // workload-cluster secret named "hcloud", and only if this field is not already "token". If + // the configured secret name is already "hcloud", the same token value is stored under both + // keys in that single secret. We recommend to use "token". // // +optional +kubebuilder:default=hcloud-token HCloudToken string `json:"hcloudToken"` diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml index aa010ce55..05026e419 100644 --- a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml +++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclusters.yaml @@ -251,10 +251,10 @@ spec: The controller reads the token from the management-cluster secret using this key and writes it to workload-cluster secrets using the same key. Upstream hcloud-ccm expects the key "token", while the legacy Syself ccm fork uses "hcloud". The configured workload-cluster - secret remains untouched. CAPH only adds the compatibility keys "token" and "hcloud" - inside the workload-cluster secret named "hcloud", unless one of them already matches this - field. If the configured secret name is already "hcloud", the same token value is stored - under all required keys in that single secret. We recommend to use "token". + secret remains untouched. CAPH only adds the compatibility key "token" inside the + workload-cluster secret named "hcloud", and only if this field is not already "token". If + the configured secret name is already "hcloud", the same token value is stored under both + keys in that single secret. We recommend to use "token". type: string hetznerRobotPassword: default: hetzner-robot-password diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml index 77d3d79ec..9f85022bd 100644 --- a/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml +++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_hetznerclustertemplates.yaml @@ -280,10 +280,10 @@ spec: The controller reads the token from the management-cluster secret using this key and writes it to workload-cluster secrets using the same key. Upstream hcloud-ccm expects the key "token", while the legacy Syself ccm fork uses "hcloud". The configured workload-cluster - secret remains untouched. CAPH only adds the compatibility keys "token" and "hcloud" - inside the workload-cluster secret named "hcloud", unless one of them already matches this - field. If the configured secret name is already "hcloud", the same token value is stored - under all required keys in that single secret. We recommend to use "token". + secret remains untouched. CAPH only adds the compatibility key "token" inside the + workload-cluster secret named "hcloud", and only if this field is not already "token". If + the configured secret name is already "hcloud", the same token value is stored under both + keys in that single secret. We recommend to use "token". type: string hetznerRobotPassword: default: hetzner-robot-password diff --git a/controllers/hetznercluster_controller.go b/controllers/hetznercluster_controller.go index 7bf9aa0ea..59e9d202a 100644 --- a/controllers/hetznercluster_controller.go +++ b/controllers/hetznercluster_controller.go @@ -548,30 +548,21 @@ func workloadClusterSecretNames(secretName string) []string { } // workloadClusterCompatibilityKeys returns the configured key for every secret. -// When reconciling the secret named "hcloud", it also returns any compatibility -// keys unless they already match the configured key. -func workloadClusterCompatibilityKeys(secretName, configuredKey string, compatibilityKeys ...string) []string { - keys := []string{configuredKey} +// When reconciling the secret named "hcloud", it also returns the compatibility +// key unless it matches the configured key. +func workloadClusterCompatibilityKeys(secretName, configuredKey, compatibilityKey string) []string { if secretName != "hcloud" { - // Only the secret named "hcloud" gets compatibility keys. - return keys + // Only the secret named "hcloud" gets the compatibility key. + return []string{configuredKey} } - for _, compatibilityKey := range compatibilityKeys { - duplicate := false - for _, key := range keys { - if key == compatibilityKey { - duplicate = true - break - } - } - if duplicate { - continue - } - keys = append(keys, compatibilityKey) + if configuredKey == compatibilityKey { + // Avoid duplicating the key when it already matches the compatibility key. + return []string{configuredKey} } - return keys + // The compatibility secret exposes both the configured key and the compatibility key. + return []string{configuredKey, compatibilityKey} } func reconcileAllWorkloadClusterSecrets(ctx context.Context, clusterScope *scope.ClusterScope, wlClient client.Client) error { @@ -615,7 +606,7 @@ func reconcileOneWorkloadClusterSecret(ctx context.Context, clusterScope *scope. wlSecret.Data = make(map[string][]byte) } - for _, key := range workloadClusterCompatibilityKeys(name, clusterScope.HetznerCluster.Spec.HetznerSecret.Key.HCloudToken, "token", "hcloud") { + for _, key := range workloadClusterCompatibilityKeys(name, clusterScope.HetznerCluster.Spec.HetznerSecret.Key.HCloudToken, "token") { wlSecret.Data[key] = hcloudToken } diff --git a/controllers/hetznercluster_controller_test.go b/controllers/hetznercluster_controller_test.go index 403731ff2..ebcf52674 100644 --- a/controllers/hetznercluster_controller_test.go +++ b/controllers/hetznercluster_controller_test.go @@ -287,8 +287,8 @@ func TestWorkloadClusterSecretNames(t *testing.T) { } } -// TestWorkloadClusterHCloudTokenKeys verifies when CAPH adds compatibility -// keys alongside the configured management-cluster key. +// TestWorkloadClusterHCloudTokenKeys verifies when CAPH adds the upstream +// compatibility key alongside the configured management-cluster key. func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { testCases := []struct { name string @@ -306,7 +306,7 @@ func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { name: "hcloud secret adds token compatibility key", secretName: "hcloud", configuredKey: "custom-token", - want: []string{"custom-token", "token", "hcloud"}, + want: []string{"custom-token", "token"}, }, { name: "non-hcloud secrets keep hcloud configured key without duplication", @@ -315,16 +315,10 @@ func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { want: []string{"hcloud"}, }, { - name: "existing token key still adds legacy hcloud compatibility key", + name: "existing token key is not duplicated", secretName: "hcloud", configuredKey: "token", - want: []string{"token", "hcloud"}, - }, - { - name: "existing hcloud key still adds upstream token compatibility key", - secretName: "hcloud", - configuredKey: "hcloud", - want: []string{"hcloud", "token"}, + want: []string{"token"}, }, { name: "other secret names only keep configured key", @@ -336,7 +330,7 @@ func TestWorkloadClusterHCloudTokenKeys(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { - got := workloadClusterCompatibilityKeys(tc.secretName, tc.configuredKey, "token", "hcloud") + got := workloadClusterCompatibilityKeys(tc.secretName, tc.configuredKey, "token") require.Equal(t, tc.want, got) }) } @@ -475,7 +469,7 @@ func TestReconcileOneWorkloadClusterSecretHCloud(t *testing.T) { require.NoError(t, wlClient.Get(ctx, client.ObjectKey{Namespace: metav1.NamespaceSystem, Name: "hcloud"}, secret)) require.Equal(t, "my-token", string(secret.Data["custom-token"])) require.Equal(t, "my-token", string(secret.Data["token"])) - require.Equal(t, "my-token", string(secret.Data["hcloud"])) + require.NotContains(t, secret.Data, "hcloud") require.Equal(t, "my-user", string(secret.Data["custom-robot-user"])) require.Equal(t, "my-password", string(secret.Data["custom-robot-password"])) require.Equal(t, "my-user", string(secret.Data["robot-user"])) diff --git a/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml b/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml index 6039ec052..3dae7e035 100644 --- a/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml +++ b/templates/cluster-templates/v1beta1/bases/hetzner-hetznerCluster.yaml @@ -26,6 +26,6 @@ spec: hetznerSecretRef: name: hcloud key: - hcloudToken: token + hcloudToken: hcloud hetznerRobotUser: robot-user hetznerRobotPassword: robot-password diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetzner-secret.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetzner-secret.yaml index 822761740..46ff2bd76 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetzner-secret.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetzner-secret.yaml @@ -1,5 +1,7 @@ apiVersion: v1 data: + hcloud: | + hcloud_secret_placeholder token: | hcloud_secret_placeholder robot-user: | From a580b0553c9d4af45f27b96d756b4784d1eaf0a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 16:56:30 +0200 Subject: [PATCH 09/14] Fix generated-file verification tooling --- test/e2e/Makefile | 1 + test/e2e/cluster-templates-v1beta1.sh | 31 +++++++++++++++------------ 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/test/e2e/Makefile b/test/e2e/Makefile index 9b9a70e30..d53bd0c7a 100644 --- a/test/e2e/Makefile +++ b/test/e2e/Makefile @@ -68,6 +68,7 @@ e2e-cilium-templates: e2e-ccm-templates: helm repo add hetzner-cloud https://charts.hetzner.cloud + helm repo add syself https://charts.syself.com helm template ccm hetzner-cloud/hcloud-cloud-controller-manager --version 1.30.1 \ --namespace kube-system \ diff --git a/test/e2e/cluster-templates-v1beta1.sh b/test/e2e/cluster-templates-v1beta1.sh index 58172dad0..3ec90c4c5 100755 --- a/test/e2e/cluster-templates-v1beta1.sh +++ b/test/e2e/cluster-templates-v1beta1.sh @@ -18,8 +18,11 @@ trap 'echo -e "\nWarning: A command has failed. Exiting the script. Line was ($0:$LINENO): $(sed -n "${LINENO}p" "$0")"; exit 3' ERR set -Eeuo pipefail -if ! command -v kustomize | grep hack/tools/bin; then - echo "fix $PATH: kustomize should be from hack/tools/bin" +REPO_ROOT=$(git rev-parse --show-toplevel) +KUSTOMIZE_BIN="$REPO_ROOT/hack/tools/bin/kustomize" + +if [[ ! -x "$KUSTOMIZE_BIN" ]]; then + echo "fix $KUSTOMIZE_BIN: missing executable kustomize in hack/tools/bin" exit 1 fi @@ -48,29 +51,29 @@ printf '%s' "$HETZNER_SSH_PRIV" | base64 --decode >tmp_ssh_priv kubectl create secret generic robot-ssh --from-literal=sshkey-name="$SSH_KEY_NAME" --from-file=ssh-privatekey=tmp_ssh_priv --from-file=ssh-publickey=tmp_ssh_pub --dry-run=client -o yaml >data/infrastructure-hetzner/v1beta1/cluster-template-hetzner-secret.yaml rm tmp_ssh_pub tmp_ssh_priv -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-network --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-network.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-network --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-network.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal.yaml -kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup.yaml +"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup.yaml sed -i "s/robot-user_secret_placeholder/$(echo -n "$HETZNER_ROBOT_USER" | base64 -w0)/" "$HETZNER_TEMPLATES"/v1beta1/*.yaml sed -i "s/robot-password_secret_placeholder/$(echo -n "$HETZNER_ROBOT_PASSWORD" | base64 -w0)/" "$HETZNER_TEMPLATES"/v1beta1/*.yaml From ee136749e99d22932f5eea9b424523f45b1e58de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 17:54:50 +0200 Subject: [PATCH 10/14] Keep kustomize path fix in verification only --- hack/verify-generated-files.sh | 1 + test/e2e/cluster-templates-v1beta1.sh | 31 ++++++++++++--------------- 2 files changed, 15 insertions(+), 17 deletions(-) diff --git a/hack/verify-generated-files.sh b/hack/verify-generated-files.sh index f530d3b1f..29d511382 100755 --- a/hack/verify-generated-files.sh +++ b/hack/verify-generated-files.sh @@ -27,6 +27,7 @@ if ! git diff --quiet || ! git diff --cached --quiet || [[ -n "$(git ls-files -- fi ( + export PATH="$(git rev-parse --show-toplevel)/hack/tools/bin:$PATH" make kubectl cd test/e2e HCLOUD_TOKEN=dummy_hcloud_token diff --git a/test/e2e/cluster-templates-v1beta1.sh b/test/e2e/cluster-templates-v1beta1.sh index 3ec90c4c5..58172dad0 100755 --- a/test/e2e/cluster-templates-v1beta1.sh +++ b/test/e2e/cluster-templates-v1beta1.sh @@ -18,11 +18,8 @@ trap 'echo -e "\nWarning: A command has failed. Exiting the script. Line was ($0:$LINENO): $(sed -n "${LINENO}p" "$0")"; exit 3' ERR set -Eeuo pipefail -REPO_ROOT=$(git rev-parse --show-toplevel) -KUSTOMIZE_BIN="$REPO_ROOT/hack/tools/bin/kustomize" - -if [[ ! -x "$KUSTOMIZE_BIN" ]]; then - echo "fix $KUSTOMIZE_BIN: missing executable kustomize in hack/tools/bin" +if ! command -v kustomize | grep hack/tools/bin; then + echo "fix $PATH: kustomize should be from hack/tools/bin" exit 1 fi @@ -51,29 +48,29 @@ printf '%s' "$HETZNER_SSH_PRIV" | base64 --decode >tmp_ssh_priv kubectl create secret generic robot-ssh --from-literal=sshkey-name="$SSH_KEY_NAME" --from-file=ssh-privatekey=tmp_ssh_priv --from-file=ssh-publickey=tmp_ssh_pub --dry-run=client -o yaml >data/infrastructure-hetzner/v1beta1/cluster-template-hetzner-secret.yaml rm tmp_ssh_pub tmp_ssh_priv -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-k8s-upgrade-kcp-scale-in.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-loadbalancer-off.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-load-balancer-extra-services.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hcloud-feature-placement-groups.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-network --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-network.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-network --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-network.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-kcp-remediation.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-md-remediation.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-node-drain.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal.yaml -"$KUSTOMIZE_BIN" build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup.yaml +kustomize build "$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup --load-restrictor LoadRestrictionsNone >"$HETZNER_TEMPLATES"/v1beta1/cluster-template-hetzner-baremetal-feature-raid-setup.yaml sed -i "s/robot-user_secret_placeholder/$(echo -n "$HETZNER_ROBOT_USER" | base64 -w0)/" "$HETZNER_TEMPLATES"/v1beta1/*.yaml sed -i "s/robot-password_secret_placeholder/$(echo -n "$HETZNER_ROBOT_PASSWORD" | base64 -w0)/" "$HETZNER_TEMPLATES"/v1beta1/*.yaml From 8194ce75a540ea829ded4e5d2d14330666c5b68e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 19:11:47 +0200 Subject: [PATCH 11/14] Configure Syself CCM to use hrobot provider IDs --- Makefile | 3 ++- .../02-topics/05-baremetal/03-creating-workload-cluster.md | 1 + test/e2e/Makefile | 1 + test/e2e/data/ccm/syself-ccm-hetzner.yaml | 2 ++ 4 files changed, 6 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 5985a0ad3..57748a7ed 100644 --- a/Makefile +++ b/Makefile @@ -204,7 +204,8 @@ else helm repo update syself KUBECONFIG=$(WORKER_CLUSTER_KUBECONFIG) helm upgrade --install ccm syself/ccm-hetzner --version 2.0.6 \ --namespace kube-system \ - --set privateNetwork.enabled=$(PRIVATE_NETWORK) + --set privateNetwork.enabled=$(PRIVATE_NETWORK) \ + --set-json 'extraEnvVars=[{"name":"HCLOUD_USE_HROBOT_PROVIDER_ID_FOR_BAREMETAL","value":"true"}]' @echo @echo 'run "kubectl --kubeconfig=$(WORKER_CLUSTER_KUBECONFIG) ..." to work with the new target cluster' @echo diff --git a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md index 0f054b042..ba83b2969 100644 --- a/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md +++ b/docs/caph/02-topics/05-baremetal/03-creating-workload-cluster.md @@ -117,6 +117,7 @@ helm repo update syself helm upgrade --install ccm syself/ccm-hetzner --version 2.0.6 \ --namespace kube-system \ + --set-json 'extraEnvVars=[{"name":"HCLOUD_USE_HROBOT_PROVIDER_ID_FOR_BAREMETAL","value":"true"}]' \ --kubeconfig workload-kubeconfig ``` diff --git a/test/e2e/Makefile b/test/e2e/Makefile index d53bd0c7a..a19b7614d 100644 --- a/test/e2e/Makefile +++ b/test/e2e/Makefile @@ -96,6 +96,7 @@ e2e-ccm-templates: --set pdb.enabled=false \ --set privateNetwork.enabled=false \ --set secret.name=hcloud \ + --set-json 'extraEnvVars=[{"name":"HCLOUD_USE_HROBOT_PROVIDER_ID_FOR_BAREMETAL","value":"true"}]' \ --set-json 'tolerations=[{"key":"node.cluster.x-k8s.io/uninitialized","operator":"Exists","effect":"NoSchedule"},{"key":"node.cilium.io/agent-not-ready","operator":"Exists","effect":"NoSchedule"}]' \ > $(REPO_ROOT)/test/e2e/data/ccm/syself-ccm-hetzner.yaml diff --git a/test/e2e/data/ccm/syself-ccm-hetzner.yaml b/test/e2e/data/ccm/syself-ccm-hetzner.yaml index 684ac29a7..7346d9657 100644 --- a/test/e2e/data/ccm/syself-ccm-hetzner.yaml +++ b/test/e2e/data/ccm/syself-ccm-hetzner.yaml @@ -119,3 +119,5 @@ spec: value: "true" - name: HCLOUD_LOAD_BALANCERS_ENABLED value: "true" + - name: HCLOUD_USE_HROBOT_PROVIDER_ID_FOR_BAREMETAL + value: "true" From 0b1ae1f2ad7643211f7ee6f6d3b95fd1bee2266b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 19:59:50 +0200 Subject: [PATCH 12/14] Re-enable bare-metal host 1763731 --- .../v1beta1/bases/hetznerbaremetalhosts-raid.yaml | 3 +-- .../v1beta1/bases/hetznerbaremetalhosts.yaml | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml index 6ffe0981f..49e7d352a 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml @@ -12,7 +12,6 @@ spec: maintenanceMode: false description: Test Machine 2 --- -# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -24,5 +23,5 @@ spec: wwn: - "0x500a07511756c36d" - "0x500a075117101996" - maintenanceMode: true + maintenanceMode: false description: Test Machine 6 diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml index 680989f32..441b1d4e5 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml @@ -71,7 +71,6 @@ spec: maintenanceMode: true #### < --- set to false, when ok. description: Test Machine 1751550 --- -# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -83,5 +82,5 @@ spec: serverID: 1763731 # ip: 144.76.101.50 rootDeviceHints: wwn: "0x500a07511756c36d" - maintenanceMode: true + maintenanceMode: false description: Test Machine 1751550 From 7ad992162a45cd4d5c588cdd7d5f5ab80f5cf5b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 20:30:28 +0200 Subject: [PATCH 13/14] Keep bare-metal host 1763731 in maintenance --- .../v1beta1/bases/hetznerbaremetalhosts-raid.yaml | 4 ++-- .../v1beta1/bases/hetznerbaremetalhosts.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml index 49e7d352a..d9c216104 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml @@ -11,7 +11,7 @@ spec: - "0x500a07510fe0cc0a" maintenanceMode: false description: Test Machine 2 ---- +# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -23,5 +23,5 @@ spec: wwn: - "0x500a07511756c36d" - "0x500a075117101996" - maintenanceMode: false + maintenanceMode: true description: Test Machine 6 diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml index 441b1d4e5..ac95d88c0 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml @@ -26,7 +26,7 @@ spec: wwn: "0x5002538d41d70a46" maintenanceMode: false description: Test Machine 1716772 ---- +# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -82,5 +82,5 @@ spec: serverID: 1763731 # ip: 144.76.101.50 rootDeviceHints: wwn: "0x500a07511756c36d" - maintenanceMode: false + maintenanceMode: true description: Test Machine 1751550 From d28154abeb484cc3ebd5ca74996eeb3ec38295cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20G=C3=BCttler?= Date: Thu, 9 Apr 2026 20:31:50 +0200 Subject: [PATCH 14/14] Restore bare-metal host maintenance manifests --- .../v1beta1/bases/hetznerbaremetalhosts-raid.yaml | 1 + .../v1beta1/bases/hetznerbaremetalhosts.yaml | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml index d9c216104..6ffe0981f 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts-raid.yaml @@ -11,6 +11,7 @@ spec: - "0x500a07510fe0cc0a" maintenanceMode: false description: Test Machine 2 +--- # 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost diff --git a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml index ac95d88c0..680989f32 100644 --- a/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml +++ b/test/e2e/data/infrastructure-hetzner/v1beta1/bases/hetznerbaremetalhosts.yaml @@ -26,7 +26,7 @@ spec: wwn: "0x5002538d41d70a46" maintenanceMode: false description: Test Machine 1716772 -# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. +--- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: @@ -71,6 +71,7 @@ spec: maintenanceMode: true #### < --- set to false, when ok. description: Test Machine 1751550 --- +# 2026-04-09: Disabled after CI hit repeated rescue-mode reboot timeouts. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: HetznerBareMetalHost metadata: