Skip to content

Commit 7b5e6f7

Browse files
sebrandon1sebrandon1
committed
Add hub-spoke trust config, root CA monitoring, and kubeconfig guidance
Hub-spoke CA trust distribution via KlusterletConfig and CA ConfigMap, made mandatory with kube-compare validation templates. Root CA expiration monitoring via ACM Policy with PrometheusRule alerts. CertificatePolicy expanded to monitor cert-manager namespace. Kubeconfig trust documentation added to all profile READMEs, applicable to any non-publicly-trusted issuer. ACME is the reference recommendation. Other issuer types are allowable but only ACME is provided in the reference.
1 parent fda0964 commit 7b5e6f7

17 files changed

Lines changed: 354 additions & 11 deletions

‎telco-core/configuration/reference-crs/optional/cert-manager/README.md‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@ Cert-manager automates the management and issuance of TLS certificates from vari
1616
- `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager
1717
- `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator
1818

19-
### Certificate Issuer
20-
- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge
19+
### Certificate Issuers
20+
21+
- `certManagerClusterIssuer.yaml` - **ACME issuer** with DNS-01 challenge (reference recommendation)
2122

2223
### Certificate Resources
2324
- `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint
@@ -34,7 +35,10 @@ Before applying these configurations, you must customize the following:
3435
1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`):
3536
- Update `email` with your contact email
3637
- Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53)
37-
- Add necessary credentials for your DNS provider
38+
- Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests
39+
40+
> **Note:** Other issuer types (e.g., CA issuer for disconnected environments with existing PKI) are allowable.
41+
> Users may configure their own ClusterIssuer; currently only ACME issuer is provided in the reference.
3842
3943
2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`):
4044
- Update `commonName` and `dnsNames` to match your cluster's domain
@@ -61,9 +65,41 @@ After applying these configurations, verify that:
6165
- API Server is using the certificate: Test HTTPS connection to API endpoint
6266
- Ingress is using the certificate: Test HTTPS connection to any route
6367

68+
## Important: Kubeconfig Trust After API Server Cert Replacement
69+
70+
> **Note:** When using a non-publicly-trusted issuer, you must complete this kubeconfig update
71+
> *before* applying the APIServer configuration (step 6 in the deployment order above).
72+
> Applying the APIServer configuration first will lock you out.
73+
74+
> **Warning:** When cert-manager replaces the API server certificate with one signed by a non-publicly-trusted CA,
75+
> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references
76+
> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands
77+
> will fail with `x509: certificate signed by unknown authority`.
78+
79+
### Updating kubeconfig
80+
81+
1. Extract the new root CA certificate:
82+
```bash
83+
oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt
84+
```
85+
86+
2. Update your kubeconfig to trust the new CA:
87+
```bash
88+
oc config set-cluster $(oc config current-context | cut -d/ -f2) \
89+
--certificate-authority=/tmp/root-ca.crt --embed-certs
90+
```
91+
92+
3. Verify connectivity:
93+
```bash
94+
oc cluster-info
95+
```
96+
97+
### Best practice for PKI environments
98+
99+
Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add the root CA PEM to `/etc/pki/ca-trust/source/anchors/` on your workstation and run `update-ca-trust`. All certificates issued from that root CA will then be trusted without per-cluster kubeconfig updates.
100+
64101
## References
65102

66103
- [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html)
67104
- [Cert-Manager Documentation](https://cert-manager.io/docs/)
68105
- [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/)
69-

‎telco-hub/configuration/reference-crs-kube-compare/compare_ignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,10 @@ required/gitops/addPluginsPolicy.yaml
3333
# not include the full policy content due to policy templating.
3434
required/acm/observabilityRoutePolicy.yaml
3535

36+
# cert-manager config CRs (user-specific, and policies with hub-side templating)
37+
optional/cert-manager/certManagerRootCAExpirationPolicy.yaml
38+
optional/cert-manager/kustomization.yaml
39+
3640
required/gitops/extra-manifests-policy.yaml
3741
# ArgoCD files
3842
kustomization.yaml

‎telco-hub/configuration/reference-crs-kube-compare/default_value.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -479,6 +479,12 @@ optional_cert_manager_certManagerCertificatePolicy:
479479
include:
480480
- openshift-ingress
481481
- openshift-config
482+
- cert-manager
483+
484+
optional_cert_manager_certManagerHubCAConfigMap:
485+
- data:
486+
ca-bundle.crt: |
487+
<replace with root CA PEM>
482488
483489
optional_cert_manager_certManagerSubscription:
484490
- spec:

‎telco-hub/configuration/reference-crs-kube-compare/metadata.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,29 @@ parts:
3131
allOrNoneOf:
3232
- path: optional/cert-manager/apiServerCertificate.yaml
3333
- path: optional/cert-manager/apiServerConfig.yaml
34+
- name: cert-manager-hub-spoke-trust
35+
description: |-
36+
Hub-spoke CA trust distribution via KlusterletConfig
37+
allOrNoneOf:
38+
- path: optional/cert-manager/certManagerHubCAConfigMap.yaml
39+
config:
40+
ignore-unspecified-fields: true
41+
- path: optional/cert-manager/certManagerKlusterletConfig.yaml
42+
config:
43+
ignore-unspecified-fields: true
3444
- name: cert-manager-monitoring
3545
description: |-
3646
Certificate monitoring policy for RHACM
3747
allOrNoneOf:
3848
- path: optional/cert-manager/certManagerCertificatePolicy.yaml
3949
- path: optional/cert-manager/certManagerCertificatePolicyPlacement.yaml
4050
- path: optional/cert-manager/certManagerCertificatePolicyPlacementBinding.yaml
51+
- name: cert-manager-root-ca-monitoring
52+
description: |-
53+
Root CA expiration monitoring via PrometheusRule
54+
allOrNoneOf:
55+
- path: optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml
56+
- path: optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml
4157
- name: optional-storage
4258
components:
4359
- name: local-storage-operator

‎telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ spec:
1919
spec:
2020
minimumDuration: {{ .spec.minimumDuration | default "720h" }}
2121
namespaceSelector:
22-
include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config") ) }}
22+
include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config" "cert-manager") ) }}
2323
remediationAction: inform
2424
severity: low
2525
remediationAction: inform

‎telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerHubCAConfigMap.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: cert-manager-ca-bundle
6+
namespace: multicluster-engine
7+
labels:
8+
import.open-cluster-management.io/ca-bundle: ""
9+
annotations:
10+
argocd.argoproj.io/sync-wave: "-29"
11+
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
12+
data:
13+
{{- if .data }}
14+
# Populate with the root CA PEM used by cert-manager to sign hub API/ingress certs.
15+
# Extract from the hub: oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d
16+
ca-bundle.crt: |
17+
{{ index .data "ca-bundle.crt" | trimSuffix "\n" | indent 4 }}
18+
{{- end }}

‎telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerKlusterletConfig.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
apiVersion: config.open-cluster-management.io/v1alpha1
3+
kind: KlusterletConfig
4+
metadata:
5+
name: {{ .metadata.name | default "cert-manager-ca-config" }}
6+
annotations:
7+
argocd.argoproj.io/sync-wave: "-29"
8+
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
9+
spec:
10+
hubKubeAPIServerConfig:
11+
serverVerificationStrategy: UseCustomCABundles
12+
trustedCABundles:
13+
- name: cert-manager-ca-bundle
14+
caBundle:
15+
namespace: multicluster-engine
16+
name: cert-manager-ca-bundle

‎telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
apiVersion: cluster.open-cluster-management.io/v1beta1
3+
kind: Placement
4+
metadata:
5+
name: root-ca-expiration-placement
6+
namespace: default
7+
spec:
8+
predicates:
9+
- requiredClusterSelector:
10+
labelSelector:
11+
matchExpressions:
12+
- key: local-cluster
13+
operator: In
14+
values:
15+
- "true"

‎telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
apiVersion: policy.open-cluster-management.io/v1
3+
kind: PlacementBinding
4+
metadata:
5+
name: root-ca-expiration-placementbinding
6+
namespace: default
7+
placementRef:
8+
name: root-ca-expiration-placement
9+
apiGroup: cluster.open-cluster-management.io
10+
kind: Placement
11+
subjects:
12+
- name: root-ca-expiration-monitor
13+
apiGroup: policy.open-cluster-management.io
14+
kind: Policy

‎telco-hub/configuration/reference-crs/optional/cert-manager/README.md‎

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@ Cert-manager automates the management and issuance of TLS certificates from vari
1616
- `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager
1717
- `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator
1818

19-
### Certificate Issuer
20-
- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge
19+
### Certificate Issuers
20+
21+
- `certManagerClusterIssuer.yaml` - **ACME issuer** with DNS-01 challenge (reference recommendation)
2122

2223
### Certificate Resources
2324
- `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint
@@ -34,7 +35,10 @@ Before applying these configurations, you must customize the following:
3435
1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`):
3536
- Update `email` with your contact email
3637
- Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53)
37-
- Add necessary credentials for your DNS provider
38+
- Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests
39+
40+
> **Note:** Other issuer types (e.g., CA issuer for disconnected environments with existing PKI) are allowable.
41+
> Users may configure their own ClusterIssuer; currently only ACME issuer is provided in the reference.
3842
3943
2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`):
4044
- Update `commonName` and `dnsNames` to match your cluster's domain
@@ -49,8 +53,9 @@ Before applying these configurations, you must customize the following:
4953
1. Deploy operator installation files (NS, OperatorGroup, Subscription)
5054
2. Wait for operator to be ready
5155
3. Deploy the ClusterIssuer
52-
4. Wait for certificates to be issued and secrets created
53-
5. Apply the APIServer and IngressController configurations
56+
4. Deploy the Certificate resources
57+
5. Wait for certificates to be issued and secrets created
58+
6. Apply the APIServer and IngressController configurations
5459

5560
## Certificate Verification
5661

@@ -60,9 +65,96 @@ After applying these configurations, verify that:
6065
- API Server is using the certificate: Test HTTPS connection to API endpoint
6166
- Ingress is using the certificate: Test HTTPS connection to any route
6267

68+
## Important: Kubeconfig Trust After API Server Cert Replacement
69+
70+
> **Note:** When using a non-publicly-trusted issuer, you must complete this kubeconfig update
71+
> *before* applying the APIServer configuration (step 6 in the deployment order above).
72+
> Applying the APIServer configuration first will lock you out.
73+
74+
> **Warning:** When cert-manager replaces the API server certificate with one signed by a non-publicly-trusted CA,
75+
> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references
76+
> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands
77+
> will fail with `x509: certificate signed by unknown authority`.
78+
79+
### Updating kubeconfig
80+
81+
1. Extract the new root CA certificate:
82+
```bash
83+
oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt
84+
```
85+
86+
2. Update your kubeconfig to trust the new CA:
87+
```bash
88+
oc config set-cluster $(oc config current-context | cut -d/ -f2) \
89+
--certificate-authority=/tmp/root-ca.crt --embed-certs
90+
```
91+
92+
3. Verify connectivity:
93+
```bash
94+
oc cluster-info
95+
```
96+
97+
### Best practice for PKI environments
98+
99+
Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add the root CA PEM to `/etc/pki/ca-trust/source/anchors/` on your workstation and run `update-ca-trust`. All certificates issued from that root CA will then be trusted without per-cluster kubeconfig updates.
100+
101+
## Hub-Spoke Trust with ACM
102+
103+
When cert-manager issues certificates for the hub's API server and ingress, managed spokes must trust the cert-manager root CA to maintain connectivity. The reference configuration includes a `KlusterletConfig` and CA ConfigMap to distribute the root CA to spokes automatically.
104+
105+
### Hub-spoke trust files
106+
107+
- `certManagerHubCAConfigMap.yaml` — ConfigMap in `multicluster-engine` namespace containing the cert-manager root CA, labeled for the import controller
108+
- `certManagerKlusterletConfig.yaml` — KlusterletConfig that switches spoke CA verification from auto-detected leaf cert to the custom root CA bundle
109+
110+
After deploying these resources, annotate each managed cluster:
111+
112+
```bash
113+
oc annotate managedcluster <spoke-name> \
114+
agent.open-cluster-management.io/klusterlet-config=cert-manager-ca-config
115+
```
116+
117+
### Why this is needed
118+
119+
By default, ACM embeds the hub's leaf serving cert (`CA:FALSE`) in the klusterlet bootstrap kubeconfig. This means every cert rotation requires a ManifestWork update, and a full CA replacement breaks all spokes immediately. The `KlusterletConfig` with `UseCustomCABundles` replaces the leaf cert with the root CA (`CA:TRUE`), so any certificate signed by that root — current or rotated — is automatically trusted.
120+
121+
### Greenfield (cert-manager before spoke deployment)
122+
123+
1. Deploy cert-manager on the hub, create the CA, issue hub API/ingress certs
124+
2. Deploy `certManagerHubCAConfigMap.yaml` and `certManagerKlusterletConfig.yaml`
125+
3. Annotate the managed cluster with `agent.open-cluster-management.io/klusterlet-config=cert-manager-ca-config` as shown above
126+
4. Deploy spokes via ZTP — they register with the root CA in their trust store
127+
5. Cert rotations are seamless with no intervention required
128+
129+
### Brownfield (cert-manager on existing hub with connected spokes)
130+
131+
The order matters — distribute the CA **before** replacing the hub certs:
132+
133+
1. Install cert-manager on the hub, create the CA — but **do not apply certs to the APIServer/IngressController yet**
134+
2. Deploy `certManagerHubCAConfigMap.yaml` with the root CA PEM
135+
3. Deploy `certManagerKlusterletConfig.yaml`
136+
4. Annotate all managed clusters with `agent.open-cluster-management.io/klusterlet-config=cert-manager-ca-config` as shown above
137+
5. Wait for the import controller to regenerate bootstrap kubeconfigs (check logs for `create a new bootstrap kubeconfig`)
138+
6. **Now** apply the cert-manager certs to the APIServer and IngressController
139+
7. Spokes stay connected because they already trust the root CA
140+
141+
### Cert rotation
142+
143+
Once the root CA is in the klusterlet's trust store, cert rotations are seamless. The klusterlet trusts any certificate signed by the root CA regardless of serial number, with no ManifestWork timing dependency.
144+
145+
## Root CA Expiration Monitoring
146+
147+
The `certManagerRootCAExpirationPolicy.yaml` creates a PrometheusRule that monitors the root CA certificate expiration using the `certmanager_certificate_expiration_timestamp_seconds` metric:
148+
149+
- **Warning** at 90 days before expiry
150+
- **Critical** at 30 days before expiry
151+
152+
This is distinct from the existing `certManagerCertificatePolicy.yaml` which monitors leaf certificate expiration in `openshift-ingress`, `openshift-config`, and `cert-manager` namespaces via ACM CertificatePolicy. Both should be deployed together for comprehensive certificate monitoring.
153+
63154
## References
64155

65156
- [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html)
66157
- [Cert-Manager Documentation](https://cert-manager.io/docs/)
67158
- [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/)
68-
159+
- [Hub-Spoke Trust — Complete Solution](https://gist.github.com/sebrandon1/7265d68c5add6adb1313dce5b695e40d)
160+
- [Hub-Spoke Trust Test Results](https://gist.github.com/sebrandon1/483180614951d23174c4e365a9a02a34)

0 commit comments

Comments
 (0)