Add CA issuer, hub-spoke trust config, and root CA monitoring

Adds CA ClusterIssuer for disconnected environments with existing PKI,
KlusterletConfig for hub-spoke CA trust distribution, and root CA
expiration monitoring via ACM Policy.

The KlusterletConfig with UseCustomCABundles ensures spokes trust the
hub's root CA (CA:TRUE) instead of the leaf serving cert (CA:FALSE).
This makes cert rotations seamless and eliminates the chicken-and-egg
problem where spokes can't receive ManifestWork updates because they
don't trust the hub's new cert.

Tested on OCP 4.21.15 hub+spoke with cert-manager v1.19.0. Full test
results: https://gist.github.com/sebrandon1/7265d68c5add6adb1313dce5b695e40d

Signed-off-by: Brandon Palm <bpalm@redhat.com>

Author: sebrandon1

telco-core/configuration/reference-crs-kube-compare/compare_ignore

@@ -38,3 +38,6 @@ custom-manifests/subscription-validator.yaml
 
 # This is an object-template-raw used only to validate pre-caching state
 custom-manifests/precache-validator.yaml
+
+# cert-manager config CRs (user-specific)
+optional/cert-manager/certManagerClusterIssuerCA.yaml

telco-core/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerClusterIssuerCA.yaml

@@ -0,0 +1,8 @@
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: {{ .metadata.name | default "ca-issuer" }}
+spec:
+  ca:
+    secretName: {{ .spec.ca.secretName | default "root-ca-secret" }}

telco-core/configuration/reference-crs/optional/cert-manager/README.md

@@ -16,8 +16,12 @@ Cert-manager automates the management and issuance of TLS certificates from vari
 - `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager
 - `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator
 
-### Certificate Issuer
-- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge
+### Certificate Issuers
+
+Two ClusterIssuer types are available. Choose one based on your environment:
+
+- `certManagerClusterIssuer.yaml` - **ACME issuer** using Let's Encrypt with DNS-01 challenge (requires internet connectivity)
+- `certManagerClusterIssuerCA.yaml` - **CA issuer** using an internal certificate authority (disconnected environments with existing PKI)
 
 ### Certificate Resources
 - `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint
@@ -31,10 +35,18 @@ Cert-manager automates the management and issuance of TLS certificates from vari
 
 Before applying these configurations, you must customize the following:
 
-1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`):
+1. **ClusterIssuer** — choose one of the following issuer configurations:
+
+   **Option A: ACME issuer** (`certManagerClusterIssuer.yaml`) — for connected environments:
    - Update `email` with your contact email
    - Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53)
-   - Add necessary credentials for your DNS provider
+   - Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests
+
+   **Option B: Existing internal CA** (disconnected environments with existing PKI):
+   - This issuer does **not** generate its own CA — it requires a pre-existing CA from your organization's PKI
+   - Create a Secret named `root-ca-secret` in the `cert-manager` namespace containing your CA's `tls.crt` and `tls.key`
+   - Deploy only `certManagerClusterIssuerCA.yaml`
+   - Update the Certificate resources to reference `ca-issuer` in the `issuerRef` field
 
 2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`):
    - Update `commonName` and `dnsNames` to match your cluster's domain
@@ -51,7 +63,8 @@ Before applying these configurations, you must customize the following:
 3. Deploy the ClusterIssuer
 4. Deploy the Certificate resources
 5. Wait for certificates to be issued and secrets created
-6. Apply the APIServer and IngressController configurations
+6. **(Option B only)** Update kubeconfig to trust the new root CA (see "Kubeconfig Trust" section below)
+7. Apply the APIServer and IngressController configurations
 
 ## Certificate Verification
 
@@ -61,9 +74,41 @@ After applying these configurations, verify that:
 - API Server is using the certificate: Test HTTPS connection to API endpoint
 - Ingress is using the certificate: Test HTTPS connection to any route
 
+## Important: Kubeconfig Trust After API Server Cert Replacement
+
+> **Note:** For Option B (Existing internal CA), you must complete this kubeconfig update
+> *before* applying the APIServer configuration (step 7 in the deployment order above).
+> Applying the APIServer configuration first will lock you out.
+
+> **Warning:** When cert-manager replaces the API server certificate with one signed by a custom CA,
+> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references
+> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands
+> will fail with `x509: certificate signed by unknown authority`.
+
+### Updating kubeconfig
+
+1. Extract the new root CA certificate:
+   ```bash
+   oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt
+   ```
+
+2. Update your kubeconfig to trust the new CA:
+   ```bash
+   oc config set-cluster $(oc config current-context | cut -d/ -f2) \
+     --certificate-authority=/tmp/root-ca.crt --embed-certs
+   ```
+
+3. Verify connectivity:
+   ```bash
+   oc cluster-info
+   ```
+
+### Best practice for PKI environments
+
+Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add the root CA PEM to `/etc/pki/ca-trust/source/anchors/` on your workstation and run `update-ca-trust`. All certificates issued from that root CA will then be trusted without per-cluster kubeconfig updates.
+
 ## References
 
 - [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html)
 - [Cert-Manager Documentation](https://cert-manager.io/docs/)
 - [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/)
-

telco-core/configuration/reference-crs/optional/cert-manager/certManagerClusterIssuerCA.yaml

@@ -0,0 +1,10 @@
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: ca-issuer
+spec:
+  ca:
+    # References a Secret in the cert-manager namespace containing tls.crt and tls.key
+    # for your internal CA. Create this Secret before applying the ClusterIssuer.
+    secretName: root-ca-secret

telco-hub/configuration/reference-crs-kube-compare/compare_ignore

@@ -33,6 +33,15 @@ required/gitops/addPluginsPolicy.yaml
 # not include the full policy content due to policy templating.
 required/acm/observabilityRoutePolicy.yaml
 
+# cert-manager config CRs (user-specific, and policies with hub-side templating)
+optional/cert-manager/certManagerClusterIssuerCA.yaml
+optional/cert-manager/certManagerRootCAExpirationPolicy.yaml
+optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml
+optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml
+optional/cert-manager/certManagerHubCAConfigMap.yaml
+optional/cert-manager/certManagerKlusterletConfig.yaml
+optional/cert-manager/kustomization.yaml
+
 required/gitops/extra-manifests-policy.yaml
 # ArgoCD files
 kustomization.yaml

telco-hub/configuration/reference-crs-kube-compare/metadata.yaml

@@ -186,9 +186,6 @@ fieldsToOmit:
         isPrefix: true
       - pathToKey: metadata.annotations."installer.open-cluster-management.io"
         isPrefix: true
-      - pathToKey: metadata.annotations.createdAt
-      - pathToKey: metadata.annotations.createdBy
-      - pathToKey: metadata.annotations."oc-mirror_version"
       - pathToKey: metadata.annotations."security.openshift.io/MinimallySufficientPodSecurityStandard"
       - pathToKey: metadata.labels."kubernetes.io/metadata.name"
       - pathToKey: metadata.labels."security.openshift.io/scc.podSecurityLabelSync"

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml

@@ -19,7 +19,7 @@ spec:
         spec:
           minimumDuration: {{ .spec.minimumDuration | default "720h" }}
           namespaceSelector:
-            include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config") ) }}
+            include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config" "cert-manager") ) }}
           remediationAction: inform
           severity: low
   remediationAction: inform

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerClusterIssuerCA.yaml

@@ -0,0 +1,11 @@
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: {{ .metadata.name | default "ca-issuer" }}
+  annotations:
+    argocd.argoproj.io/sync-wave: "-28"
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  ca:
+    secretName: {{ .spec.ca.secretName | default "root-ca-secret" }}

telco-hub/configuration/reference-crs/optional/cert-manager/README.md

@@ -16,8 +16,12 @@ Cert-manager automates the management and issuance of TLS certificates from vari
 - `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager
 - `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator
 
-### Certificate Issuer
-- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge
+### Certificate Issuers
+
+Two ClusterIssuer types are available. Choose one based on your environment:
+
+- `certManagerClusterIssuer.yaml` - **ACME issuer** using Let's Encrypt with DNS-01 challenge (requires internet connectivity)
+- `certManagerClusterIssuerCA.yaml` - **CA issuer** using an internal certificate authority (disconnected environments with existing PKI)
 
 ### Certificate Resources
 - `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint
@@ -31,10 +35,18 @@ Cert-manager automates the management and issuance of TLS certificates from vari
 
 Before applying these configurations, you must customize the following:
 
-1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`):
+1. **ClusterIssuer** — choose one of the following issuer configurations:
+
+   **Option A: ACME issuer** (`certManagerClusterIssuer.yaml`) — for connected environments:
    - Update `email` with your contact email
    - Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53)
-   - Add necessary credentials for your DNS provider
+   - Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests
+
+   **Option B: Existing internal CA** (disconnected environments with existing PKI):
+   - This issuer does **not** generate its own CA — it requires a pre-existing CA from your organization's PKI
+   - Create a Secret named `root-ca-secret` in the `cert-manager` namespace containing your CA's `tls.crt` and `tls.key`
+   - Deploy only `certManagerClusterIssuerCA.yaml`
+   - Update the Certificate resources to reference `ca-issuer` in the `issuerRef` field
 
 2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`):
    - Update `commonName` and `dnsNames` to match your cluster's domain
@@ -49,8 +61,10 @@ Before applying these configurations, you must customize the following:
 1. Deploy operator installation files (NS, OperatorGroup, Subscription)
 2. Wait for operator to be ready
 3. Deploy the ClusterIssuer
-4. Wait for certificates to be issued and secrets created
-5. Apply the APIServer and IngressController configurations
+4. Deploy the Certificate resources
+5. Wait for certificates to be issued and secrets created
+6. **(Option B only)** Update kubeconfig to trust the new root CA (see "Kubeconfig Trust" section below)
+7. Apply the APIServer and IngressController configurations
 
 ## Certificate Verification
 
@@ -60,9 +74,96 @@ After applying these configurations, verify that:
 - API Server is using the certificate: Test HTTPS connection to API endpoint
 - Ingress is using the certificate: Test HTTPS connection to any route
 
+## Important: Kubeconfig Trust After API Server Cert Replacement
+
+> **Note:** For Option B (Existing internal CA), you must complete this kubeconfig update
+> *before* applying the APIServer configuration (step 7 in the deployment order above).
+> Applying the APIServer configuration first will lock you out.
+
+> **Warning:** When cert-manager replaces the API server certificate with one signed by a custom CA,
+> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references
+> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands
+> will fail with `x509: certificate signed by unknown authority`.
+
+### Updating kubeconfig
+
+1. Extract the new root CA certificate:
+   ```bash
+   oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt
+   ```
+
+2. Update your kubeconfig to trust the new CA:
+   ```bash
+   oc config set-cluster $(oc config current-context | cut -d/ -f2) \
+     --certificate-authority=/tmp/root-ca.crt --embed-certs
+   ```
+
+3. Verify connectivity:
+   ```bash
+   oc cluster-info
+   ```
+
+### Best practice for PKI environments
+
+Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add the root CA PEM to `/etc/pki/ca-trust/source/anchors/` on your workstation and run `update-ca-trust`. All certificates issued from that root CA will then be trusted without per-cluster kubeconfig updates.
+
+## Hub-Spoke Trust with ACM
+
+When cert-manager issues certificates for the hub's API server and ingress, managed spokes must trust the cert-manager root CA to maintain connectivity. The reference configuration includes a `KlusterletConfig` and CA ConfigMap to distribute the root CA to spokes automatically.
+
+### Hub-spoke trust files
+
+- `certManagerHubCAConfigMap.yaml` — ConfigMap in `multicluster-engine` namespace containing the cert-manager root CA, labeled for the import controller
+- `certManagerKlusterletConfig.yaml` — KlusterletConfig that switches spoke CA verification from auto-detected leaf cert to the custom root CA bundle
+
+After deploying these resources, annotate each managed cluster:
+
+```bash
+oc annotate managedcluster <spoke-name> \
+  agent.open-cluster-management.io/klusterlet-config=cert-manager-ca-config
+```
+
+### Why this is needed
+
+By default, ACM embeds the hub's leaf serving cert (`CA:FALSE`) in the klusterlet bootstrap kubeconfig. This means every cert rotation requires a ManifestWork update, and a full CA replacement breaks all spokes immediately. The `KlusterletConfig` with `UseCustomCABundles` replaces the leaf cert with the root CA (`CA:TRUE`), so any certificate signed by that root — current or rotated — is automatically trusted.
+
+### Greenfield (cert-manager before spoke deployment)
+
+1. Deploy cert-manager on the hub, create the CA, issue hub API/ingress certs
+2. Deploy `certManagerHubCAConfigMap.yaml` and `certManagerKlusterletConfig.yaml`
+3. Annotate the managed cluster (or configure as a default KlusterletConfig)
+4. Deploy spokes via ZTP — they register with the root CA in their trust store
+5. Cert rotations are seamless with no intervention required
+
+### Brownfield (cert-manager on existing hub with connected spokes)
+
+The order matters — distribute the CA **before** replacing the hub certs:
+
+1. Install cert-manager on the hub, create the CA — but **do not apply certs to the APIServer/IngressController yet**
+2. Deploy `certManagerHubCAConfigMap.yaml` with the root CA PEM
+3. Deploy `certManagerKlusterletConfig.yaml`
+4. Annotate all managed clusters
+5. Wait for the import controller to regenerate bootstrap kubeconfigs (check logs for `create a new bootstrap kubeconfig`)
+6. **Now** apply the cert-manager certs to the APIServer and IngressController
+7. Spokes stay connected because they already trust the root CA
+
+### Cert rotation
+
+Once the root CA is in the klusterlet's trust store, cert rotations are seamless. The klusterlet trusts any certificate signed by the root CA regardless of serial number, with no ManifestWork timing dependency.
+
+## Root CA Expiration Monitoring
+
+The `certManagerRootCAExpirationPolicy.yaml` creates a PrometheusRule that monitors the root CA certificate expiration using the `certmanager_certificate_expiration_timestamp_seconds` metric:
+
+- **Warning** at 90 days before expiry
+- **Critical** at 30 days before expiry
+
+This is distinct from the existing `certManagerCertificatePolicy.yaml` which monitors leaf certificate expiration in `openshift-ingress`, `openshift-config`, and `cert-manager` namespaces via ACM CertificatePolicy. Both should be deployed together for comprehensive certificate monitoring.
+
 ## References
 
 - [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html)
 - [Cert-Manager Documentation](https://cert-manager.io/docs/)
 - [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/)
-
+- [Hub-Spoke Trust — Complete Solution](https://gist.github.com/sebrandon1/7265d68c5add6adb1313dce5b695e40d)
+- [Hub-Spoke Trust Test Results](https://gist.github.com/sebrandon1/483180614951d23174c4e365a9a02a34)

telco-hub/configuration/reference-crs/optional/cert-manager/certManagerCertificatePolicy.yaml

@@ -22,6 +22,7 @@ spec:
             include:
               - openshift-ingress
               - openshift-config
+              - cert-manager
           remediationAction: inform
           severity: low
   remediationAction: inform