Add hub-spoke trust config, root CA monitoring, and kubeconfig guidance

telco-core/configuration/reference-crs/optional/cert-manager/README.md

@@ -16,8 +16,9 @@ Cert-manager automates the management and issuance of TLS certificates from vari
 - `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager
 - `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator
 
-### Certificate Issuer
-- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge
+### Certificate Issuers
+
+- `certManagerClusterIssuer.yaml` - **ACME issuer** with DNS-01 challenge (reference recommendation)
 
 ### Certificate Resources
 - `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint
@@ -34,13 +35,22 @@ Before applying these configurations, you must customize the following:
 1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`):
    - Update `email` with your contact email
    - Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53)
-   - Add necessary credentials for your DNS provider
+   - Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests
+
+   > **Note:** Other issuer types (e.g., CA issuer for disconnected environments with existing PKI) are allowable.
+   > Users may configure their own ClusterIssuer; currently only ACME issuer is provided in the reference.
 
 2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`):
    - Update `commonName` and `dnsNames` to match your cluster's domain
    - Example: Replace `api.example.com` with your actual API endpoint
    - Example: Replace `*.apps.example.com` with your actual wildcard domain
 
+   > **Important:** The reference configuration uses ECDSA P-256 for Certificate resources, which is the recommended algorithm for TLS 1.3.
+   > While RSA certificates are still supported for authentication in TLS 1.3, RSA key exchange was removed and RSA does not provide Forward Secrecy.
+   > 
+   > **Note:** Lifecycle-agent currently has limited support for ECDSA certificates (being addressed in lifecycle-agent PR #7610).
+   > For testing purposes, QE may temporarily use RSA certificates, but production deployments should use ECDSA.
+
 3. **APIServer Configuration** (`apiServerConfig.yaml`):
    - Update the `names` field to match your API Server FQDN
 
@@ -61,9 +71,75 @@ After applying these configurations, verify that:
 - API Server is using the certificate: Test HTTPS connection to API endpoint
 - Ingress is using the certificate: Test HTTPS connection to any route
 
+## Important: Kubeconfig Trust After API Server Cert Replacement
+
+> **Note:** When using a non-publicly-trusted issuer, you must complete this kubeconfig update
+> *before* applying the APIServer configuration (step 6 in the deployment order above).
+> Applying the APIServer configuration first will lock you out.
+
+> **Warning:** When cert-manager replaces the API server certificate with one signed by a non-publicly-trusted CA,
+> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references
+> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands
+> will fail with `x509: certificate signed by unknown authority`.
+
+### Updating kubeconfig
+
+1. Extract the new root CA certificate:
+   ```bash
+   oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt
+   ```
+
+2. Update your kubeconfig to trust the new CA:
+   ```bash
+   oc config set-cluster $(oc config current-context | cut -d/ -f2) \
+     --certificate-authority=/tmp/root-ca.crt --embed-certs
+   ```
+
+3. Verify connectivity:
+   ```bash
+   oc cluster-info
+   ```
+
+### Best practice for PKI environments
+
+Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add this root CA to your workstation's system trust store so all certificates issued from it are automatically trusted by workstation tools and browsers. Note that adding the CA to your system trust store does not automatically update existing kubeconfigs with embedded certificate data — see the "Clearing kubeconfig certificate data" section below.
+
+#### Adding CA to system trust store
+
+**Red Hat/Fedora/CentOS:**
+```bash
+sudo cp /tmp/root-ca.crt /etc/pki/ca-trust/source/anchors/
+sudo update-ca-trust
+```
+
+**Debian/Ubuntu:**
+```bash
+sudo cp /tmp/root-ca.crt /usr/local/share/ca-certificates/root-ca.crt
+sudo update-ca-certificates
+```
+
+**macOS:**
+```bash
+sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /tmp/root-ca.crt
+```
+
+#### Clearing kubeconfig certificate data
+
+**Important:** `oc` and `kubectl` do NOT fall back to the OS trust store if your kubeconfig contains embedded certificate data (`certificate-authority-data` field). You must clear this field to use the system trust store:
+
+```bash
+# Remove embedded certificate data from current cluster
+CLUSTER_NAME=$(oc config view --minify -o jsonpath='{.clusters[0].name}')
+oc config unset "clusters.${CLUSTER_NAME}.certificate-authority-data"
+
+# Verify the OS trust store is now used
+oc cluster-info
+```
+
+After completing these steps, all future clusters using certificates signed by your root CA will be automatically trusted without per-cluster kubeconfig updates.
+
 ## References
 
 - [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html)
 - [Cert-Manager Documentation](https://cert-manager.io/docs/)
 - [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/)
-

telco-core/install/example-standard-clusterinstance.yaml

@@ -31,6 +31,10 @@ spec:
       common: "core"
       version: "4.22"
       region: "zone-1"
+  # Uncomment to apply KlusterletConfig for cert-manager hub-spoke trust (optional)
+  # extraAnnotations:
+  #   ManagedCluster:
+  #     agent.open-cluster-management.io/klusterlet-config: "cert-manager-ca-config"
   clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23

telco-hub/configuration/reference-crs-kube-compare/compare_ignore

@@ -33,6 +33,10 @@ required/gitops/addPluginsPolicy.yaml
 # not include the full policy content due to policy templating.
 required/acm/observabilityRoutePolicy.yaml
 
+# cert-manager config CRs (user-specific, and policies with hub-side templating)
+optional/cert-manager/certManagerRootCAExpirationPolicy.yaml
+optional/cert-manager/kustomization.yaml
+
 required/gitops/extra-manifests-policy.yaml
 # ArgoCD files
 kustomization.yaml

telco-hub/configuration/reference-crs-kube-compare/default_value.yaml

@@ -479,6 +479,12 @@ optional_cert_manager_certManagerCertificatePolicy:
       include:
         - openshift-ingress
         - openshift-config
+        - cert-manager
+
+optional_cert_manager_certManagerHubCAConfigMap:
+- data:
+    ca-bundle.crt: |
+      <replace with root CA PEM>
 
 optional_cert_manager_certManagerSubscription:
 - spec:

telco-hub/configuration/reference-crs-kube-compare/metadata.yaml

@@ -31,13 +31,29 @@ parts:
         allOrNoneOf:
           - path: optional/cert-manager/apiServerCertificate.yaml
           - path: optional/cert-manager/apiServerConfig.yaml
+      - name: cert-manager-hub-spoke-trust
+        description: |-
+          Hub-spoke CA trust distribution via KlusterletConfig
+        allOrNoneOf:
+          - path: optional/cert-manager/certManagerHubCAConfigMap.yaml
+            config:
+              ignore-unspecified-fields: true
+          - path: optional/cert-manager/certManagerKlusterletConfig.yaml
+            config:
+              ignore-unspecified-fields: true
       - name: cert-manager-monitoring
         description: |-
           Certificate monitoring policy for RHACM
         allOrNoneOf:
           - path: optional/cert-manager/certManagerCertificatePolicy.yaml
           - path: optional/cert-manager/certManagerCertificatePolicyPlacement.yaml
           - path: optional/cert-manager/certManagerCertificatePolicyPlacementBinding.yaml
+      - name: cert-manager-root-ca-monitoring
+        description: |-
+          Root CA expiration monitoring via PrometheusRule
+        allOrNoneOf:
+          - path: optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml
+          - path: optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml
   - name: optional-storage
     components:
       - name: local-storage-operator

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml

@@ -19,7 +19,7 @@ spec:
         spec:
           minimumDuration: {{ .spec.minimumDuration | default "720h" }}
           namespaceSelector:
-            include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config") ) }}
+            include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config" "cert-manager") ) }}
           remediationAction: inform
           severity: low
   remediationAction: inform

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerHubCAConfigMap.yaml

@@ -0,0 +1,18 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cert-manager-hub-ca-bundle
+  namespace: multicluster-engine
+  labels:
+    import.open-cluster-management.io/ca-bundle: "true"
+  annotations:
+    argocd.argoproj.io/sync-wave: "-29"
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+data:
+{{- if .data }}
+  # Populate with the root CA PEM used by cert-manager to sign hub API/ingress certs.
+  # Extract from the hub: oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d
+  ca-bundle.crt: |
+{{ index .data "ca-bundle.crt" | trimSuffix "\n" | indent 4 }}
+{{- end }}

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerKlusterletConfig.yaml

@@ -0,0 +1,16 @@
+---
+apiVersion: config.open-cluster-management.io/v1alpha1
+kind: KlusterletConfig
+metadata:
+  name: {{ .metadata.name | default "cert-manager-ca-config" }}
+  annotations:
+    argocd.argoproj.io/sync-wave: "-29"
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  hubKubeAPIServerConfig:
+    serverVerificationStrategy: UseCustomCABundles
+    trustedCABundles:
+    - name: cert-manager-hub-ca-bundle
+      caBundle:
+        namespace: multicluster-engine
+        name: cert-manager-hub-ca-bundle

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml

@@ -0,0 +1,18 @@
+---
+apiVersion: cluster.open-cluster-management.io/v1beta1
+kind: Placement
+metadata:
+  name: root-ca-expiration-placement
+  namespace: default
+  annotations:
+    argocd.argoproj.io/sync-wave: "-11"
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  predicates:
+    - requiredClusterSelector:
+        labelSelector:
+          matchExpressions:
+            - key: local-cluster
+              operator: In
+              values:
+                - "true"

telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: policy.open-cluster-management.io/v1
+kind: PlacementBinding
+metadata:
+  name: root-ca-expiration-placementbinding
+  namespace: default
+  annotations:
+    argocd.argoproj.io/sync-wave: "-11"
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+placementRef:
+  name: root-ca-expiration-placement
+  apiGroup: cluster.open-cluster-management.io
+  kind: Placement
+subjects:
+  - name: root-ca-expiration-monitor
+    apiGroup: policy.open-cluster-management.io
+    kind: Policy