Merge pull request #111 from restatedev/feat/trusted-ca-certs

feat: support trusted CA certificates in RestateCluster

Author: lukebond

AGENTS.md

@@ -5,3 +5,7 @@ Instructions for AI coding agents working on this repository.
 ## Release Notes
 
 When making changes, check whether the change warrants a release note by reviewing the guidelines in `release-notes/README.md`. If it does, create a release note file in `release-notes/unreleased/` as part of the same change.
+
+## Trusted CA Certs Init Container
+
+The trusted CA certs feature (`spec.security.trustedCaCerts`) uses an init container that reads the system CA bundle from `/etc/ssl/certs/ca-certificates.crt` (Debian/Alpine path). If the Restate server base image is changed to a different distro (e.g. RHEL uses `/etc/pki/tls/certs/ca-bundle.crt`), the `SYSTEM_CA_BUNDLE` constant in `src/controllers/restatecluster/reconcilers/compute.rs` must be updated.

CLAUDE.md

@@ -261,7 +261,7 @@ kubectl delete pod -n restate-operator -l app=restate-operator
 - `image.pullPolicy` - Pull policy (default: `IfNotPresent`)
 - `awsPodIdentityAssociationCluster` - Enables EKS Pod Identity support
 - `gcpWorkloadIdentity` - Enables GCP Workload Identity via Config Connector
-- `canaryImage` - Container image for canary jobs (default: `busybox:uclibc`); must provide `grep` and `wget`
+- `canaryImage` - Container image for canary jobs and the trusted CA certs init container (default: `busybox:uclibc`); must provide `cat`, `grep` and `wget`
 - `operatorNamespace` - Namespace where operator runs
 - `operatorLabelName/Value` - Labels for network policy selectors
 

README.md

@@ -165,6 +165,7 @@ This feature is particularly useful for Raft-based metadata clusters where manua
 | `awsPodIdentityAssociationRoleArn` | `string` | **Use this to grant your Restate cluster fine-grained access to other AWS resources (like S3) without managing static credentials.** Creates a `PodIdentityAssociation` to grant the cluster an IAM role. Requires the ACK EKS controller. |
 | `awsPodSecurityGroups` | `array` | **Use this to isolate your Restate cluster within specific AWS Security Groups for enhanced network control and auditing.** Creates a `SecurityGroupPolicy` to place pods into these security groups. Requires the Security Groups for Pods CRD. |
 | `requestSigningPrivateKey` | `object` | Configures a private key to sign outbound requests from this cluster. Can be sourced from a `secret` or a CSI `secretProvider`. See details below. |
+| `trustedCaCerts` | `array` | Optional list of Secrets containing trusted CA certificates. Each cert is appended to the system CA bundle via an init container. See details below. |
 
 ---
 
@@ -193,6 +194,30 @@ This feature is particularly useful for Raft-based metadata clusters where manua
 
 ---
 
+#### `spec.security.trustedCaCerts`
+
+Use this to trust custom CA certificates (e.g. for calling SDK services behind an internal load balancer with a private certificate, or for object store access via a private CA) without building a custom Restate image.
+The operator adds an init container that concatenates the system CA bundle with your custom certificates, and sets `SSL_CERT_FILE` to point to the combined bundle.
+
+Each entry references a Kubernetes Secret:
+
+| Field | Type | Description |
+|---|---|---|
+| `secretName` | `string` | **Required**. Name of the Secret containing the CA certificate. |
+| `key` | `string` | **Required**. Key within the Secret that contains the PEM-encoded certificate. |
+
+**Example:**
+
+```yaml
+spec:
+  security:
+    trustedCaCerts:
+      - secretName: internal-ca
+        key: ca.pem
+```
+
+---
+
 #### `spec.config`
 
 This field allows you to provide a TOML-encoded configuration string for the Restate server. This maps directly to the Restate server's configuration file. You can use this to configure aspects like roles, metadata storage, snapshotting, and more.
@@ -645,7 +670,8 @@ the `RestateCluster` spec.
 ### Canary Image
 
 Both EKS Pod Identity and GCP Workload Identity use a canary job to validate that credentials are available before
-starting the Restate cluster. By default, this uses the `busybox:uclibc` image from Docker Hub. In environments where
+starting the Restate cluster. The trusted CA certs feature also uses this image for its init container.
+By default, this uses the `busybox:uclibc` image from Docker Hub. In environments where
 nodes cannot pull from Docker Hub (e.g. air-gapped or restricted registries), you can override this with the
 `canaryImage` Helm value:
 
@@ -661,7 +687,7 @@ docker tag busybox:uclibc my-private-registry.example.com/busybox:uclibc
 docker push my-private-registry.example.com/busybox:uclibc
 ```
 
-If using a different base image, it must provide `grep` and `wget`.
+If using a different base image, it must provide `cat`, `grep` and `wget`.
 
 ### EKS Security Groups for Pods
 

charts/restate-operator-helm/values.yaml

@@ -16,7 +16,7 @@ podAnnotations: {}
 awsPodIdentityAssociationCluster: null
 gcpWorkloadIdentity: null
 clusterDns: null # defaults to "cluster.local" in the operator binary
-canaryImage: null # defaults to "busybox:uclibc"; image must provide grep and wget
+canaryImage: null # defaults to "busybox:uclibc"; image must provide cat, grep and wget
 
 podSecurityContext:
   fsGroup: 2000

crd/RestateCluster.pkl

@@ -150,6 +150,10 @@ class Security {
   /// Annotations to set on the ServiceAccount created for Restate
   serviceAccountAnnotations: Mapping<String, String>?
 
+  /// Optional list of Secrets containing trusted CA certificates.
+  /// Each cert is appended to the system CA bundle via an init container.
+  trustedCaCerts: Listing<TrustedCACert>?
+
   /// Annotations to set on the Service created for Restate
   serviceAnnotations: Mapping<String, String>?
 }
@@ -202,6 +206,15 @@ class SecretProvider {
   provider: String?
 }
 
+/// A reference to a Secret containing a trusted CA certificate.
+class TrustedCACert {
+  /// Name of the Secret containing the CA certificate
+  secretName: String
+
+  /// Key within the Secret that contains the PEM-encoded certificate
+  key: String
+}
+
 /// Storage configuration
 class Storage {
   /// storageClassName is the name of the StorageClass required by the claim. More info:

crd/restateclusters.yaml

@@ -1470,6 +1470,24 @@ spec:
                     description: Annotations to set on the Service created for Restate
                     nullable: true
                     type: object
+                  trustedCaCerts:
+                    description: |-
+                      Optional list of Secrets containing trusted CA certificates.
+                      Each cert is appended to the system CA bundle via an init container.
+                    items:
+                      properties:
+                        key:
+                          description: Key within the Secret that contains the PEM-encoded certificate
+                          type: string
+                        secretName:
+                          description: Name of the Secret containing the CA certificate
+                          type: string
+                      required:
+                      - key
+                      - secretName
+                      type: object
+                    nullable: true
+                    type: array
                 type: object
               storage:
                 description: Storage configuration

release-notes/unreleased/trusted-ca-certs.md

@@ -0,0 +1,15 @@
+## Trusted CA Certificates
+
+You can now configure custom trusted CA certificates for RestateCluster via `spec.security.trustedCaCerts`.
+This is useful when Restate needs to trust internal CAs, for example when accessing an object store with a private certificate authority.
+
+The operator adds an init container that concatenates the system CA bundle with your custom certificates into a single PEM file,
+and sets `SSL_CERT_FILE` on the Restate container to point to the combined bundle. Changing the Secret references (name or key) triggers a pod rollout.
+
+```yaml
+spec:
+  security:
+    trustedCaCerts:
+      - secretName: internal-ca
+        key: ca.pem
+```