fix: canary base image

CLAUDE.md

@@ -261,7 +261,7 @@ kubectl delete pod -n restate-operator -l app=restate-operator
 - `image.pullPolicy` - Pull policy (default: `IfNotPresent`)
 - `awsPodIdentityAssociationCluster` - Enables EKS Pod Identity support
 - `gcpWorkloadIdentity` - Enables GCP Workload Identity via Config Connector
-- `canaryImage` - Container image for canary jobs and the trusted CA certs init container (default: `busybox:uclibc`); must provide `cat`, `grep` and `wget`
+- `canaryImage` - Container image for canary jobs and the trusted CA certs init container (default: `alpine:3.21`); must provide `cat`, `grep`, `wget` and a CA bundle at `/etc/ssl/certs/ca-certificates.crt`
 - `operatorNamespace` - Namespace where operator runs
 - `operatorLabelName/Value` - Labels for network policy selectors
 

README.md

@@ -671,24 +671,24 @@ the `RestateCluster` spec.
 
 Both EKS Pod Identity and GCP Workload Identity use a canary job to validate that credentials are available before
 starting the Restate cluster. The trusted CA certs feature also uses this image for its init container.
-By default, this uses the `busybox:uclibc` image from Docker Hub. In environments where
-nodes cannot pull from Docker Hub (e.g. air-gapped or restricted registries), you can override this with the
-`canaryImage` Helm value:
+By default, this uses the `alpine:3.21` image from Docker Hub. The image must include a
+CA certificate bundle at `/etc/ssl/certs/ca-certificates.crt` (required by the trusted CA
+certs init container) and provide `cat`, `grep` and `wget`. In environments where nodes
+cannot pull from Docker Hub (e.g. air-gapped or restricted registries), you can override
+this with the `canaryImage` Helm value:
 
 ```yaml
-canaryImage: my-private-registry.example.com/busybox:uclibc
+canaryImage: my-private-registry.example.com/alpine:3.21
 ```
 
 The simplest approach is to mirror the default image:
 
 ```bash
-docker pull busybox:uclibc
-docker tag busybox:uclibc my-private-registry.example.com/busybox:uclibc
-docker push my-private-registry.example.com/busybox:uclibc
+docker pull alpine:3.21
+docker tag alpine:3.21 my-private-registry.example.com/alpine:3.21
+docker push my-private-registry.example.com/alpine:3.21
 ```
 
-If using a different base image, it must provide `cat`, `grep` and `wget`.
-
 ### EKS Security Groups for Pods
 
 [EKS Security Groups for Pods](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) allows

charts/restate-operator-helm/values.yaml

@@ -16,7 +16,7 @@ podAnnotations: {}
 awsPodIdentityAssociationCluster: null
 gcpWorkloadIdentity: null
 clusterDns: null # defaults to "cluster.local" in the operator binary
-canaryImage: null # defaults to "busybox:uclibc"; image must provide cat, grep and wget
+canaryImage: null # defaults to "alpine:3.21"; image must provide cat, grep, wget and a CA bundle at /etc/ssl/certs/ca-certificates.crt
 
 podSecurityContext:
   fsGroup: 2000

release-notes/unreleased/fix-canary-image-ca-bundle.md

@@ -0,0 +1,40 @@
+# Release Notes: Fix canary image missing CA bundle
+
+## Bug Fix
+
+### What Changed
+The default `canaryImage` has been changed from `busybox:uclibc` to `alpine:3.21`.
+
+### Why This Matters
+The `trustedCaCerts` feature uses an init container (the canary image) to concatenate
+system CA certificates with custom trusted CAs. The init container reads the system CA
+bundle from `/etc/ssl/certs/ca-certificates.crt`, but `busybox:uclibc` does not ship a
+CA bundle at that path, causing the init container to fail with:
+
+```
+cat: can't open '/etc/ssl/certs/ca-certificates.crt': No such file or directory
+```
+
+This made `trustedCaCerts` non-functional with the default canary image.
+
+### Impact on Users
+- **Existing deployments using `trustedCaCerts`**: Will work after upgrading. If you
+  previously worked around this by setting `canaryImage` to an image with a CA bundle,
+  you can remove that override.
+- **Existing deployments not using `trustedCaCerts`**: No impact. The canary image is
+  also used for Pod Identity and Workload Identity canary jobs, which do not depend on
+  the CA bundle and will continue to work with `alpine:3.21`.
+- **Custom `canaryImage` overrides**: If you use a custom canary image, ensure it
+  includes a CA bundle at `/etc/ssl/certs/ca-certificates.crt` if you plan to use
+  `trustedCaCerts`.
+
+### Migration Guidance
+No action required. The default will change automatically on upgrade.
+
+If you override `canaryImage` in your Helm values and want to use `trustedCaCerts`,
+ensure your image includes a CA certificate bundle:
+
+```yaml
+# Image must have /etc/ssl/certs/ca-certificates.crt and provide cat, grep, wget
+canaryImage: my-registry.example.com/alpine:3.21
+```

src/controllers/restatecluster/reconcilers/compute.rs

@@ -290,8 +290,9 @@ fn env(cluster_name: &str, custom: Option<&[EnvVar]>) -> Vec<EnvVar> {
     }
 }
 
-// Debian/Alpine system CA bundle path. If the Restate server base image changes to a
-// different distro (e.g. RHEL uses /etc/pki/tls/certs/ca-bundle.crt), this must be updated.
+// Debian/Alpine system CA bundle path, read from the canary image's filesystem.
+// The default canary image (alpine:3.21) ships this path. If a custom canary image uses
+// a different distro (e.g. RHEL uses /etc/pki/tls/certs/ca-bundle.crt), this must be updated.
 const SYSTEM_CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
 const COMBINED_CA_VOLUME: &str = "combined-ca-certs";
 const COMBINED_CA_MOUNT: &str = "/combined-certs";

src/main.rs

@@ -66,7 +66,7 @@ struct Arguments {
         long = "canary-image",
         env = "CANARY_IMAGE",
         value_name = "IMAGE",
-        default_value = "busybox:uclibc"
+        default_value = "alpine:3.21"
     )]
     canary_image: String,
 }