feat(ca-bundle): inject org CA into Ray head + worker pods via ConfigMap (#15) (#16)

oren-openteams · web-flow · commit a5dcb886993c · 2026-06-19T10:25:55.000-04:00
* feat(ca-bundle): inject org CA into Ray head + worker pods (#15) Add a `singleuser.gallery`-style opt-in for organisation-supplied CA bundles. When `orgCABundle.secretName` is set, the chart renders: - An initContainer (default image `alpine:3.20`) that concatenates the base image's system trust store with the deployer-supplied CA into a shared emptyDir at /shared/combined-ca.crt - Two volumes on each pod spec: the deployer Secret (read-only) and the shared emptyDir - SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE env vars on both head and worker pointing at the combined bundle, alongside any existing head.containerEnv / worker.containerEnv entries - A read-only volumeMount of the combined bundle on each main container When `secretName` is empty (default), zero CA-related output is rendered — no initContainer, no volumes, no env vars. Backward- compatible. Implemented as five named helpers in _helpers.tpl (.enabled, .initContainers, .volumes, .volumeMounts, .env) so head and worker share one source of truth, and the rayservice template gates each block on the same `if .secretName` predicate. Covers everything that honors SSL_CERT_FILE: requests, urllib3, curl, git, pip, torch.hub, stdlib urllib, properly-constructed httpx clients. Does NOT cover httpx.Client(verify=True) defaults, which hardcode cafile=certifi.where() and ignore env vars — caller code needs to pass verify=ssl.create_default_context() explicitly. README calls this out as a known coverage caveat. Closes #15 * Add GIT_SSL_CAINFO env var and document ArgoCD ignoreDifferences footgun git's libcurl ignores SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE and reads only GIT_SSL_CAINFO, so `pip install git+https://...` and other git-over-HTTPS calls failed certificate verification in worker pods even with the CA bundle mounted. Set GIT_SSL_CAINFO to the combined bundle too. Also document the ArgoCD footgun: the example sync policy's broad /spec/rayClusterConfig ignore rule combined with RespectIgnoreDifferences makes ArgoCD silently skip the CA injection (which lives under that path) while still reporting a healthy sync. See #17.
diff --git a/README.md b/README.md
@@ -123,6 +123,7 @@ spec:
 - `managedNamespaceMetadata` with `nebari.dev/managed: "true"` is required for the nebari-operator to manage NebariApp resources
 - `redirectURI` must be `/oauth2/callback` (Envoy Gateway rejects `/`)
 - Set `serve.enabled: false` to keep the serve endpoint internal-only (recommended — notebooks access it via cluster DNS)
+- The `/spec/rayClusterConfig` ignore rule combined with `RespectIgnoreDifferences=true` makes ArgoCD stop managing everything under `rayClusterConfig`. If you enable [`orgCABundle`](#organization-ca-bundle-injection), the CA injection lives under that path and will **silently not apply** — see the ArgoCD footgun warning in that section before turning it on.
 
 ## Connecting from Jupyter
 
@@ -225,6 +226,50 @@ Key values in `chart/values.yaml`:
 |-------|---------|-------------|
 | `serveApplications` | `[]` | Declarative Serve applications (see [Ray Serve config](https://docs.ray.io/en/latest/serve/production-guide/config.html)) |
 
+### Organization CA Bundle Injection
+
+For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.configMapName` at a ConfigMap containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` / `GIT_SSL_CAINFO` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs.
+
+`GIT_SSL_CAINFO` is set in addition to the three OpenSSL env vars because git's libcurl ignores `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` and reads only `GIT_SSL_CAINFO`. Without it, `pip install git+https://...` and other git-over-HTTPS operations in worker contexts fail certificate verification even though plain `requests`/`pip` calls succeed.
+
+| Value | Default | Description |
+|-------|---------|-------------|
+| `orgCABundle.configMapName` | `""` | Name of a ConfigMap with key `ca.crt` containing the org CA (PEM). Empty disables injection — no behavior change. |
+| `orgCABundle.initImage` | `alpine:3.20` | Image used by the bundle-building initContainer. Only needs `sh` + `cat`. |
+
+```yaml
+# Create the ConfigMap out-of-band (gitops, kubectl, etc.):
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: org-ca-bundle
+data:
+  ca.crt: |
+    -----BEGIN CERTIFICATE-----
+    ...your org CA...
+    -----END CERTIFICATE-----
+---
+# Then point the chart at it:
+orgCABundle:
+  configMapName: org-ca-bundle
+```
+
+> **Why ConfigMap rather than Secret?** A CA certificate is public material by design — the PKI trust model relies on root CAs being widely distributed. Kubernetes itself uses a ConfigMap for the cluster's own CA distribution (`kube-root-ca.crt`, auto-projected into every namespace), and cert-manager's trust-manager subproject does the same. Use a ConfigMap here; reserve Secret for things that actually need confidentiality (private keys, OAuth client secrets, etc.).
+
+> **⚠️ ArgoCD footgun — the CA bundle silently won't apply.** The ArgoCD `Application` shown under [On a Nebari cluster (via ArgoCD)](#on-a-nebari-cluster-via-argocd) sets `RespectIgnoreDifferences=true` together with an `ignoreDifferences` rule on `/spec/rayClusterConfig`. With server-side apply, that combination tells ArgoCD to **stop managing every field under `rayClusterConfig`** — which is exactly where this chart injects the initContainer, volumes, volumeMounts, and CA env vars for the head and worker pods. The result is a silent failure: ArgoCD reports a healthy, fully-synced `Application`, but the running RayService never gets the CA bundle, and TLS calls keep failing with `CERTIFICATE_VERIFY_FAILED`.
+>
+> If you enable `orgCABundle` on a cluster managed by ArgoCD with the example sync policy, you must **narrow the ignore rule** so the CA fields are still reconciled. The broad `/spec/rayClusterConfig` ignore exists only to suppress the autoscaler/runtime mutations KubeRay makes; replace it with targeted pointers (or drop it and ignore only the specific subpaths KubeRay rewrites). After changing it, confirm the head and worker pods actually carry `SSL_CERT_FILE` (`kubectl exec ... -- printenv SSL_CERT_FILE`) rather than trusting the ArgoCD sync status. See [#17](https://github.com/nebari-dev/nebari-rayserve-pack/issues/17) for details.
+
+**Coverage caveat — httpx default `verify=True`:** httpx hardcodes its SSL context to `cafile=certifi.where()`, which means it **ignores** `SSL_CERT_FILE`. Application code making httpx calls that need to traverse a TLS-inspecting proxy must construct an explicit context:
+
+```python
+import ssl, httpx
+client = httpx.Client(verify=ssl.create_default_context())
+# or per-call: httpx.get(url, verify=ssl.create_default_context())
+```
+
+`ssl.create_default_context()` with no `cafile=` honors `SSL_CERT_FILE` / `SSL_CERT_DIR` per the standard OpenSSL convention, so it picks up the bundle this chart injected. Other Python HTTP clients (`requests`, `urllib3`, stdlib `urllib`) and most non-Python TLS tooling honor the env vars automatically.
+
 ## Architecture
 
 ```mermaid
diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl
@@ -62,6 +62,89 @@ Ray serve service name - RayService creates a service named <rayservice-name>-se
 {{- printf "%s-serve-svc" (include "nebari-rayserve.fullname" .) }}
 {{- end }}
 
+{{/*
+Whether organization CA bundle injection is enabled.
+*/}}
+{{- define "nebari-rayserve.orgCABundle.enabled" -}}
+{{- if and .Values.orgCABundle .Values.orgCABundle.configMapName -}}
+true
+{{- end }}
+{{- end }}
+
+{{/*
+initContainers block for the combined-CA bundle build step. Renders empty
+when orgCABundle injection is disabled. Used by both head and worker pod
+specs so the SSL_CERT_FILE bundle exists before the main container starts.
+*/}}
+{{- define "nebari-rayserve.orgCABundle.initContainers" -}}
+{{- if include "nebari-rayserve.orgCABundle.enabled" . -}}
+- name: build-ca-bundle
+  image: {{ .Values.orgCABundle.initImage | quote }}
+  command:
+    - sh
+    - -c
+    - |
+      cat /etc/ssl/certs/ca-certificates.crt \
+          /var/local/org-ca/ca.crt > /shared/combined-ca.crt
+  volumeMounts:
+    - name: org-ca
+      mountPath: /var/local/org-ca
+      readOnly: true
+    - name: combined-ca
+      mountPath: /shared
+{{- end }}
+{{- end }}
+
+{{/*
+Volumes block for the org-ca ConfigMap (deployer-supplied) + the shared
+emptyDir that the initContainer writes the combined bundle into. Renders
+empty when orgCABundle injection is disabled.
+*/}}
+{{- define "nebari-rayserve.orgCABundle.volumes" -}}
+{{- if include "nebari-rayserve.orgCABundle.enabled" . -}}
+- name: org-ca
+  configMap:
+    name: {{ .Values.orgCABundle.configMapName | quote }}
+- name: combined-ca
+  emptyDir: {}
+{{- end }}
+{{- end }}
+
+{{/*
+Container volumeMounts for the combined-CA bundle (read-only). The main
+Ray container mounts only the combined-ca volume — it never sees the raw
+org-ca ConfigMap. Renders empty when orgCABundle injection is disabled.
+*/}}
+{{- define "nebari-rayserve.orgCABundle.volumeMounts" -}}
+{{- if include "nebari-rayserve.orgCABundle.enabled" . -}}
+- name: combined-ca
+  mountPath: /shared
+  readOnly: true
+{{- end }}
+{{- end }}
+
+{{/*
+Container env entries pointing the standard OpenSSL trust-store env vars
+at the combined-CA bundle. Anything that honors SSL_CERT_FILE /
+REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE picks it up automatically.
+GIT_SSL_CAINFO is set separately because git's libcurl ignores the three
+above and reads only GIT_SSL_CAINFO — without it `pip install git+https://...`
+and other git-over-HTTPS calls fail certificate verification.
+Renders empty when orgCABundle injection is disabled.
+*/}}
+{{- define "nebari-rayserve.orgCABundle.env" -}}
+{{- if include "nebari-rayserve.orgCABundle.enabled" . -}}
+- name: SSL_CERT_FILE
+  value: /shared/combined-ca.crt
+- name: REQUESTS_CA_BUNDLE
+  value: /shared/combined-ca.crt
+- name: CURL_CA_BUNDLE
+  value: /shared/combined-ca.crt
+- name: GIT_SSL_CAINFO
+  value: /shared/combined-ca.crt
+{{- end }}
+{{- end }}
+
 {{/*
 Tolerations for a Ray group (head or worker). Pass the group config
 (.Values.head or .Values.worker).
diff --git a/chart/templates/rayservice.yaml b/chart/templates/rayservice.yaml
@@ -27,17 +27,30 @@ spec:
           {{- with .Values.head.runtimeClassName }}
           runtimeClassName: {{ . }}
           {{- end }}
+          {{- with (include "nebari-rayserve.orgCABundle.initContainers" .) }}
+          initContainers:
+            {{- . | nindent 12 }}
+          {{- end }}
+          {{- with (include "nebari-rayserve.orgCABundle.volumes" .) }}
+          volumes:
+            {{- . | nindent 12 }}
+          {{- end }}
           {{- with (include "nebari-rayserve.tolerations" .Values.head) }}
           tolerations:
             {{- . | nindent 12 }}
           {{- end }}
           containers:
             - name: ray-head
               image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
-              {{- with .Values.head.containerEnv }}
+              {{- $headEnv := concat (.Values.head.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }}
+              {{- with $headEnv }}
               env:
                 {{- toYaml . | nindent 16 }}
               {{- end }}
+              {{- with (include "nebari-rayserve.orgCABundle.volumeMounts" .) }}
+              volumeMounts:
+                {{- . | nindent 16 }}
+              {{- end }}
               resources:
                 {{- toYaml .Values.head.resources | nindent 16 }}
               ports:
@@ -68,17 +81,30 @@ spec:
             {{- with .Values.worker.runtimeClassName }}
             runtimeClassName: {{ . }}
             {{- end }}
+            {{- with (include "nebari-rayserve.orgCABundle.initContainers" .) }}
+            initContainers:
+              {{- . | nindent 14 }}
+            {{- end }}
+            {{- with (include "nebari-rayserve.orgCABundle.volumes" .) }}
+            volumes:
+              {{- . | nindent 14 }}
+            {{- end }}
             {{- with (include "nebari-rayserve.tolerations" .Values.worker) }}
             tolerations:
               {{- . | nindent 14 }}
             {{- end }}
             containers:
               - name: ray-worker
                 image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
-                {{- with .Values.worker.containerEnv }}
+                {{- $workerEnv := concat (.Values.worker.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }}
+                {{- with $workerEnv }}
                 env:
                   {{- toYaml . | nindent 18 }}
                 {{- end }}
+                {{- with (include "nebari-rayserve.orgCABundle.volumeMounts" .) }}
+                volumeMounts:
+                  {{- . | nindent 18 }}
+                {{- end }}
                 resources:
                   {{- toYaml .Values.worker.resources | nindent 18 }}
                 {{- with .Values.worker.readinessProbe }}
diff --git a/chart/values.yaml b/chart/values.yaml
@@ -72,6 +72,56 @@ serve:
 #           num_replicas: 2
 serveApplications: []
 
+# =============================================================================
+# Organization CA Bundle Injection
+# =============================================================================
+# Mount an organization-managed CA bundle into the Ray head and worker pods
+# so outbound HTTPS calls (model registries, dataset URLs, Hugging Face hub,
+# S3, etc.) succeed in environments running a TLS-inspecting proxy
+# (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.).
+#
+# When `configMapName` is set, the chart adds:
+#   - An initContainer that concatenates the base image's system trust store
+#     (/etc/ssl/certs/ca-certificates.crt) with the deployer-supplied CA
+#     (mounted from the ConfigMap at the key `ca.crt`) into
+#     /shared/combined-ca.crt
+#   - A shared emptyDir volume that the main container mounts read-only
+#   - SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE env vars on both
+#     head and worker pointing at /shared/combined-ca.crt
+#
+# When `configMapName` is empty (default), nothing is rendered — the
+# resulting RayService is byte-identical to the no-CA-injection case.
+#
+# Why ConfigMap rather than Secret: a CA certificate is public material
+# by design (the entire PKI trust model relies on root CAs being widely
+# distributed — Mozilla's bundle ships in every browser/OS, corporate
+# inspecting proxy roots are pushed to every device that traverses them).
+# Kubernetes itself uses a ConfigMap for the cluster's own CA distribution
+# (`kube-root-ca.crt`, auto-projected into every namespace). cert-manager's
+# trust-manager subproject distributes CA bundles as ConfigMaps via its
+# Bundle CR. We follow that precedent here.
+#
+# Known coverage gap: httpx clients constructed with the default
+# `verify=True` hardcode `cafile=certifi.where()` and ignore SSL_CERT_FILE.
+# Application code that needs its httpx calls covered must pass
+# `verify=ssl.create_default_context()` (or equivalent) explicitly.
+#
+# ConfigMap shape (created out-of-band, e.g. by your gitops layer):
+#   apiVersion: v1
+#   kind: ConfigMap
+#   metadata:
+#     name: org-ca-bundle
+#   data:
+#     ca.crt: |
+#       -----BEGIN CERTIFICATE-----
+#       ...your org CA...
+#       -----END CERTIFICATE-----
+orgCABundle:
+  configMapName: ""
+  # Image used by the initContainer that builds the combined CA bundle.
+  # Only needs `sh` and `cat` — defaults to a tiny Alpine image.
+  initImage: "alpine:3.20"
+
 # =============================================================================
 # Ray Image
 # =============================================================================
