Skip to content

Commit 7ec2a95

Browse files
authored
feat: add creation of Gateway CR after post-hook (opendatahub-io#68)
* feat: add creation of Gateway CR after post-hook - use the hardcoded GW CR name - use a flag in values.yaml , default to true - check on GWC CR, GW CRD, before create it Signed-off-by: Wen Zhou <wenzhou@redhat.com> * fix: lint Signed-off-by: Wen Zhou <wenzhou@redhat.com> --------- Signed-off-by: Wen Zhou <wenzhou@redhat.com>
1 parent b016061 commit 7ec2a95

10 files changed

Lines changed: 1221 additions & 4 deletions

charts/rhai-on-xks-chart/README.md

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,12 +79,33 @@ helm upgrade rhaii ./charts/rhai-on-xks-chart/ \
7979
8080
## How It Works
8181

82-
The chart performs a **two-phase installation**:
82+
The chart performs a **multi-phase installation**:
8383

8484
1. **Phase 1 — Helm install:** deploys all operator resources (Deployments, RBAC, CRDs, etc.)
85-
2. **Phase 2 — Post-install hook:** a Helm hook Job runs after install/upgrade to create the Custom Resources that configure the operators
85+
2. **Phase 2 — Post-install hook (weight 1):** a Helm hook Job creates the Custom Resources (Kserve CR, KubernetesEngine CR) that configure the operators
86+
3. **Phase 3 — Post-install hook (weight 2):** a Helm hook Job waits for dependencies (Gateway API CRDs, cert-manager CA secret, GatewayClass `istio`) and then creates the `inference-gateway` Gateway CR along with its supporting ConfigMaps
8687

87-
This two-phase approach is necessary because the CRs depend on CRDs that are only available after the operators are deployed.
88+
Phase 2 and 3 are necessary because the CRs depend on CRDs and resources that are only available after the operators are deployed and reconciled.
89+
90+
### Inference Gateway
91+
92+
By default (`components.kserve.gateway.create: true`), the chart creates a Gateway CR named `inference-gateway` in the applications namespace. This gateway is required for KServe model inference traffic. The hook:
93+
94+
1. Waits for Gateway API CRDs to be installed (by the cloud manager)
95+
2. Waits for the cert-manager CA secret (`opendatahub-ca`)
96+
3. Creates a CA bundle ConfigMap (`rhaii-ca-bundle`)
97+
4. Creates a gateway config ConfigMap (`inference-gateway-config`) with CA bundle mount for istio-proxy and Azure-specific health probe annotation (Azure only)
98+
5. Waits for the `istio` GatewayClass (created by Sail Operator)
99+
6. Creates the `inference-gateway` Gateway CR
100+
101+
To disable automatic gateway creation:
102+
103+
```yaml
104+
components:
105+
kserve:
106+
gateway:
107+
create: false
108+
```
88109
89110
## Managed Dependencies
90111

charts/rhai-on-xks-chart/api-docs.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Red Hat OpenShift AI Operator Helm chart (non-OLM installation)
2222
| azure.kubernetesEngine.spec.dependencies.sailOperator.configuration | object | `{}` | |
2323
| azure.kubernetesEngine.spec.dependencies.sailOperator.managementPolicy | string | `"Managed"` | |
2424
| components.kserve.enabled | bool | `true` | |
25+
| components.kserve.gateway.create | bool | `true` | |
2526
| components.kserve.spec | object | `{}` | |
2627
| coreweave.cloudManager.image | string | `"quay.io/opendatahub/opendatahub-operator:latest"` | |
2728
| coreweave.cloudManager.imagePullPolicy | string | `"Always"` | |

charts/rhai-on-xks-chart/templates/hooks/post-install-crs-rbac.yaml

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
{{- $createKserve := .Values.components.kserve.enabled }}
44
{{- $createAzureKE := and .Values.azure.enabled .Values.azure.kubernetesEngine.enabled }}
55
{{- $createCoreweaveKE := and .Values.coreweave.enabled .Values.coreweave.kubernetesEngine.enabled }}
6-
{{- if or $createKserve $createAzureKE $createCoreweaveKE }}
6+
{{- $createGateway := and .Values.components.kserve.enabled .Values.components.kserve.gateway.create }}
7+
{{- if or $createKserve $createAzureKE $createCoreweaveKE $createGateway }}
78
apiVersion: v1
89
kind: ServiceAccount
910
metadata:
@@ -46,6 +47,44 @@ rules:
4647
- create
4748
- patch
4849
- update
50+
{{- if $createGateway }}
51+
- apiGroups:
52+
- gateway.networking.k8s.io
53+
resources:
54+
- gateways
55+
verbs:
56+
- get
57+
- create
58+
- patch
59+
- update
60+
- apiGroups:
61+
- gateway.networking.k8s.io
62+
resources:
63+
- gatewayclasses
64+
verbs:
65+
- get
66+
- apiGroups:
67+
- apiextensions.k8s.io
68+
resources:
69+
- customresourcedefinitions
70+
verbs:
71+
- get
72+
- apiGroups:
73+
- ""
74+
resources:
75+
- secrets
76+
verbs:
77+
- get
78+
- apiGroups:
79+
- ""
80+
resources:
81+
- configmaps
82+
verbs:
83+
- get
84+
- create
85+
- patch
86+
- update
87+
{{- end }}
4988
---
5089
apiVersion: rbac.authorization.k8s.io/v1
5190
kind: ClusterRoleBinding
Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
{{- if .Values.enabled }}
2+
{{- if and .Values.components.kserve.enabled .Values.components.kserve.gateway.create }}
3+
{{- $appNs := .Values.rhaiOperator.applicationsNamespace }}
4+
apiVersion: batch/v1
5+
kind: Job
6+
metadata:
7+
name: rhaii-post-create-gateway
8+
namespace: {{ .Release.Namespace }}
9+
labels:
10+
{{- include "rhai-on-xks-chart.labels" . | nindent 4 }}
11+
annotations:
12+
helm.sh/hook: post-install,post-upgrade
13+
helm.sh/hook-weight: "2"
14+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
15+
spec:
16+
backoffLimit: 3
17+
template:
18+
metadata:
19+
labels:
20+
{{- include "rhai-on-xks-chart.labels" . | nindent 8 }}
21+
spec:
22+
restartPolicy: Never
23+
serviceAccountName: rhaii-post-install-hook
24+
{{- include "rhai-on-xks-chart.imagePullSecrets" . | nindent 6 }}
25+
securityContext:
26+
runAsNonRoot: true
27+
seccompProfile:
28+
type: RuntimeDefault
29+
volumes:
30+
- name: tmp
31+
emptyDir: {}
32+
containers:
33+
- name: create-gateway
34+
image: registry.redhat.io/openshift4/ose-cli-rhel9:v4.21.0
35+
resources:
36+
limits:
37+
cpu: 200m
38+
memory: 128Mi
39+
requests:
40+
cpu: 50m
41+
memory: 64Mi
42+
securityContext:
43+
runAsUser: 65534
44+
allowPrivilegeEscalation: false
45+
readOnlyRootFilesystem: true
46+
capabilities:
47+
drop:
48+
- ALL
49+
volumeMounts:
50+
- name: tmp
51+
mountPath: /tmp
52+
command:
53+
- /bin/bash
54+
- -c
55+
- |
56+
set -euo pipefail
57+
TIMEOUT=300
58+
INTERVAL=5
59+
ELAPSED=0
60+
61+
echo "Step 1: Waiting for Gateway API CRDs required by Gateway CR 'inference-gateway'..."
62+
until kubectl get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1; do
63+
if [ "$ELAPSED" -ge "$TIMEOUT" ]; then
64+
echo "ERROR: Timed out waiting for Gateway API CRDs after ${TIMEOUT}s"
65+
exit 1
66+
fi
67+
echo "CRD not yet available, retrying in ${INTERVAL}s... (${ELAPSED}/${TIMEOUT}s)"
68+
sleep "$INTERVAL"
69+
ELAPSED=$((ELAPSED + INTERVAL))
70+
done
71+
echo "Gateway API CRDs are available."
72+
73+
echo "Step 2: Waiting for cert-manager CA secret required by Gateway CR 'inference-gateway'..."
74+
ELAPSED=0
75+
until kubectl get secret opendatahub-ca -n cert-manager >/dev/null 2>&1; do
76+
if [ "$ELAPSED" -ge "$TIMEOUT" ]; then
77+
echo "ERROR: Timed out waiting for CA secret after ${TIMEOUT}s"
78+
exit 1
79+
fi
80+
echo "CA secret not yet available, retrying in ${INTERVAL}s... (${ELAPSED}/${TIMEOUT}s)"
81+
sleep "$INTERVAL"
82+
ELAPSED=$((ELAPSED + INTERVAL))
83+
done
84+
echo "CA secret is available."
85+
86+
echo "Step 3: Creating CA bundle ConfigMap for Gateway CR 'inference-gateway'..."
87+
kubectl get secret opendatahub-ca -n cert-manager \
88+
-o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt
89+
kubectl create configmap rhaii-ca-bundle \
90+
--from-file=ca.crt=/tmp/ca.crt \
91+
-n {{ $appNs }} \
92+
--dry-run=client -o yaml | kubectl apply -f -
93+
echo "CA bundle ConfigMap created."
94+
95+
echo "Step 4: Create ConfigMap used by Gateway CR 'inference-gateway'..."
96+
kubectl apply -f - <<'EOF'
97+
apiVersion: v1
98+
kind: ConfigMap
99+
metadata:
100+
name: inference-gateway-config
101+
namespace: {{ $appNs }}
102+
data:
103+
deployment: |
104+
spec:
105+
template:
106+
spec:
107+
volumes:
108+
- name: rhaii-ca-bundle
109+
configMap:
110+
name: rhaii-ca-bundle
111+
containers:
112+
- name: istio-proxy
113+
volumeMounts:
114+
- name: rhaii-ca-bundle
115+
mountPath: /var/run/secrets/rhaii
116+
readOnly: true
117+
{{- if .Values.azure.enabled }}
118+
service: |
119+
metadata:
120+
annotations:
121+
service.beta.kubernetes.io/port_80_health-probe_protocol: tcp
122+
{{- end }}
123+
EOF
124+
echo "ConfigMap used by Gateway CR 'inference-gateway' created."
125+
126+
echo "Step 5: Waiting for GatewayClass 'istio' required by Gateway CR 'inference-gateway'..."
127+
ELAPSED=0
128+
until kubectl get gatewayclass istio >/dev/null 2>&1; do
129+
if [ "$ELAPSED" -ge "$TIMEOUT" ]; then
130+
echo "ERROR: Timed out waiting for GatewayClass 'istio' after ${TIMEOUT}s"
131+
exit 1
132+
fi
133+
echo "GatewayClass 'istio' not yet available, retrying in ${INTERVAL}s... (${ELAPSED}/${TIMEOUT}s)"
134+
sleep "$INTERVAL"
135+
ELAPSED=$((ELAPSED + INTERVAL))
136+
done
137+
echo "GatewayClass 'istio' is available."
138+
139+
echo "Step 6: Creating Gateway CR 'inference-gateway'..."
140+
kubectl apply -f - <<'EOF'
141+
apiVersion: gateway.networking.k8s.io/v1
142+
kind: Gateway
143+
metadata:
144+
name: inference-gateway
145+
namespace: {{ $appNs }}
146+
spec:
147+
gatewayClassName: istio
148+
listeners:
149+
- name: http
150+
port: 80
151+
protocol: HTTP
152+
allowedRoutes:
153+
namespaces:
154+
from: All
155+
infrastructure:
156+
labels:
157+
serving.kserve.io/gateway: kserve-ingress-gateway
158+
parametersRef:
159+
group: ""
160+
kind: ConfigMap
161+
name: inference-gateway-config
162+
EOF
163+
echo "Gateway CR 'inference-gateway' created successfully."
164+
{{- end }}
165+
{{- end }}

0 commit comments

Comments
 (0)