Skip to content

[Feat][Helm] Support customizable resources and init containers for RayCluster #992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TheCodingSheikh wants to merge 1 commit into
base: main
Choose a base branch
from
+537 −1
Open

[Feat][Helm] Support customizable resources and init containers for RayCluster #992

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion helm/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.11
version: 0.1.12

maintainers:
- name: apostac
Expand Down
22 changes: 22 additions & 0 deletions helm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,28 @@ This table documents all available configuration values for the Production Stack
| `servingEngineSpec.modelSpec[].keda.advanced.scalingModifiers.metricType` | string | `"AverageValue"` | Metric type (AverageValue or Value) |
| `servingEngineSpec.modelSpec[].keda.advanced.scalingModifiers.formula` | string | - | Formula to compose metrics together |

#### Ray Cluster Configuration

Set `servingEngineSpec.modelSpec[].raySpec.enabled: true` to deploy the model as a multi-node `RayCluster` (via KubeRay) instead of a standard `Deployment`. Worker resources and the worker init container are taken from the top-level `modelSpec` (`requestCPU`/`requestMemory`/`requestGPU` or `resources`, and `initContainer`), while head-node resources and the head-node init container are configured under `raySpec.headNode`.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `servingEngineSpec.modelSpec[].raySpec.enabled` | boolean | `false` | Deploy the model as a `RayCluster` instead of a `Deployment` |
| `servingEngineSpec.modelSpec[].raySpec.headNode.requestCPU` | integer | `1` | CPU request for the Ray head node container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.requestMemory` | string | `"1Gi"` | Memory request for the Ray head node container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.requestGPU` | integer | `1` | GPU request for the Ray head node container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.resources` | map | `{}` | (Optional) Raw Kubernetes `resources` block for the head container. When set, overrides the `request*` fields above |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.name` | string | - | (Optional) Name of an init container to run before the Ray head container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.image` | string | - | (Optional) Image for the head init container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.command` | list | `[]` | (Optional) Command for the head init container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.args` | list | `[]` | (Optional) Args for the head init container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.env` | list | `[]` | (Optional) Environment variables for the head init container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.resources` | map | `{}` | (Optional) Resource requests/limits for the head init container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.mountPvcStorage` | boolean | `false` | (Optional) Mount the model's PVC into the head init container |
| `servingEngineSpec.modelSpec[].raySpec.headNode.initContainer.extraVolumeMounts` | list | `[]` | (Optional) Additional volume mounts for the head init container |

> **Note**: Ray worker pods reuse the top-level `modelSpec` fields — set `modelSpec.requestCPU`/`requestMemory`/`requestGPU` (or `modelSpec.resources` for a raw override) to size the workers, and `modelSpec.initContainer` to inject an init container into each worker pod.

#### Serving Engine Monitoring Configuration

| Field | Type | Default | Description |
Expand Down
98 changes: 98 additions & 0 deletions helm/templates/ray-cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,51 @@ spec:
securityContext:
{{- toYaml . | nindent 10 }}
{{- end }}
{{- if and (hasKey $modelSpec "raySpec") (hasKey $modelSpec.raySpec "headNode") (hasKey $modelSpec.raySpec.headNode "initContainer") }}

@gemini-code-assist gemini-code-assist Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If initContainer is defined but lacks a name or image (e.g., if it is defined as an empty map {}), Helm will render invalid Kubernetes YAML because name and image are required fields for containers. We should defensively check that both name and image are specified before rendering the initContainers block.

        {{- if and (hasKey $modelSpec "raySpec") (hasKey $modelSpec.raySpec "headNode") (hasKey $modelSpec.raySpec.headNode "initContainer") $modelSpec.raySpec.headNode.initContainer.name $modelSpec.raySpec.headNode.initContainer.image }}

{{- $container := $modelSpec.raySpec.headNode.initContainer }}
initContainers:
- name: {{ $container.name }}
image: {{ $container.image }}
{{- if $container.command }}
command: {{ toYaml $container.command | nindent 14 }}
{{- end }}
{{- if $container.args }}
args: {{ toYaml $container.args | nindent 14 }}
{{- end }}
{{- if $modelSpec.envFromSecret }}

@gemini-code-assist gemini-code-assist Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If envFromSecret is defined but does not contain a name key, rendering will produce an empty name field under secretRef, which is invalid. We should defensively check that envFromSecret.name is defined.

            {{- if and $modelSpec.envFromSecret $modelSpec.envFromSecret.name }}

envFrom:
- secretRef:
name: {{ $modelSpec.envFromSecret.name }}
{{- end }}
env:
- name: RELEASE_NAME
value: {{ .Release.Name }}
{{- if and .Values.cacheserverSpec.enabled .Values.cacheserverSpec.servicePort }}
- name: LMCACHE_SERVER_SERVICE_PORT
value: {{ .Values.cacheserverSpec.servicePort | quote }}
{{- end }}
{{- if $container.env }}
{{- toYaml $container.env | nindent 14 }}
{{- end }}
{{- if $container.resources }}
resources: {{ toYaml $container.resources | nindent 14 }}
{{- end }}
{{- $pvcMountEnabled := and (hasKey $container "mountPvcStorage") $container.mountPvcStorage (hasKey $modelSpec "pvcStorage") }}

@gemini-code-assist gemini-code-assist Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

We can simplify the $pvcMountEnabled calculation by directly checking the truthiness of the values instead of using hasKey multiple times. In Go templates, missing keys evaluate to nil (falsy), so this is completely safe and much cleaner.

            {{- $pvcMountEnabled := and $container.mountPvcStorage $modelSpec.pvcStorage }}

{{- if or $pvcMountEnabled $container.extraVolumeMounts }}
volumeMounts:
{{- if $pvcMountEnabled }}
- name: {{ .Release.Name }}-storage
mountPath: /data
{{- end }}
{{- with $container.extraVolumeMounts }}
{{- toYaml . | nindent 14 }}
{{- end }}
{{- end }}
{{- if .Values.servingEngineSpec.containerSecurityContext }}
securityContext:
{{- toYaml .Values.servingEngineSpec.containerSecurityContext | nindent 14 }}
{{- end }}
{{- end }}
containers:
- name: vllm-ray-head
image: "{{ required "Required value 'modelSpec.repository' must be defined !" $modelSpec.repository }}:{{ required "Required value 'modelSpec.tag' must be defined !" $modelSpec.tag }}"
Expand Down Expand Up @@ -175,7 +220,11 @@ spec:
livenessProbe:
exec:
command: ["/bin/bash", "-c", "echo TBD"]
{{- if $modelSpec.raySpec.headNode.resources }}
resources: {{ toYaml $modelSpec.raySpec.headNode.resources | nindent 14 }}
{{- else }}
resources: {{- include "chart.resources" $modelSpec.raySpec.headNode | nindent 14 }}
{{- end }}
startupProbe:
exec:
command: ["/bin/bash", "-c", "python3 /scripts/wait_for_ray.py"]
Expand Down Expand Up @@ -302,6 +351,51 @@ spec:
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if hasKey $modelSpec "initContainer" }}

@gemini-code-assist gemini-code-assist Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similarly to the head node, if the worker's initContainer is defined but lacks a name or image, it will render invalid Kubernetes YAML. We should defensively check that both name and image are specified before rendering the initContainers block.

          {{- if and (hasKey $modelSpec "initContainer") $modelSpec.initContainer.name $modelSpec.initContainer.image }}

{{- $container := $modelSpec.initContainer }}
initContainers:
- name: {{ $container.name }}
image: {{ $container.image }}
{{- if $container.command }}
command: {{ toYaml $container.command | nindent 16 }}
{{- end }}
{{- if $container.args }}
args: {{ toYaml $container.args | nindent 16 }}
{{- end }}
{{- if $modelSpec.envFromSecret }}

@gemini-code-assist gemini-code-assist Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensively check that envFromSecret.name is defined before rendering the envFrom block for the worker node's initContainer.

              {{- if and $modelSpec.envFromSecret $modelSpec.envFromSecret.name }}

envFrom:
- secretRef:
name: {{ $modelSpec.envFromSecret.name }}
{{- end }}
env:
- name: RELEASE_NAME
value: {{ .Release.Name }}
{{- if and .Values.cacheserverSpec.enabled .Values.cacheserverSpec.servicePort }}
- name: LMCACHE_SERVER_SERVICE_PORT
value: {{ .Values.cacheserverSpec.servicePort | quote }}
{{- end }}
{{- if $container.env }}
{{- toYaml $container.env | nindent 16 }}
{{- end }}
{{- if $container.resources }}
resources: {{ toYaml $container.resources | nindent 16 }}
{{- end }}
{{- $pvcMountEnabled := and (hasKey $container "mountPvcStorage") $container.mountPvcStorage (hasKey $modelSpec "pvcStorage") }}

@gemini-code-assist gemini-code-assist Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Simplify the $pvcMountEnabled calculation for the worker node's initContainer by directly checking the truthiness of the values.

              {{- $pvcMountEnabled := and $container.mountPvcStorage $modelSpec.pvcStorage }}

{{- if or $pvcMountEnabled $container.extraVolumeMounts }}
volumeMounts:
{{- if $pvcMountEnabled }}
- name: {{ .Release.Name }}-storage
mountPath: /data
{{- end }}
{{- with $container.extraVolumeMounts }}
{{- toYaml . | nindent 16 }}
{{- end }}
{{- end }}
{{- if .Values.servingEngineSpec.containerSecurityContext }}
securityContext:
{{- toYaml .Values.servingEngineSpec.containerSecurityContext | nindent 16 }}
{{- end }}
{{- end }}
containers:
- name: vllm-ray-worker
image: "{{ required "Required value 'modelSpec.repository' must be defined !" $modelSpec.repository }}:{{ required "Required value 'modelSpec.tag' must be defined !" $modelSpec.tag }}"
Expand Down Expand Up @@ -421,7 +515,11 @@ spec:
livenessProbe:
exec:
command: ["/bin/bash", "-c", "echo TBD"]
{{- if $modelSpec.resources }}
resources: {{ toYaml $modelSpec.resources | nindent 16 }}
{{- else }}
resources: {{- include "chart.resources" $modelSpec | nindent 16 }}
{{- end }}
{{- if or (hasKey $modelSpec "pvcStorage") (and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize")) (hasKey $modelSpec "chatTemplate") (hasKey $modelSpec "extraVolumeMounts") $modelSpec.extraVolumeMounts }}
volumeMounts:
{{- end }}
Expand Down
Loading