Skip to content

Commit 401be77

Browse files
ighosh98ighosh98
authored
Merge pull request GoogleCloudPlatform#4298 from ighosh98/helm-migration
dependency manager module implementation for helm dependencies
2 parents 7db071f + bcbd1d6 commit 401be77

7 files changed

Lines changed: 389 additions & 1 deletion

File tree

‎community/modules/management/dependencies-installer/README.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
2+
Copyright 2025 Google LLC
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
16+
## Requirements
17+
18+
| Name | Version |
19+
|------|---------|
20+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3 |
21+
| <a name="requirement_google"></a> [google](#requirement\_google) | > 5.0 |
22+
| <a name="requirement_helm"></a> [helm](#requirement\_helm) | ~> 2.17 |
23+
24+
## Providers
25+
26+
| Name | Version |
27+
|------|---------|
28+
| <a name="provider_google"></a> [google](#provider\_google) | > 5.0 |
29+
30+
## Modules
31+
32+
| Name | Source | Version |
33+
|------|--------|---------|
34+
| <a name="module_install_gpu_operator"></a> [install\_gpu\_operator](#module\_install\_gpu\_operator) | ./helm_install | n/a |
35+
| <a name="module_install_jobset"></a> [install\_jobset](#module\_install\_jobset) | ./helm_install | n/a |
36+
| <a name="module_install_kueue"></a> [install\_kueue](#module\_install\_kueue) | ./helm_install | n/a |
37+
| <a name="module_install_nvidia_dra_driver"></a> [install\_nvidia\_dra\_driver](#module\_install\_nvidia\_dra\_driver) | ./helm_install | n/a |
38+
39+
## Resources
40+
41+
| Name | Type |
42+
|------|------|
43+
| [google_client_config.default](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_config) | data source |
44+
| [google_container_cluster.gke_cluster](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/container_cluster) | data source |
45+
46+
## Inputs
47+
48+
| Name | Description | Type | Default | Required |
49+
|------|-------------|------|---------|:--------:|
50+
| <a name="input_cluster_id"></a> [cluster\_id](#input\_cluster\_id) | An identifier for the gke cluster resource with format projects/<project\_id>/locations/<region>/clusters/<name>. | `string` | n/a | yes |
51+
| <a name="input_gke_cluster_exists"></a> [gke\_cluster\_exists](#input\_gke\_cluster\_exists) | A static flag that signals to downstream modules that a cluster has been created. Needed by community/modules/scripts/kubernetes-operations. | `bool` | `false` | no |
52+
| <a name="input_gpu_operator"></a> [gpu\_operator](#input\_gpu\_operator) | Install [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html) which uses the [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) to automate the management of all NVIDIA software components needed to provision GPU. | <pre>object({<br/> install = optional(bool, false)<br/> version = optional(string, "v25.3.0")<br/> })</pre> | `{}` | no |
53+
| <a name="input_jobset"></a> [jobset](#input\_jobset) | Install [Jobset](https://github.com/kubernetes-sigs/jobset) which manages a group of K8s [jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) as a unit. | <pre>object({<br/> install = optional(bool, false)<br/> version = optional(string, "v0.7.2")<br/> })</pre> | `{}` | no |
54+
| <a name="input_kueue"></a> [kueue](#input\_kueue) | Install and configure [Kueue](https://kueue.sigs.k8s.io/docs/overview/) workload scheduler. A configuration yaml/template file can be provided with config\_path to be applied right after kueue installation. If a template file provided, its variables can be set to config\_template\_vars. | <pre>object({<br/> install = optional(bool, false)<br/> version = optional(string, "v0.11.4")<br/> config_path = optional(string, null)<br/> config_template_vars = optional(map(any), null)<br/> })</pre> | `{}` | no |
55+
| <a name="input_nvidia_dra_driver"></a> [nvidia\_dra\_driver](#input\_nvidia\_dra\_driver) | Installs [Nvidia DRA driver](https://github.com/NVIDIA/k8s-dra-driver-gpu) which supports Dynamic Resource Allocation for NVIDIA GPUs in Kubernetes | <pre>object({<br/> install = optional(bool, false)<br/> version = optional(string, "v25.3.0-rc.2")<br/> })</pre> | `{}` | no |
56+
| <a name="input_project_id"></a> [project\_id](#input\_project\_id) | The project ID that hosts the gke cluster. | `string` | n/a | yes |
57+
58+
## Outputs
59+
60+
No outputs.
61+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎community/modules/management/dependencies-installer/helm_install/metadata.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2024 "Google LLC"
1+
# Copyright 2025 "Google LLC"
22
#
33
# Licensed under the Apache License, Version 2.0 (the "License");
44
# you may not use this file except in compliance with the License.

‎community/modules/management/dependencies-installer/main.tf‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
locals {
18+
cluster_id_parts = split("/", var.cluster_id)
19+
cluster_name = local.cluster_id_parts[5]
20+
cluster_location = local.cluster_id_parts[3]
21+
project_id = var.project_id != null ? var.project_id : local.cluster_id_parts[1]
22+
23+
install_gpu_operator = try(var.gpu_operator.install, false)
24+
install_nvidia_dra_driver = try(var.nvidia_dra_driver.install, false)
25+
}
26+
27+
data "google_container_cluster" "gke_cluster" {
28+
project = local.project_id
29+
name = local.cluster_name
30+
location = local.cluster_location
31+
}
32+
33+
data "google_client_config" "default" {}
34+
35+
module "install_kueue" {
36+
source = "./helm_install"
37+
depends_on = [var.gke_cluster_exists]
38+
39+
release_name = "kueue"
40+
41+
chart_name = "oci://registry.k8s.io/kueue/charts/kueue"
42+
chart_version = var.kueue.version # Specify your desired Kueue version
43+
44+
create_namespace = true # Helm can also create the namespace
45+
wait = true
46+
timeout = 600 # seconds
47+
}
48+
49+
module "install_jobset" {
50+
source = "./helm_install"
51+
depends_on = [var.gke_cluster_exists, module.install_kueue]
52+
release_name = "jobset-controller" # The release name for your JobSet installation
53+
chart_name = "oci://registry.k8s.io/jobset/charts/jobset" # The Helm repository URL for nvidia charts
54+
chart_version = var.jobset.version
55+
create_namespace = true
56+
namespace = "jobset-system"
57+
}
58+
59+
module "install_nvidia_dra_driver" {
60+
count = local.install_nvidia_dra_driver ? 1 : 0
61+
depends_on = [var.gke_cluster_exists]
62+
source = "./helm_install"
63+
64+
release_name = "nvidia-dra-driver-gpu" # The release name
65+
chart_repository = "https://helm.ngc.nvidia.com/nvidia" # The Helm repository URL for nvidia charts
66+
chart_name = "nvidia-dra-driver-gpu" # The chart name
67+
chart_version = var.nvidia_dra_driver.version # The chart version
68+
namespace = "nvidia-dra-driver-gpu" # The target namespace
69+
create_namespace = true # Equivalent to --create-namespace
70+
71+
# Use the 'values' argument to pass the YAML content
72+
# This corresponds to the -f <(cat <<EOF ... EOF) part
73+
values_yaml = [<<EOF
74+
nvidiaDriverRoot: /home/kubernetes/bin/nvidia
75+
nvidiaCtkPath: /home/kubernetes/bin/nvidia/nvidia-ctk
76+
resources:
77+
gpus:
78+
enabled: false
79+
80+
controller:
81+
affinity:
82+
nodeAffinity:
83+
requiredDuringSchedulingIgnoredDuringExecution:
84+
nodeSelectorTerms:
85+
- matchExpressions:
86+
- key: "nvidia.com/gpu"
87+
operator: "DoesNotExist"
88+
89+
kubeletPlugin:
90+
affinity:
91+
nodeAffinity:
92+
requiredDuringSchedulingIgnoredDuringExecution:
93+
nodeSelectorTerms:
94+
- matchExpressions:
95+
- key: cloud.google.com/gke-accelerator
96+
operator: In
97+
values:
98+
- nvidia-gb200
99+
- key: kubernetes.io/arch
100+
operator: In
101+
values:
102+
- arm64
103+
104+
tolerations:
105+
- key: nvidia.com/gpu
106+
operator: Equal
107+
value: present
108+
effect: NoSchedule
109+
- key: kubernetes.io/arch
110+
operator: Equal
111+
value: arm64
112+
effect: NoSchedule
113+
114+
EOF
115+
]
116+
117+
atomic = true
118+
cleanup_on_fail = true
119+
}
120+
121+
module "install_gpu_operator" {
122+
count = local.install_gpu_operator ? 1 : 0
123+
source = "./helm_install"
124+
chart_repository = "https://helm.ngc.nvidia.com/nvidia"
125+
depends_on = [var.gke_cluster_exists]
126+
127+
namespace = "gpu-operator"
128+
create_namespace = true
129+
130+
release_name = "gpu-operator"
131+
chart_name = "gpu-operator"
132+
chart_version = var.gpu_operator.version
133+
wait = true
134+
135+
# Use the 'values' argument to pass the YAML content
136+
# This corresponds to the -f <(cat <<EOF ... EOF) part
137+
values_yaml = [<<EOF
138+
hostPaths:
139+
driverInstallDir: /home/kubernetes/bin/nvidia
140+
toolkit:
141+
installDir: /home/kubernetes/bin/nvidia
142+
cdi:
143+
enabled: true
144+
default: true
145+
driver:
146+
enabled: false
147+
148+
daemonsets:
149+
tolerations:
150+
- key: nvidia.com/gpu
151+
operator: Equal
152+
value: present
153+
effect: NoSchedule
154+
- key: kubernetes.io/arch
155+
operator: Equal
156+
value: arm64
157+
effect: NoSchedule
158+
159+
node-feature-discovery:
160+
worker:
161+
tolerations:
162+
- key: kubernetes.io/arch
163+
operator: Equal
164+
value: arm64
165+
effect: NoSchedule
166+
- key: "node-role.kubernetes.io/master"
167+
operator: "Equal"
168+
value: ""
169+
effect: "NoSchedule"
170+
- key: "node-role.kubernetes.io/control-plane"
171+
operator: "Equal"
172+
value: ""
173+
effect: "NoSchedule"
174+
- key: nvidia.com/gpu
175+
operator: Exists
176+
effect: NoSchedule
177+
EOF
178+
]
179+
180+
atomic = true
181+
cleanup_on_fail = true
182+
183+
}

‎community/modules/management/dependencies-installer/metadata.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2025 "Google LLC"
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
---
15+
16+
spec:
17+
requirements:
18+
services:
19+
- container.googleapis.com

‎community/modules/management/dependencies-installer/providers.tf‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
provider "helm" {
18+
kubernetes {
19+
host = "https://${data.google_container_cluster.gke_cluster.endpoint}"
20+
token = data.google_client_config.default.access_token
21+
cluster_ca_certificate = base64decode(
22+
data.google_container_cluster.gke_cluster.master_auth[0].cluster_ca_certificate,
23+
)
24+
}
25+
}

‎community/modules/management/dependencies-installer/variables.tf‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "project_id" {
18+
description = "The project ID that hosts the gke cluster."
19+
type = string
20+
}
21+
22+
variable "cluster_id" {
23+
description = "An identifier for the gke cluster resource with format projects/<project_id>/locations/<region>/clusters/<name>."
24+
type = string
25+
nullable = false
26+
}
27+
28+
variable "gke_cluster_exists" {
29+
description = "A static flag that signals to downstream modules that a cluster has been created. Needed by community/modules/scripts/kubernetes-operations."
30+
type = bool
31+
default = false
32+
}
33+
34+
variable "gpu_operator" {
35+
description = "Install [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html) which uses the [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) to automate the management of all NVIDIA software components needed to provision GPU."
36+
type = object({
37+
install = optional(bool, false)
38+
version = optional(string, "v25.3.0")
39+
})
40+
default = {}
41+
}
42+
43+
variable "nvidia_dra_driver" {
44+
description = "Installs [Nvidia DRA driver](https://github.com/NVIDIA/k8s-dra-driver-gpu) which supports Dynamic Resource Allocation for NVIDIA GPUs in Kubernetes"
45+
type = object({
46+
install = optional(bool, false)
47+
version = optional(string, "v25.3.0-rc.2")
48+
})
49+
default = {}
50+
}
51+
52+
variable "kueue" {
53+
description = "Install and configure [Kueue](https://kueue.sigs.k8s.io/docs/overview/) workload scheduler. A configuration yaml/template file can be provided with config_path to be applied right after kueue installation. If a template file provided, its variables can be set to config_template_vars."
54+
type = object({
55+
install = optional(bool, false)
56+
version = optional(string, "v0.11.4")
57+
config_path = optional(string, null)
58+
config_template_vars = optional(map(any), null)
59+
})
60+
default = {}
61+
}
62+
63+
variable "jobset" {
64+
description = "Install [Jobset](https://github.com/kubernetes-sigs/jobset) which manages a group of K8s [jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) as a unit."
65+
type = object({
66+
install = optional(bool, false)
67+
version = optional(string, "v0.7.2")
68+
})
69+
default = {}
70+
}

‎community/modules/management/dependencies-installer/versions.tf‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
terraform {
18+
required_providers {
19+
google = {
20+
source = "hashicorp/google"
21+
version = "> 5.0"
22+
}
23+
helm = {
24+
source = "hashicorp/helm"
25+
version = "~> 2.17"
26+
}
27+
}
28+
29+
required_version = ">= 1.3"
30+
}

0 commit comments

Comments
 (0)