Skip to content

Commit 1fceded

Browse files
kadupoornimakadupoornima
authored
Merge pull request #4560 from kadupoornima/develop-imitation
Add G4 GKE base blueprints
2 parents ffb5f57 + 036ba20 commit 1fceded

5 files changed

Lines changed: 324 additions & 1 deletion

File tree

‎examples/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ md_toc github examples/README.md | sed -e "s/\s-\s/ * /"
6161
* [gke-tpu-v6](#gke-tpu-v6--) ![community-badge] ![experimental-badge]
6262
* [xpk-n2-filestore](#xpk-n2-filestore--) ![community-badge] ![experimental-badge]
6363
* [gke-h4d](#gke-h4d-) ![core-badge]
64+
* [gke-g4](#gke-g4-) ![core-badge]
6465
* [Blueprint Schema](#blueprint-schema)
6566
* [Writing an HPC Blueprint](#writing-an-hpc-blueprint)
6667
* [Blueprint Boilerplate](#blueprint-boilerplate)
@@ -1490,6 +1491,12 @@ This blueprint uses GKE to provision a Kubernetes cluster and a H4D node pool, a
14901491

14911492
[gke-h4d]: ../examples/gke-h4d
14921493

1494+
### [gke-g4] ![core-badge]
1495+
1496+
This blueprint uses GKE to provision a Kubernetes cluster and a G4 node pool, along with networks and service accounts. Information about G4 machines can be found [here](https://cloud.google.com/blog/products/compute/introducing-g4-vm-with-nvidia-rtx-pro-6000). The deployment instructions can be found in the [README](/examples/gke-g4/README.md).
1497+
1498+
[gke-g4]: ../examples/gke-g4
1499+
14931500
## Blueprint Schema
14941501

14951502
Similar documentation can be found on

‎examples/gke-g4/README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# GKE G4 Blueprint
2+
3+
This blueprint uses GKE to provision a Kubernetes cluster and a G4 node pool, along with networks and service accounts. More information about G4 machines can be found here:
4+
5+
* [Blog post](https://cloud.google.com/blog/products/compute/introducing-g4-vm-with-nvidia-rtx-pro-6000)
6+
* [Documentation](https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus)
7+
8+
> **_NOTE:_** The required GKE version for G4 support is >= 1.32.4-gke.1698000.
9+
10+
## Steps to deploy the G4 blueprint
11+
12+
1. Install Cluster Toolkit
13+
1. Install [dependencies](https://cloud.google.com/cluster-toolkit/docs/setup/install-dependencies).
14+
1. Set up [Cluster Toolkit](https://cloud.google.com/cluster-toolkit/docs/setup/configure-environment).
15+
1. Switch to the Cluster Toolkit directory
16+
17+
```sh
18+
cd cluster-toolkit
19+
```
20+
21+
1. Get the IP address for your host machine
22+
23+
```sh
24+
curl ifconfig.me
25+
```
26+
27+
1. Update the vars block of the `gke-g4-deployment.yaml` file.
28+
1. `project_id`: ID of the project where you are deploying the cluster.
29+
1. `deployment_name`: Name of the deployment.
30+
1. `region`: Compute region used for the deployment.
31+
1. `zone`: Compute zone used for the deployment.
32+
1. `machine_type`: The VM shape. See allowed values at https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus.
33+
1. `num_gpus`: Number of GPUS in the VM. Can be found at https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus.
34+
1. `static_node_count`: Number of nodes to create.
35+
1. `authorized_cidr`: update the IP address in `<your-ip-address>/32`.
36+
1. Build the Cluster Toolkit binary
37+
38+
```sh
39+
make
40+
```
41+
42+
1. Provision the GKE cluster
43+
44+
```sh
45+
./gcluster deploy -d examples/gke-g4/gke-g4-deployment.yaml examples/gke-g4/gke-g4.yaml
46+
```
47+
48+
These four options are displayed:
49+
50+
```sh
51+
(D)isplay full proposed changes,
52+
(A)pply proposed changes,
53+
(S)top and exit,
54+
(C)ontinue without applying
55+
```
56+
57+
Type `a` and hit enter to create the cluster.
58+
59+
## Clean Up
60+
To destroy all resources associated with creating the GKE cluster, run the following command:
61+
62+
```sh
63+
./gcluster destroy CLUSTER-NAME
64+
```
65+
66+
Replace `CLUSTER-NAME` with the `deployment_name` used in the blueprint vars block.

‎examples/gke-g4/gke-g4-deployment.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Copyright 2025 "Google LLC"
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
terraform_backend_defaults:
16+
type: gcs
17+
configuration:
18+
# The GCS bucket used for storing terraform state
19+
bucket:
20+
21+
vars:
22+
# Your GCP Project ID
23+
project_id:
24+
25+
# This should be unique across all of your Cluster Toolkit Deployments.
26+
deployment_name:
27+
28+
# The GCP Region used for this deployment.
29+
region:
30+
31+
# The GCP Zone used for this deployment.
32+
zone:
33+
34+
# The VM shape. See allowed values here: https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus
35+
machine_type:
36+
37+
# Number of GPUS in the VM. Can be found here: https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus
38+
num_gpus:
39+
40+
# The number of nodes to be created
41+
static_node_count:
42+
43+
# Cidr block containing the IP of the machine calling terraform.
44+
# The following line must be updated for this example to work.
45+
authorized_cidr:

‎examples/gke-g4/gke-g4.yaml‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# Copyright 2025 "Google LLC"
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
blueprint_name: gke-g4
16+
17+
vars:
18+
# The following variables should be over-written in the deployment.yaml file.
19+
# Your GCP Project ID
20+
project_id:
21+
22+
# This should be unique across all of your Cluster Toolkit Deployments.
23+
deployment_name: gke-g4
24+
25+
# The GCP Region used for this deployment.
26+
region:
27+
28+
# The GCP Zone used for this deployment.
29+
zone:
30+
31+
# The VM shape. See allowed values here: https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus
32+
machine_type:
33+
34+
# Number of GPUS in the VM. Can be found here: https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus
35+
num_gpus:
36+
37+
# The number of nodes to be created.
38+
static_node_count:
39+
40+
# Cidr block containing the IP of the machine calling terraform.
41+
# The following line must be updated for this example to work.
42+
authorized_cidr:
43+
44+
system_node_pool_disk_size_gb: 100
45+
46+
deployment_groups:
47+
- group: primary
48+
modules:
49+
- id: gke-g4-net-0
50+
source: modules/network/vpc
51+
settings:
52+
network_name: $(vars.deployment_name)-net-0
53+
subnetworks:
54+
- subnet_name: $(vars.deployment_name)-sub-0
55+
subnet_region: $(vars.region)
56+
subnet_ip: 192.168.0.0/18
57+
secondary_ranges_list:
58+
- subnetwork_name: $(vars.deployment_name)-sub-0
59+
ranges:
60+
- range_name: pods
61+
ip_cidr_range: 10.4.0.0/14
62+
- range_name: services
63+
ip_cidr_range: 10.0.32.0/20
64+
firewall_rules:
65+
- name: $(vars.deployment_name)-internal-0
66+
ranges: [192.168.0.0/16]
67+
allow:
68+
- protocol: tcp
69+
ports: ["0-65535"]
70+
- protocol: udp
71+
ports: ["0-65535"]
72+
- protocol: icmp
73+
74+
- id: gke-g4-net-1
75+
source: modules/network/vpc
76+
settings:
77+
network_name: $(vars.deployment_name)-net-1
78+
subnetworks:
79+
- subnet_name: $(vars.deployment_name)-sub-1
80+
subnet_region: $(vars.region)
81+
subnet_ip: 192.168.64.0/18
82+
firewall_rules:
83+
- name: $(vars.deployment_name)-internal-1
84+
ranges: [192.168.0.0/16]
85+
allow:
86+
- protocol: tcp
87+
ports: ["0-65535"]
88+
- protocol: udp
89+
ports: ["0-65535"]
90+
- protocol: icmp
91+
92+
- id: node_pool_service_account
93+
source: community/modules/project/service-account
94+
settings:
95+
name: gke-np-sa
96+
project_roles:
97+
- logging.logWriter
98+
- monitoring.metricWriter
99+
- monitoring.viewer
100+
- stackdriver.resourceMetadata.writer
101+
- storage.objectViewer
102+
- artifactregistry.reader
103+
104+
- id: workload_service_account
105+
source: community/modules/project/service-account
106+
settings:
107+
name: gke-wl-sa
108+
project_roles:
109+
- logging.logWriter
110+
- monitoring.metricWriter
111+
- monitoring.viewer
112+
- stackdriver.resourceMetadata.writer
113+
- storage.objectAdmin
114+
- artifactregistry.reader
115+
116+
- id: g4-cluster
117+
source: modules/scheduler/gke-cluster
118+
use: [gke-g4-net-0, workload_service_account]
119+
settings:
120+
system_node_pool_machine_type: "e2-standard-16"
121+
system_node_pool_disk_size_gb: $(vars.system_node_pool_disk_size_gb)
122+
system_node_pool_taints: []
123+
enable_dcgm_monitoring: true
124+
enable_private_endpoint: false # Allows access from authorized public IPs
125+
configure_workload_identity_sa: true
126+
master_authorized_networks:
127+
- cidr_block: $(vars.authorized_cidr) # Allows your machine to run the kubectl command. Required for multi network setup.
128+
display_name: "kubectl-access-network"
129+
additional_networks:
130+
$(concat(
131+
[{
132+
network=gke-g4-net-1.network_name,
133+
subnetwork=gke-g4-net-1.subnetwork_name,
134+
subnetwork_project=vars.project_id,
135+
nic_type="GVNIC",
136+
queue_count=null,
137+
network_ip=null,
138+
stack_type=null,
139+
access_config=[{nat_ip=null, public_ptr_domain_name=null, network_tier=null}],
140+
ipv6_access_config=[],
141+
alias_ip_range=[]
142+
}]
143+
))
144+
# Cluster versions cannot be updated through the toolkit after creation
145+
# Please manage cluster version from the Google Cloud Console directly
146+
version_prefix: "1.32."
147+
release_channel: RAPID
148+
maintenance_exclusions:
149+
- name: no-minor-or-node-upgrades-indefinite
150+
start_time: "2025-08-01T00:00:00Z"
151+
end_time: "2026-04-10T00:00:00Z"
152+
exclusion_scope: NO_MINOR_OR_NODE_UPGRADES
153+
outputs: [instructions]
154+
155+
- id: g4-pool
156+
source: modules/compute/gke-node-pool
157+
use: [g4-cluster, node_pool_service_account]
158+
settings:
159+
machine_type: $(vars.machine_type)
160+
auto_upgrade: true
161+
zones: [$(vars.zone)]
162+
static_node_count: $(vars.static_node_count)
163+
guest_accelerator:
164+
- type: nvidia-rtx-pro-6000
165+
count: $(vars.num_gpus)
166+
gpu_driver_installation_config:
167+
gpu_driver_version: LATEST # TODO: Move to a fixed GPU driver version.
168+
additional_networks:
169+
$(concat(
170+
[{
171+
network=gke-g4-net-1.network_name,
172+
subnetwork=gke-g4-net-1.subnetwork_name,
173+
subnetwork_project=vars.project_id,
174+
nic_type="GVNIC",
175+
queue_count=null,
176+
network_ip=null,
177+
stack_type=null,
178+
access_config=[{nat_ip=null, public_ptr_domain_name=null, network_tier=null}],
179+
ipv6_access_config=[],
180+
alias_ip_range=[]
181+
}]
182+
))
183+
outputs: [instructions]
184+
185+
# Install Kueue and Jobset
186+
- id: workload-manager-install
187+
source: modules/management/kubectl-apply
188+
use: [g4-cluster]
189+
settings:
190+
kueue:
191+
install: true
192+
jobset:
193+
install: true
194+
195+
- id: nvidia_smi_job_template
196+
source: modules/compute/gke-job-template
197+
use: [g4-pool]
198+
settings:
199+
image: nvidia/cuda:13.0.0-base-ubuntu24.04
200+
command:
201+
- nvidia-smi
202+
node_count: $(vars.static_node_count)
203+
name: run-nvidia-smi
204+
k8s_service_account_name: workload-identity-k8s-sa
205+
outputs: [instructions]

‎tools/validate_configs/validate_configs.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ check_background() {
120120
fi
121121
}
122122

123-
CONFIGS=$(find examples/ community/examples/ tools/validate_configs/test_configs/ docs/tutorials/ docs/videos/build-your-own-blueprint/ -name "*.yaml" -type f -not -path 'examples/machine-learning/a3-megagpu-8g/*' -not -path 'examples/machine-learning/a3-ultragpu-8g/*' -not -path 'examples/machine-learning/build-service-images/*' -not -path 'examples/gke-a3-ultragpu/*' -not -path 'examples/hypercompute_clusters/*' -not -path 'examples/gke-consumption-options/*' -not -path 'examples/gke-a4/*' -not -path 'examples/gke-a3-megagpu/*' -not -path 'examples/machine-learning/a4-highgpu-8g/*' -not -path 'examples/machine-learning/a4x-highgpu-4g/*' -not -path 'community/examples/gke-tpu-v6/*' -not -path 'community/examples/xpk-n2-filestore/*' -not -path 'examples/gke-a4x/*' -not -path 'examples/science/af3-slurm/*' -not -path 'examples/gke-h4d/*' -not -path 'community/examples/hpc-slinky/*')
123+
CONFIGS=$(find examples/ community/examples/ tools/validate_configs/test_configs/ docs/tutorials/ docs/videos/build-your-own-blueprint/ -name "*.yaml" -type f -not -path 'examples/machine-learning/a3-megagpu-8g/*' -not -path 'examples/machine-learning/a3-ultragpu-8g/*' -not -path 'examples/machine-learning/build-service-images/*' -not -path 'examples/gke-a3-ultragpu/*' -not -path 'examples/hypercompute_clusters/*' -not -path 'examples/gke-consumption-options/*' -not -path 'examples/gke-a4/*' -not -path 'examples/gke-a3-megagpu/*' -not -path 'examples/machine-learning/a4-highgpu-8g/*' -not -path 'examples/machine-learning/a4x-highgpu-4g/*' -not -path 'community/examples/gke-tpu-v6/*' -not -path 'community/examples/xpk-n2-filestore/*' -not -path 'examples/gke-a4x/*' -not -path 'examples/science/af3-slurm/*' -not -path 'examples/gke-h4d/*' -not -path 'community/examples/hpc-slinky/*' -not -path 'examples/gke-g4/*')
124124
# Exclude blueprints that use v5 modules.
125125
declare -A EXCLUDE_EXAMPLE
126126
EXCLUDE_EXAMPLE["tools/validate_configs/test_configs/two-clusters-sql.yaml"]=

0 commit comments

Comments
 (0)