Merge pull request GoogleCloudPlatform#4740 from rachit-google/h4dblueprint_refactoring

H4d blueprint refactored

Author: rachit-google

examples/README.md

@@ -1049,7 +1049,7 @@ that have local ssd drives deployed.
 Creates a basic auto-scaling Slurm cluster with mostly default settings. The
 blueprint also creates two new VPC networks, one configured for RDMA networking and the other for non-RDMA networking, along with two filestore instances mounted to `/home` and `/apps`. There is an `h4d` partition that uses compute-optimized `h4d-highmem-192-lssd` machine type.
 
-[hpc-slurm-h4d.yaml]: ../examples/hpc-slurm-h4d.yaml
+[hpc-slurm-h4d.yaml]: ../examples/hpc-slurm-h4d/hpc-slurm-h4d.yaml
 
 ### [hpc-slinky.yaml] ![community-badge] ![experimental-badge]
 

examples/hpc-slurm-h4d/README.md

@@ -0,0 +1,58 @@
+# Slurm H4d Blueprints
+
+## Slurm clusters
+For further information on deploying an H4d cluster with Slurm, please
+see:
+
+[Create an RDMA-enabled HPC Slurm cluster with H4D instances](https://cloud.google.com/compute/docs/hpc/create-a-slurm-cluster-h4d)
+
+## Configuration
+
+Before deploying, you need to edit the `hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml` file to match your Google Cloud environment and requirements.
+
+Key variables to update in `vars`:
+
+* `bucket`: Inside `terraform_backend_defaults.configuration`, replace `BUCKET_NAME` with the name of your GCS bucket for Terraform state.
+* `project_id`: Your Google Cloud Project ID.
+* `deployment_name`: A unique name for this deployment (e.g., `slurm-h4d-test`).
+* `region`: The Google Cloud region for deployment (e.g., `us-central1`).
+* `zone`: The Google Cloud zone for deployment (e.g., `us-central1-a`).
+* `h4d_cluster_size`:The number of static nodes in the cluster.
+* `h4d_reservation_name`: Reservation name if using reserved instances  
+
+### Additional ways to provision
+Cluster toolkit also supports DWS Flex-Start, Spot VMs, as well as reservations as ways to provision instances.
+
+[For more information on DWS Flex-Start in Slurm](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/docs/slurm-dws-flex.md)
+[For more information on Spot VMs](https://cloud.google.com/compute/docs/instances/spot)
+
+We provide ways to enable the alternative provisioning models in the
+`hpc-slurm-h4d-deployment.yaml` file.
+
+To make use of these other models, replace `h4d_reservation_name:` in the
+deployment file with the variable of choice below.
+
+`h4d_enable_spot_vm: true` for spot or `h4d_dws_flex_enabled: true` for DWS Flex-Start.
+
+## Deployment Steps
+
+1. Navigate to the root directory of your Cluster Toolkit checkout or the directory containing the `hpc-slurm-h4d` folder.
+2. Run the `./gcluster deploy` command, providing the `hpc-slurm-h4d-deployment.yaml` as the deployment file. The `gcluster` tool will use this file to resolve variables in the `hpc-slurm-h4d.yaml` located in the same directory.
+
+    ```bash
+    ./gcluster deploy -d examples/hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml examples/hpc-slurm-h4d/hpc-slurm-h4d.yaml -w --auto-approve
+    ```
+
+   * `-w`: Use this flag to force an overwrite of any existing deployment artifacts and configurations from a previous run *with the same deployment name*. (e.g., `slurm-h4d`).
+   * `--auto-approve`: Automatically approves the Terraform apply step. Omit this if you want to review the plan first.
+
+    The tool will create a deployment folder (e.g., `slurm-h4d`), generate Terraform files, and then apply them to create the resources in your GCP project.
+
+## Clean Up
+
+```bash
+#!/bin/bash
+./gcluster destroy [DEPLOYMENT_FOLDER] --auto-approve
+```
+
+Replace [DEPLOYMENT_FOLDER] with the appropriate value.

examples/hpc-slurm-h4d/hpc-slurm-h4d-deployment.yaml

@@ -0,0 +1,32 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
+terraform_backend_defaults:
+  type: gcs
+  configuration:
+    # The GCS bucket used for storing terraform state
+    bucket:   # Bucket Name
+
+vars:
+  # --- Required Variables ---
+  project_id:               # GCP Project ID
+  deployment_name:          # Unique name for the deployment
+  region:                   # GCP region for the deployment
+  zone:                     # GCP zone for the deployment
+  h4d_cluster_size:         # Number of Nodes in the Cluster
+  h4d_reservation_name:     # Specify a reservation name if using reserved instances
+
+  # Additional provisioning models (pick only one), can be used to substitute `h4d_reservation_name`:
+  # h4d_dws_flex_enabled: false   # Set to true to use Dynamic Workload Scheduler (DWS) Flex-Start, for more info visit: https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/docs/slurm-dws-flex.md
+  # h4d_enable_spot_vm: false     # Set to true to use Spot VMs for H4D nodes, for more info visit: https://cloud.google.com/compute/docs/instances/spot

examples/hpc-slurm-h4d.yaml examples/hpc-slurm-h4d/hpc-slurm-h4d.yamlexamples/hpc-slurm-h4d.yaml renamed to examples/hpc-slurm-h4d/hpc-slurm-h4d.yaml

@@ -22,10 +22,11 @@ vars:
   instance_image:
     family: slurm-gcp-6-11-hpc-rocky-linux-8
     project: schedmd-slurm-public
+  h4d_cluster_size: 2
 
   #Provisioning model, select one, lack of selection will rely on on-demand capacity
-  h4d_dws_flex_enabled: false
   h4d_reservation_name: ""
+  h4d_dws_flex_enabled: false
   h4d_enable_spot_vm: false
 
 # Documentation for each of the modules used below can be found at
@@ -83,7 +84,7 @@ deployment_groups:
       allow_automatic_updates: false
       bandwidth_tier: gvnic_enabled
       machine_type: h4d-highmem-192-lssd
-      node_count_static: 2
+      node_count_static: $(vars.h4d_cluster_size)
       node_count_dynamic_max: 0
       enable_placement: false
       enable_spot_vm: $(vars.h4d_enable_spot_vm)

tools/validate_configs/validate_configs.sh

@@ -120,7 +120,7 @@ check_background() {
 	fi
 }
 
-CONFIGS=$(find examples/ community/examples/ tools/validate_configs/test_configs/ docs/tutorials/ docs/videos/build-your-own-blueprint/ -name "*.yaml" -type f -not -path 'examples/machine-learning/a3-megagpu-8g/*' -not -path 'examples/machine-learning/a3-ultragpu-8g/*' -not -path 'examples/machine-learning/build-service-images/*' -not -path 'examples/gke-a3-ultragpu/*' -not -path 'examples/hypercompute_clusters/*' -not -path 'examples/gke-consumption-options/*' -not -path 'examples/gke-a4/*' -not -path 'examples/gke-a3-megagpu/*' -not -path 'examples/machine-learning/a4-highgpu-8g/*' -not -path 'examples/machine-learning/a4x-highgpu-4g/*' -not -path 'community/examples/gke-tpu-v6/*' -not -path 'community/examples/xpk-n2-filestore/*' -not -path 'examples/gke-a4x/*' -not -path 'examples/science/af3-slurm/*' -not -path 'examples/gke-h4d/*' -not -path 'community/examples/hpc-slinky/*' -not -path 'examples/gke-g4/*' -not -path 'community/examples/slurm-gke/*')
+CONFIGS=$(find examples/ community/examples/ tools/validate_configs/test_configs/ docs/tutorials/ docs/videos/build-your-own-blueprint/ -name "*.yaml" -type f -not -path 'examples/machine-learning/a3-megagpu-8g/*' -not -path 'examples/machine-learning/a3-ultragpu-8g/*' -not -path 'examples/machine-learning/build-service-images/*' -not -path 'examples/gke-a3-ultragpu/*' -not -path 'examples/hypercompute_clusters/*' -not -path 'examples/gke-consumption-options/*' -not -path 'examples/gke-a4/*' -not -path 'examples/gke-a3-megagpu/*' -not -path 'examples/machine-learning/a4-highgpu-8g/*' -not -path 'examples/machine-learning/a4x-highgpu-4g/*' -not -path 'community/examples/gke-tpu-v6/*' -not -path 'community/examples/xpk-n2-filestore/*' -not -path 'examples/gke-a4x/*' -not -path 'examples/science/af3-slurm/*' -not -path 'examples/gke-h4d/*' -not -path 'community/examples/hpc-slinky/*' -not -path 'examples/gke-g4/*' -not -path 'community/examples/slurm-gke/*' -not -path 'examples/hpc-slurm-h4d/*')
 # Exclude blueprints that use v5 modules.
 declare -A EXCLUDE_EXAMPLE
 EXCLUDE_EXAMPLE["tools/validate_configs/test_configs/two-clusters-sql.yaml"]=