Merge pull request #4833 from GoogleCloudPlatform/release-candidate

Release candidate: v1.72.0

Author: Neelabh94

cmd/root.go

@@ -53,7 +53,7 @@ HPC deployments on the Google Cloud Platform.`,
 				logging.Fatal("cmd.Help function failed: %s", err)
 			}
 		},
-		Version:     "v1.71.0",
+		Version:     "v1.72.0",
 		Annotations: annotation,
 	}
 )

community/examples/gke-tpu-v6/README.md

@@ -92,10 +92,11 @@ This section guides you through the cluster creation process, ensuring that your
 
 ## Advanced Blueprint: GKE TPU with GCS Integration
 
-This repository also includes an advanced blueprint, `gke-tpu-v6-gcs.yaml`, designed for production-ready workloads. It builds on the basic blueprint by adding several key features:
+This repository also includes an advanced blueprint, `gke-tpu-v6-advanced.yaml`, designed for production-ready workloads. It builds on the basic blueprint by adding several key features:
 * **Dedicated Service Accounts** for nodes and workloads, following security best practices.
 * **Automatic creation of two GCS buckets** for training data and checkpoints.
 * **Performance-tuned GCS FUSE mounts** pre-configured in the cluster as Persistent Volumes.
+* **Optional High-Performance Storage: [Managed Lustre](https://cloud.google.com/managed-lustre/docs/overview)** for high-performance, fully managed parallel file system optimized for heavy AI and HPC workloads. For details of configuring Managed Lustre, please refer to the [appendix](#understanding-managed-lustre-integration)
 
 ### Deploying the Advanced Blueprint
 
@@ -114,14 +115,6 @@ The process is nearly identical to the basic deployment.
 
 1. After deployment, the blueprint will output instructions for running a fio benchmark job. This job serves as a validation test to confirm that the GCS mounts are working correctly for both reading and writing. Follow the printed instructions to run the test.
 
-### Understanding the GCS Integration
-
-The advanced blueprint provisions several key technologies to create a robust data pipeline for your TPU workloads. Here are some resources to understand how they work together:
-* [Cloud Storage Overview](https://cloud.google.com/storage/docs/introduction#quickstarts): Start here to understand what Cloud Storage buckets are and their role in storing large-scale data.
-* [Cloud TPU Storage Options](https://cloud.google.com/tpu/docs/storage-options): Learn about the recommended storage patterns for Cloud TPUs, including why GCS FUSE is a best practice for providing training data.
-* [Access GCS Buckets with the GCS FUSE CSI Driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver): This is the core technical guide explaining how GKE mounts GCS buckets into your pods, which this blueprint automates.
-* [Configure Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity): Read this to understand the secure, recommended method for GKE applications to access Google Cloud services like GCS, which this blueprint configures for you.
-
 ## Run the sample job
 
 The [tpu-available-chips.yaml](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/examples/gke-tpu-v6/tpu-available-chips.yaml) file creates a service and a job resource in kubernetes. It is based on https://cloud.google.com/kubernetes-engine/docs/how-to/tpus#tpu-chips-node-pool. The  workload returns the number of TPU chips across all of the nodes in a multi-host TPU slice.
@@ -180,6 +173,78 @@ To avoid recurring charges for the resources used on this page, clean up the res
    ./gcluster destroy gke-tpu-v6/
    ```
 
-Useful TPU links:
+## Appendix
+
+### Useful TPU links
 1. [TPU architecture](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm)
 2. [TPU v6](https://cloud.google.com/tpu/docs/v6e)
+
+### Understanding the GCS Integration
+
+The advanced blueprint provisions several key technologies to create a robust data pipeline for your TPU workloads. Here are some resources to understand how they work together:
+* [Cloud Storage Overview](https://cloud.google.com/storage/docs/introduction#quickstarts): Start here to understand what Cloud Storage buckets are and their role in storing large-scale data.
+* [Cloud TPU Storage Options](https://cloud.google.com/tpu/docs/storage-options): Learn about the recommended storage patterns for Cloud TPUs, including why GCS FUSE is a best practice for providing training data.
+* [Access GCS Buckets with the GCS FUSE CSI Driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver): This is the core technical guide explaining how GKE mounts GCS buckets into your pods, which this blueprint automates.
+* [Configure Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity): Read this to understand the secure, recommended method for GKE applications to access Google Cloud services like GCS, which this blueprint configures for you.
+
+### Understanding Managed Lustre integration
+The advanced blueprint `gke-tpu-v6-advanced.yaml` can also be configured to deploy a Managed Lustre filesystem. Google Cloud **Managed Lustre** delivers a high-performance, fully managed parallel file system optimized for AI and HPC applications. With multi-petabyte-scale capacity and up to 1 TBps throughput, [Managed Lustre](https://cloud.google.com/architecture/optimize-ai-ml-workloads-managed-lustre) facilitates the migration of demanding workloads to the cloud.
+
+#### Enabling Managed Lustre
+To enable Managed Lustre, you must make these changes before deploying:
+
+1. In the `gke-tpu-v6-advanced.yaml`:
+Find the **vars:** section and **uncomment** the Managed Lustre variables. The defaults provide a high-performance **36000GiB** (~35.16TiB) filesystem with **18 GB/s** of throughput.
+
+2. In the `gke-tpu-v6-advanced.yaml`:
+Find the section commented # --- MANAGED LUSTRE ADDITIONS ---. **Uncomment** the entire block of four modules: `private_service_access`, `lustre_firewall_rule`, `managed-lustre`, and `lustre-pv`.
+
+After making these changes, run the `gcluster deploy` command as usual.
+
+#### Using Managed Lustre in a Pod
+Once deployed, the `Lustre` filesystem is available to the cluster as a `Persistent Volume (PV)`. To use it in your workloads, you need to create a `Persistent Volume Claim (PVC)` and mount it in your pod.
+
+#### Testing the Lustre Mount
+
+1. Create a file named `lustre-claim-pod.yaml`:
+
+    ```yaml
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    metadata:
+      name: my-lustre-claim
+    spec:
+      accessModes:
+      - ReadWriteMany
+      # storageClassName must be empty to bind to the manually created PV
+      storageClassName: ""
+      resources:
+        requests:
+          # This size must match lustre_size_gib from your variables
+          storage: 36000Gi
+    ---
+    apiVersion: v1
+    kind: Pod
+    metadata:
+      name: lustre-test-pod
+    spec:
+      containers:
+      - name: test-container
+        image: ubuntu:22.04
+        command: ["/bin/sleep", "infinity"]
+        volumeMounts:
+        - name: lustre-storage
+          mountPath: /mnt/lustre
+      volumes:
+      - name: lustre-storage
+        persistentVolumeClaim:
+          claimName: my-lustre-claim
+    ```
+
+2. Apply the manifest to your cluster:
+
+    ```yaml
+    kubectl apply -f lustre-claim-pod.yaml
+    ```
+
+The pod will start, and the Managed Lustre filesystem will be available inside the container at `/mnt/lustre`.

community/examples/gke-tpu-v6/gke-tpu-v6-advanced.yaml

@@ -51,14 +51,32 @@ vars:
 
   system_node_pool_disk_size_gb: 200
   v6e_node_pool_disk_size_gb: 100
+  version_prefix: "1.33."
+
+  # # To enable Managed-Lustre please uncomment this section and fill out the settings.
+  # # Additionally, please uncomment the private_service_access, lustre_firewall_rule, managed-lustre and lustre-pv modules.
+  # # Managed Lustre is only supported in specific regions and zones
+  # # Please refer https://cloud.google.com/managed-lustre/docs/locations
+
+  # # Managed-Lustre instance name. This should be unique for each deployment.
+  # lustre_instance_id: $(vars.deployment_name)
+
+  # # The values of size_gib and per_unit_storage_throughput are co-related
+  # # Please refer https://cloud.google.com/managed-lustre/docs/create-instance#performance-tiers
+  # # Storage capacity of the lustre instance in GiB
+  # lustre_size_gib: 36000
+
+  # # Maximum throughput of the lustre instance in MBps per TiB
+  # per_unit_storage_throughput: 500
 
 
 deployment_groups:
 - group: primary
   modules:
-  - id: gke-tpu-v6-net
+  - id: gke-tpu-v6-net-0
     source: modules/network/vpc
     settings:
+      network_name: $(vars.deployment_name)-net-0
       subnetworks:
       - subnet_name: $(vars.deployment_name)-sub-0
         subnet_region: $(vars.region)
@@ -72,7 +90,25 @@ deployment_groups:
           ip_cidr_range: 10.0.32.0/20
       firewall_rules:
       - name: $(vars.deployment_name)-internal-0
-        ranges: [192.168.0.0/18]
+        ranges: [192.168.0.0/16]
+        allow:
+        - protocol: tcp
+          ports: ["0-65535"]
+        - protocol: udp
+          ports: ["0-65535"]
+        - protocol: icmp
+
+  - id: gke-tpu-v6-net-1
+    source: modules/network/vpc
+    settings:
+      network_name: $(vars.deployment_name)-net-1
+      subnetworks:
+      - subnet_name: $(vars.deployment_name)-sub-1
+        subnet_region: $(vars.region)
+        subnet_ip: 192.168.64.0/18
+      firewall_rules:
+      - name: $(vars.deployment_name)-internal-1
+        ranges: [192.168.0.0/16]
         allow:
         - protocol: tcp
           ports: ["0-65535"]
@@ -125,20 +161,36 @@ deployment_groups:
 
   - id: gke-tpu-v6-cluster
     source: modules/scheduler/gke-cluster
-    use: [gke-tpu-v6-net, workload_service_account]
+    use: [gke-tpu-v6-net-0, workload_service_account]
     settings:
       system_node_pool_machine_type: "n2-standard-8"
       system_node_pool_disk_size_gb: $(vars.system_node_pool_disk_size_gb)
       system_node_pool_taints: []
       enable_private_endpoint: false # Allows access from authorized public IPs
       configure_workload_identity_sa: true
       enable_gcsfuse_csi: true
+      enable_managed_lustre_csi: true
       master_authorized_networks:
       - cidr_block: $(vars.authorized_cidr) # Allows your machine to run the kubectl command. Required for multi network setup.
         display_name: "kubectl-access-network"
+      additional_networks:
+        $(concat(
+          [{
+            network=gke-tpu-v6-net-1.network_name,
+            subnetwork=gke-tpu-v6-net-1.subnetwork_name,
+            subnetwork_project=vars.project_id,
+            nic_type="GVNIC",
+            queue_count=null,
+            network_ip=null,
+            stack_type=null,
+            access_config=[{nat_ip=null, public_ptr_domain_name=null, network_tier=null}],
+            ipv6_access_config=[],
+            alias_ip_range=[]
+          }]
+        ))
       # Cluster versions cannot be updated through the toolkit after creation
       # Please manage cluster version from the Google Cloud Console directly
-      version_prefix: "1.32."
+      version_prefix: $(vars.version_prefix)
       release_channel: RAPID
       maintenance_exclusions:
       - name: no-minor-or-node-upgrades-indefinite
@@ -159,6 +211,21 @@ deployment_groups:
       zones: [$(vars.zone)]
       disk_size_gb: $(vars.v6e_node_pool_disk_size_gb)
       static_node_count: $(vars.static_node_count)
+      additional_networks:
+        $(concat(
+          [{
+            network=gke-tpu-v6-net-1.network_name,
+            subnetwork=gke-tpu-v6-net-1.subnetwork_name,
+            subnetwork_project=vars.project_id,
+            nic_type="GVNIC",
+            queue_count=null,
+            network_ip=null,
+            stack_type=null,
+            access_config=[{nat_ip=null, public_ptr_domain_name=null, network_tier=null}],
+            ipv6_access_config=[],
+            alias_ip_range=[]
+          }]
+        ))
       reservation_affinity:
         consume_reservation_type: SPECIFIC_RESERVATION
         specific_reservations:
@@ -168,6 +235,51 @@ deployment_groups:
         tpu_topology: $(vars.tpu_topology)
     outputs: [instructions]
 
+  # # --- MANAGED LUSTRE ADDITIONS ---
+  # # Private Service Access (PSA) requires the compute.networkAdmin role which is
+  # # included in the Owner role, but not Editor.
+  # # PSA is required for all Managed Lustre functionality.
+  # # https://cloud.google.com/vpc/docs/configure-private-services-access#permissions
+  # - id: private_service_access
+  #   source: community/modules/network/private-service-access
+  #   use: [gke-tpu-v6-net-0]
+  #   settings:
+  #     prefix_length: 24
+
+  # # Firewall to allow Managed Lustre connection
+  # - id: lustre_firewall_rule
+  #   source: modules/network/firewall-rules
+  #   use: [gke-tpu-v6-net-0]
+  #   settings:
+  #     ingress_rules:
+  #     - name: $(vars.deployment_name)-allow-lustre-traffic
+  #       description: Allow Managed Lustre traffic
+  #       source_ranges:
+  #       - $(private_service_access.cidr_range)
+  #       allow:
+  #       - protocol: tcp
+  #       ports:
+  #       - "988"
+
+  # - id: managed-lustre
+  #   source: modules/file-system/managed-lustre
+  #   use: [gke-tpu-v6-net-0, private_service_access]
+  #   settings:
+  #     name: $(vars.lustre_instance_id)
+  #     local_mount: /lustre
+  #     remote_mount: lustrefs
+  #     size_gib: $(vars.lustre_size_gib)
+  #     per_unit_storage_throughput: $(vars.per_unit_storage_throughput)
+  #     zone: $(vars.zone)
+  # # Please ensure the zone of the gke cluster falls in the list of available zones of the managed-lustre instance. If not,
+  # # please explicitly specify a valid zone for managed-lustre instance. Refer https://cloud.google.com/managed-lustre/docs/locations
+
+  # - id: lustre-pv
+  #   source: modules/file-system/gke-persistent-volume
+  #   use: [managed-lustre, gke-tpu-v6-cluster]
+  #   settings:
+  #     capacity_gib: $(vars.lustre_size_gib)
+
   - id: workload-manager-install
     source: modules/management/kubectl-apply
     use: [gke-tpu-v6-cluster]