Merge branch 'develop' into dependabot/pip/community/front-end/ofe/django-5.2.8

Author: Neelabh94

community/examples/gke-tpu-v6/README.md

@@ -92,10 +92,11 @@ This section guides you through the cluster creation process, ensuring that your
 
 ## Advanced Blueprint: GKE TPU with GCS Integration
 
-This repository also includes an advanced blueprint, `gke-tpu-v6-gcs.yaml`, designed for production-ready workloads. It builds on the basic blueprint by adding several key features:
+This repository also includes an advanced blueprint, `gke-tpu-v6-advanced.yaml`, designed for production-ready workloads. It builds on the basic blueprint by adding several key features:
 * **Dedicated Service Accounts** for nodes and workloads, following security best practices.
 * **Automatic creation of two GCS buckets** for training data and checkpoints.
 * **Performance-tuned GCS FUSE mounts** pre-configured in the cluster as Persistent Volumes.
+* **Optional High-Performance Storage: [Managed Lustre](https://cloud.google.com/managed-lustre/docs/overview)** for high-performance, fully managed parallel file system optimized for heavy AI and HPC workloads. For details of configuring Managed Lustre, please refer to the [appendix](#understanding-managed-lustre-integration)
 
 ### Deploying the Advanced Blueprint
 
@@ -114,14 +115,6 @@ The process is nearly identical to the basic deployment.
 
 1. After deployment, the blueprint will output instructions for running a fio benchmark job. This job serves as a validation test to confirm that the GCS mounts are working correctly for both reading and writing. Follow the printed instructions to run the test.
 
-### Understanding the GCS Integration
-
-The advanced blueprint provisions several key technologies to create a robust data pipeline for your TPU workloads. Here are some resources to understand how they work together:
-* [Cloud Storage Overview](https://cloud.google.com/storage/docs/introduction#quickstarts): Start here to understand what Cloud Storage buckets are and their role in storing large-scale data.
-* [Cloud TPU Storage Options](https://cloud.google.com/tpu/docs/storage-options): Learn about the recommended storage patterns for Cloud TPUs, including why GCS FUSE is a best practice for providing training data.
-* [Access GCS Buckets with the GCS FUSE CSI Driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver): This is the core technical guide explaining how GKE mounts GCS buckets into your pods, which this blueprint automates.
-* [Configure Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity): Read this to understand the secure, recommended method for GKE applications to access Google Cloud services like GCS, which this blueprint configures for you.
-
 ## Run the sample job
 
 The [tpu-available-chips.yaml](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/examples/gke-tpu-v6/tpu-available-chips.yaml) file creates a service and a job resource in kubernetes. It is based on https://cloud.google.com/kubernetes-engine/docs/how-to/tpus#tpu-chips-node-pool. The  workload returns the number of TPU chips across all of the nodes in a multi-host TPU slice.
@@ -180,6 +173,78 @@ To avoid recurring charges for the resources used on this page, clean up the res
    ./gcluster destroy gke-tpu-v6/
    ```
 
-Useful TPU links:
+## Appendix
+
+### Useful TPU links
 1. [TPU architecture](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm)
 2. [TPU v6](https://cloud.google.com/tpu/docs/v6e)
+
+### Understanding the GCS Integration
+
+The advanced blueprint provisions several key technologies to create a robust data pipeline for your TPU workloads. Here are some resources to understand how they work together:
+* [Cloud Storage Overview](https://cloud.google.com/storage/docs/introduction#quickstarts): Start here to understand what Cloud Storage buckets are and their role in storing large-scale data.
+* [Cloud TPU Storage Options](https://cloud.google.com/tpu/docs/storage-options): Learn about the recommended storage patterns for Cloud TPUs, including why GCS FUSE is a best practice for providing training data.
+* [Access GCS Buckets with the GCS FUSE CSI Driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver): This is the core technical guide explaining how GKE mounts GCS buckets into your pods, which this blueprint automates.
+* [Configure Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity): Read this to understand the secure, recommended method for GKE applications to access Google Cloud services like GCS, which this blueprint configures for you.
+
+### Understanding Managed Lustre integration
+The advanced blueprint `gke-tpu-v6-advanced.yaml` can also be configured to deploy a Managed Lustre filesystem. Google Cloud **Managed Lustre** delivers a high-performance, fully managed parallel file system optimized for AI and HPC applications. With multi-petabyte-scale capacity and up to 1 TBps throughput, [Managed Lustre](https://cloud.google.com/architecture/optimize-ai-ml-workloads-managed-lustre) facilitates the migration of demanding workloads to the cloud.
+
+#### Enabling Managed Lustre
+To enable Managed Lustre, you must make these changes before deploying:
+
+1. In the `gke-tpu-v6-advanced.yaml`:
+Find the **vars:** section and **uncomment** the Managed Lustre variables. The defaults provide a high-performance **36000GiB** (~35.16TiB) filesystem with **18 GB/s** of throughput.
+
+2. In the `gke-tpu-v6-advanced.yaml`:
+Find the section commented # --- MANAGED LUSTRE ADDITIONS ---. **Uncomment** the entire block of four modules: `private_service_access`, `lustre_firewall_rule`, `managed-lustre`, and `lustre-pv`.
+
+After making these changes, run the `gcluster deploy` command as usual.
+
+#### Using Managed Lustre in a Pod
+Once deployed, the `Lustre` filesystem is available to the cluster as a `Persistent Volume (PV)`. To use it in your workloads, you need to create a `Persistent Volume Claim (PVC)` and mount it in your pod.
+
+#### Testing the Lustre Mount
+
+1. Create a file named `lustre-claim-pod.yaml`:
+
+    ```yaml
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    metadata:
+      name: my-lustre-claim
+    spec:
+      accessModes:
+      - ReadWriteMany
+      # storageClassName must be empty to bind to the manually created PV
+      storageClassName: ""
+      resources:
+        requests:
+          # This size must match lustre_size_gib from your variables
+          storage: 36000Gi
+    ---
+    apiVersion: v1
+    kind: Pod
+    metadata:
+      name: lustre-test-pod
+    spec:
+      containers:
+      - name: test-container
+        image: ubuntu:22.04
+        command: ["/bin/sleep", "infinity"]
+        volumeMounts:
+        - name: lustre-storage
+          mountPath: /mnt/lustre
+      volumes:
+      - name: lustre-storage
+        persistentVolumeClaim:
+          claimName: my-lustre-claim
+    ```
+
+2. Apply the manifest to your cluster:
+
+    ```yaml
+    kubectl apply -f lustre-claim-pod.yaml
+    ```
+
+The pod will start, and the Managed Lustre filesystem will be available inside the container at `/mnt/lustre`.

community/examples/gke-tpu-v6/gke-tpu-v6-advanced.yaml

@@ -51,14 +51,32 @@ vars:
 
   system_node_pool_disk_size_gb: 200
   v6e_node_pool_disk_size_gb: 100
+  version_prefix: "1.33."
+
+  # # To enable Managed-Lustre please uncomment this section and fill out the settings.
+  # # Additionally, please uncomment the private_service_access, lustre_firewall_rule, managed-lustre and lustre-pv modules.
+  # # Managed Lustre is only supported in specific regions and zones
+  # # Please refer https://cloud.google.com/managed-lustre/docs/locations
+
+  # # Managed-Lustre instance name. This should be unique for each deployment.
+  # lustre_instance_id: $(vars.deployment_name)
+
+  # # The values of size_gib and per_unit_storage_throughput are co-related
+  # # Please refer https://cloud.google.com/managed-lustre/docs/create-instance#performance-tiers
+  # # Storage capacity of the lustre instance in GiB
+  # lustre_size_gib: 36000
+
+  # # Maximum throughput of the lustre instance in MBps per TiB
+  # per_unit_storage_throughput: 500
 
 
 deployment_groups:
 - group: primary
   modules:
-  - id: gke-tpu-v6-net
+  - id: gke-tpu-v6-net-0
     source: modules/network/vpc
     settings:
+      network_name: $(vars.deployment_name)-net-0
       subnetworks:
       - subnet_name: $(vars.deployment_name)-sub-0
         subnet_region: $(vars.region)
@@ -72,7 +90,25 @@ deployment_groups:
           ip_cidr_range: 10.0.32.0/20
       firewall_rules:
       - name: $(vars.deployment_name)-internal-0
-        ranges: [192.168.0.0/18]
+        ranges: [192.168.0.0/16]
+        allow:
+        - protocol: tcp
+          ports: ["0-65535"]
+        - protocol: udp
+          ports: ["0-65535"]
+        - protocol: icmp
+
+  - id: gke-tpu-v6-net-1
+    source: modules/network/vpc
+    settings:
+      network_name: $(vars.deployment_name)-net-1
+      subnetworks:
+      - subnet_name: $(vars.deployment_name)-sub-1
+        subnet_region: $(vars.region)
+        subnet_ip: 192.168.64.0/18
+      firewall_rules:
+      - name: $(vars.deployment_name)-internal-1
+        ranges: [192.168.0.0/16]
         allow:
         - protocol: tcp
           ports: ["0-65535"]
@@ -125,20 +161,36 @@ deployment_groups:
 
   - id: gke-tpu-v6-cluster
     source: modules/scheduler/gke-cluster
-    use: [gke-tpu-v6-net, workload_service_account]
+    use: [gke-tpu-v6-net-0, workload_service_account]
     settings:
       system_node_pool_machine_type: "n2-standard-8"
       system_node_pool_disk_size_gb: $(vars.system_node_pool_disk_size_gb)
       system_node_pool_taints: []
       enable_private_endpoint: false # Allows access from authorized public IPs
       configure_workload_identity_sa: true
       enable_gcsfuse_csi: true
+      enable_managed_lustre_csi: true
       master_authorized_networks:
       - cidr_block: $(vars.authorized_cidr) # Allows your machine to run the kubectl command. Required for multi network setup.
         display_name: "kubectl-access-network"
+      additional_networks:
+        $(concat(
+          [{
+            network=gke-tpu-v6-net-1.network_name,
+            subnetwork=gke-tpu-v6-net-1.subnetwork_name,
+            subnetwork_project=vars.project_id,
+            nic_type="GVNIC",
+            queue_count=null,
+            network_ip=null,
+            stack_type=null,
+            access_config=[{nat_ip=null, public_ptr_domain_name=null, network_tier=null}],
+            ipv6_access_config=[],
+            alias_ip_range=[]
+          }]
+        ))
       # Cluster versions cannot be updated through the toolkit after creation
       # Please manage cluster version from the Google Cloud Console directly
-      version_prefix: "1.32."
+      version_prefix: $(vars.version_prefix)
       release_channel: RAPID
       maintenance_exclusions:
       - name: no-minor-or-node-upgrades-indefinite
@@ -159,6 +211,21 @@ deployment_groups:
       zones: [$(vars.zone)]
       disk_size_gb: $(vars.v6e_node_pool_disk_size_gb)
       static_node_count: $(vars.static_node_count)
+      additional_networks:
+        $(concat(
+          [{
+            network=gke-tpu-v6-net-1.network_name,
+            subnetwork=gke-tpu-v6-net-1.subnetwork_name,
+            subnetwork_project=vars.project_id,
+            nic_type="GVNIC",
+            queue_count=null,
+            network_ip=null,
+            stack_type=null,
+            access_config=[{nat_ip=null, public_ptr_domain_name=null, network_tier=null}],
+            ipv6_access_config=[],
+            alias_ip_range=[]
+          }]
+        ))
       reservation_affinity:
         consume_reservation_type: SPECIFIC_RESERVATION
         specific_reservations:
@@ -168,6 +235,51 @@ deployment_groups:
         tpu_topology: $(vars.tpu_topology)
     outputs: [instructions]
 
+  # # --- MANAGED LUSTRE ADDITIONS ---
+  # # Private Service Access (PSA) requires the compute.networkAdmin role which is
+  # # included in the Owner role, but not Editor.
+  # # PSA is required for all Managed Lustre functionality.
+  # # https://cloud.google.com/vpc/docs/configure-private-services-access#permissions
+  # - id: private_service_access
+  #   source: community/modules/network/private-service-access
+  #   use: [gke-tpu-v6-net-0]
+  #   settings:
+  #     prefix_length: 24
+
+  # # Firewall to allow Managed Lustre connection
+  # - id: lustre_firewall_rule
+  #   source: modules/network/firewall-rules
+  #   use: [gke-tpu-v6-net-0]
+  #   settings:
+  #     ingress_rules:
+  #     - name: $(vars.deployment_name)-allow-lustre-traffic
+  #       description: Allow Managed Lustre traffic
+  #       source_ranges:
+  #       - $(private_service_access.cidr_range)
+  #       allow:
+  #       - protocol: tcp
+  #       ports:
+  #       - "988"
+
+  # - id: managed-lustre
+  #   source: modules/file-system/managed-lustre
+  #   use: [gke-tpu-v6-net-0, private_service_access]
+  #   settings:
+  #     name: $(vars.lustre_instance_id)
+  #     local_mount: /lustre
+  #     remote_mount: lustrefs
+  #     size_gib: $(vars.lustre_size_gib)
+  #     per_unit_storage_throughput: $(vars.per_unit_storage_throughput)
+  #     zone: $(vars.zone)
+  # # Please ensure the zone of the gke cluster falls in the list of available zones of the managed-lustre instance. If not,
+  # # please explicitly specify a valid zone for managed-lustre instance. Refer https://cloud.google.com/managed-lustre/docs/locations
+
+  # - id: lustre-pv
+  #   source: modules/file-system/gke-persistent-volume
+  #   use: [managed-lustre, gke-tpu-v6-cluster]
+  #   settings:
+  #     capacity_gib: $(vars.lustre_size_gib)
+
   - id: workload-manager-install
     source: modules/management/kubectl-apply
     use: [gke-tpu-v6-cluster]

community/examples/sycomp/README.md

@@ -0,0 +1,117 @@
+# Sycomp Intelligent Data Storage Platform Blueprint
+
+This document provides instructions on how to use a blueprint to deploy or expand a `Sycomp Intelligent Data Storage Platform` cluster and enable a `SLURM` cluster to access the data via `NFS` on Google Cloud Platform (GCP)
+using the Google Cluster Toolkit.
+
+The directory contains four example blueprints that can be used to deploy or expand a `Sycomp Storage` cluster:
+
+- sycomp-storage.yaml
+
+  Blueprint used to deploy a `Sycomp Storage` cluster consisting of 3 storage servers.
+
+- sycomp-storage-expansion.yaml
+
+  Blueprint used to expand the `Sycomp Storage` cluster to 4 storage servers.
+
+- sycomp-storage-ece.yaml
+
+  Blueprint used to deploy a `Sycomp Storage` cluster consisting of 7 storage servers using `ECE` (Erasure Code Edition) software RAID.
+
+- sycomp-storage-slurm.yaml
+
+  Blueprint used to deploy a `SLURM` cluster and a `Sycomp Storage` cluster with three servers. The `SLURM` compute nodes are configured as `NFS` clients and have the ability to use the Sycomp Storage filesystem.
+
+## Prerequisites
+
+1. Google Cloud SDK is installed and configured.
+2. The Google Cloud Cluster Toolkit is set up [Set up Cluster Toolkit](https://cloud.google.com/cluster-toolkit/docs/setup/configure-environment).
+3. You have a Google Cloud project and the required permissions to create VPCs, and Compute Engine instances.
+4. The following APIs must be enabled in your project:
+   - Compute Engine API (`compute.googleapis.com`)
+5. You have an SSH key pair (`~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub` or similar). If you don't have one, you can generate it using `ssh-keygen -t rsa`.
+6. You have a valid **Customer Token** provided by Sycomp.
+7. You have a valid **Project access token** provided by Sycomp.
+
+> **_NOTE:_** To get a customer and project access token, contact Sycomp (sycompstorage@sycomp.com).
+>
+> **_NOTE:_** To avoid repeatedly entering passwords, you can set `credential.helper` in git.
+
+```shell
+git config credential.helper cache
+```
+
+## Configuration
+
+Before deploying you need to edit the blueprint yaml files and replace the placeholder values.
+
+Required parameter updates for each blueprint:
+
+- **`vars` block:**
+  - `project_id`: Your Google Cloud project ID.
+  - `deployment_name`: A unique name for this deployment (e.g., `sycomp-storage`).
+  - `region`: The region where you want to deploy the cluster.
+  - `zone`: The zone where you want to deploy the cluster.
+
+- **`network1` module settings:**
+  - `source`: `modules/network/vpc` is the default and deploys the cluster in a newly created network. To use an existing network, change `source` to `modules/network/pre-existing-vpc`.
+  - `network_name`: A name for the new VPC network (e.g., `sycomp-net`).
+  - `subnetwork_name`: A name for the new subnetwork (e.g., `sycomp-subnet`).
+  - `allowed_ssh_ip_ranges`: A list of IP address ranges in CIDR format that
+     are allowed to connect via SSH. **You must include the IP address of the
+     machine you are running the deployment from.** For example: `["1.2.3.0/24"]`.
+
+- **`sycomp-scale-gcp` module settings:**
+  - `security.ssh.ssh_user_name`: The username for SSH access to the management node (e.g., `sycomp`).
+  - `security.ssh.private_key`: The file path to your SSH private key (e.g., `~/.ssh/id_rsa`).
+  - `security.ssh.public_key`: The file path to your SSH public key (e.g., `~/.ssh/id_rsa.pub`).
+  - `security.customer_token.token`: Your Sycomp Customer Token. To get a customer token, contact Sycomp (`sycompstorage@sycomp.com`).
+  - `name_prefix` (Optional): By default, `sycomp-scale-gcp` sets the value of `name_prefix` based on the `deployment_name`. Since `name_prefix` only accepts **alphanumeric** characters (letters and numbers), `sycomp-scale-gcp` automatically **removes any non-alphanumeric characters** (such as hyphens and underscores) before assigning the value. For example, a `deployment_name` of `sycomp-storage-1` would result in a `name_prefix` of `sycompstorage1`.
+  - **`scale_config`** (Optional):
+    - `scale_node_count`: The number of nodes in the Sycomp Storage cluster. Default is `3`.
+    - `scale_volumes`: Configuration for the data disks. Default is 4 disks of 250GiB each per storage node.
+
+- **`sycomp-scale-expansion` module settings:**
+  - `name_prefix`: Must match the `name_prefix` of the cluster to be expanded. Note: if not set explicitly, `name_prefix` is derived from `deployment_name` by removing non-alphanumeric characters. You can find the correct value for an existing cluster by checking its Terraform outputs or by inspecting resource names in the Google Cloud Console.
+  - `add_scale_nodes`: The number of nodes to be added.
+
+## Deployment
+
+Once the blueprint file (e.g., `sycomp-storage.yaml`) is configured, you can deploy the cluster from the blueprint by following these steps from your terminal.
+
+1. **Authenticate with Google Cloud:**
+
+   ```bash
+   gcloud auth login
+   gcloud auth application-default login
+   ```
+
+2. **Create the deployment directory:**
+
+   ```bash
+   # Replace <blueprint-filename> with the specific blueprint filename (e.g., sycomp-storage.yaml).
+   gcluster create community/examples/sycomp/<blueprint-filename>
+   ```
+
+   This command will create a new directory named after your `deployment_name` (e.g., `sycomp-storage`).
+
+3. **Deploy the resources:**
+
+   ```bash
+   # The terraform init command will prompt for a username and password.
+   # Enter any username and use the project access token from Sycomp as the password.
+   # To get a project access token, contact Sycomp (sycompstorage@sycomp.com).
+   #
+   # Replace <deployment_name> with the `deployment_name` from your blueprint (e.g., "sycomp-storage").
+   terraform -chdir=<deployment_name>/primary init
+   gcluster deploy <deployment_name>
+   ```
+
+   This process will take several minutes as it provisions the Sycomp Storage cluster nodes.
+
+## Cleanup
+
+To remove all resources created by this blueprint, run the following command:
+
+```bash
+gcluster destroy <deployment_name>
+```