Merge branch 'develop' into main

Author: ankitkumar-quad

.gemini/config.yaml

@@ -0,0 +1,25 @@
+# Copyright 2025 "Google LLC"
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+have_fun: false
+code_review:
+  disable: false
+  comment_severity_threshold: MEDIUM
+  max_review_comments: -1
+  pull_request_opened:
+    help: false
+    summary: true
+    code_review: true
+    include_drafts: true
+ignore_patterns: []

.gemini/styleguide.md

@@ -0,0 +1,49 @@
+# Cluster Toolkit - Code Review Style Guide for Gemini
+
+When reviewing Pull Requests for the Google Cloud Cluster Toolkit, please adopt the persona of an expert Software Engineer. Your primary focus should be on ensuring changes enhance the project's long-term health. Prioritize the following:
+
+* **Technical Excellence:** Ensure code is well-structured, efficient, and follows best practices.
+* **Maintainability:** Code should be easy to understand, modify, and extend.
+* **Testing:** Changes must be well-tested. Encourage comprehensive unit and integration tests.
+* **Documentation:** Ensure documentation is updated, including in-code comments, module READMEs, and index files.
+
+Pay close attention to the following specifics:
+
+1. **Blueprint Authoring (YAML):**
+   * Ensure the `use` block is preferred for module dependencies within blueprints. Explicit variable linking (e.g., `setting = $(module.output)`) should only be used when necessary to resolve ambiguity.
+   * Verify that module sources are correct and the referenced modules exist.
+   * Check for logical grouping of modules within `deployment_groups`.
+   * Ensure variable usage is correct (e.g., `$(vars.name)`, `$(module.id.output)`).
+   * Validate the overall structure and syntax of the YAML blueprint.
+
+2. **Terraform Module Development (HCL):**
+   * Verify module inputs and outputs are consistent and well-defined.
+   * Check for clear variable definitions in `variables.tf` with descriptions, types, and sensible defaults where applicable.
+   * Ensure resources within the module are logically structured.
+   * Encourage the use of best practices for writing clean and maintainable Terraform code.
+   * Ensure new modules are placed in the correct directory (`modules/` or `community/modules/`) and within the appropriate subdirectory (e.g., `compute`, `network`, `file-system`, `scheduler`, etc.).
+
+3. **Go Language:**
+   * Follow standard Go idioms and best practices (e.g., error handling, naming).
+   * Ensure code is well-commented, especially public functions and complex logic.
+   * Check for test coverage for new or modified Go code.
+
+4. **Documentation:**
+   * **CRITICAL:** If new modules (core or community) are added, ensure they are added to the index in `modules/README.md`.
+   * **CRITICAL:** If new examples (core or community) are added, ensure they are added to the index in `examples/README.md`.
+   * In-code comments should be clear and explain the *why* not just the *what*.
+   * Module `README.md` files should be clear and provide sufficient information on usage, inputs, and outputs.
+
+5. **Testing:**
+   * New features or bug fixes should ideally be accompanied by tests.
+   * Tests should be clear and cover both happy paths and edge cases.
+   * Encourage the use of the existing testing frameworks and patterns within the project.
+
+6. **PR Description:**
+   * The PR description should clearly explain the purpose of the change and the problem it solves.
+   * It should mention how the changes were tested.
+
+7. **Structure:**
+   * Confirm adherence to the project structure (e.g., core vs. community).
+
+By focusing on these areas, you can help maintain the quality and consistency of the Cluster Toolkit codebase.

.github/workflows/pr-description-check.yml

@@ -0,0 +1,70 @@
+# Copyright 2025 "Google LLC"
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Check PR Description
+on:
+  pull_request:
+    types:
+    - opened
+    - edited
+    - synchronize
+jobs:
+  check-description:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+    - name: Check PR description
+      env:
+        PR_BODY: ${{ github.event.pull_request.body }}
+      run: |
+        TEMPLATE_PATH=".github/pull_request_template.md"
+
+        if [ -f "$TEMPLATE_PATH" ]; then
+          TEMPLATE_BODY=$(cat "$TEMPLATE_PATH")
+        else
+          TEMPLATE_BODY=""
+        fi
+
+        # Function to normalize text: removes empty lines, trims leading/trailing whitespace, and removes carriage returns
+        normalize_text() {
+          echo -n "$1" | tr -d '\r' | awk 'NF' | awk '{$1=$1};1'
+        }
+
+        # 1. Check if the PR body is empty or contains only whitespace
+        if [ -z "$(echo "$PR_BODY" | tr -d '[:space:]')" ]; then
+          echo "Error: PR description is empty or contains only whitespace."
+          exit 1
+        fi
+
+        # 2. Check if the PR body is identical to the template
+        NORMALIZED_PR_BODY=$(normalize_text "$PR_BODY")
+        NORMALIZED_TEMPLATE_BODY=$(normalize_text "$TEMPLATE_BODY")
+
+        if [[ "$NORMALIZED_PR_BODY" == "$NORMALIZED_TEMPLATE_BODY" ]]; then
+          echo "Error: PR description only contains the template text. Please add a description of your changes."
+          exit 1
+        fi
+
+        # 3. Check if at least 10 characters have been added to the description
+        #    Use `diff` to isolate the added lines between the template and the PR body.
+        ADDED_TEXT=$(diff --changed-group-format="%>" --unchanged-group-format="" <(echo "$NORMALIZED_TEMPLATE_BODY") <(echo "$NORMALIZED_PR_BODY") || true)
+        ADDED_TEXT_LEN=${#ADDED_TEXT}
+
+        if [[ "$ADDED_TEXT_LEN" -lt 10 ]]; then
+          echo "Error: PR description must contain at least 10 added characters."
+          exit 1
+        fi
+
+        echo "PR description check passed."

community/modules/scripts/wait-for-startup/scripts/wait-for-startup-status.sh

@@ -58,11 +58,12 @@ fetch_cmd="gcloud compute instances get-serial-port-output ${INSTANCE_NAME} --po
 # finishes on the new guest agent
 FINISH_LINE="startup-script exit status"
 # Match string for failures on the new guest agent
-FINISH_LINE_ERR="Script.*failed with error:"
+FINISH_LINE_ERR="Script \"startup-script\" failed with error:"
 
 # NEW: Accept also these finish lines as success.
 STARTUP_SCRIPT_SUCCEEDED_LINE="google-startup-scripts.service: Succeeded."
 STARTUP_SCRIPT_FINISHED_LINE="Finished Google Compute Engine Startup Scripts."
+STARTUP_SCRIPT_SERVICE_FINISHED_LINE="Finished google-startup-scripts.service - Google Compute Engine Startup Scripts."
 
 NON_FATAL_ERRORS=(
 	"Internal error"
@@ -72,7 +73,7 @@ until [[ now -gt deadline ]]; do
 	ser_log=$(
 		set -o pipefail
 		${fetch_cmd} 2>"${error_file}" |
-			c1grep "${FINISH_LINE}\|${FINISH_LINE_ERR}\|${STARTUP_SCRIPT_SUCCEEDED_LINE}\|${STARTUP_SCRIPT_FINISHED_LINE}"
+			c1grep "${FINISH_LINE}\|${FINISH_LINE_ERR}\|${STARTUP_SCRIPT_SUCCEEDED_LINE}\|${STARTUP_SCRIPT_FINISHED_LINE}\|${STARTUP_SCRIPT_SERVICE_FINISHED_LINE}"
 	) || {
 		err=$(cat "${error_file}")
 		echo "$err"
@@ -89,37 +90,49 @@ until [[ now -gt deadline ]]; do
 		fi
 	}
 	if [[ -n "${ser_log}" ]]; then break; fi
-	echo "Could not detect end of startup script. Sleeping."
 	sleep 5
 	now=$(date +%s)
 done
 
 # This line checks for an exit code - the assumption is that there is a number
-# at the end of the line and it is an exit code
-STATUS=$(sed -r 's/.*([0-9]+)\s*$/\1/' <<<"${ser_log}" | uniq)
+# at the end of the line and it is an exit code.
+# Modified to correctly extract the last numeric exit status from the relevant log line.
+LAST_EXIT_STATUS=$(echo "${ser_log}" | grep -oP "(?<=Script \"startup-script\" failed with error: exit status )[0-9]+" | tail -n 1)
+if [[ -z "${LAST_EXIT_STATUS}" ]]; then
+	LAST_EXIT_STATUS=$(echo "${ser_log}" | grep -oP "(?<=startup-script exit status )[0-9]+" | tail -n 1)
+fi
+
 # This specific text is monitored for in tests, do not change.
 INSPECT_OUTPUT_TEXT="To inspect the startup script output, please run:"
-if [[ "${STATUS}" == 0 ]]; then
-	echo "startup-script finished successfully"
-elif [[ "${STATUS}" == 1 ]]; then
+
+# --- Prioritize explicit failure from the script itself ---
+if [[ "${LAST_EXIT_STATUS}" == 1 ]]; then
 	echo "startup-script finished with errors, ${INSPECT_OUTPUT_TEXT}"
 	echo "${fetch_cmd}"
+	exit 1
+# --- Then explicit success from the script itself ---
+elif [[ "${LAST_EXIT_STATUS}" == 0 ]]; then
+	echo "startup-script finished successfully"
+	exit 0
 elif echo "${ser_log}" | grep -qE "${STARTUP_SCRIPT_SUCCEEDED_LINE}"; then
 	echo "startup-script finished successfully (startup script succeeded line detected)"
 	exit 0
 elif echo "${ser_log}" | grep -qE "${STARTUP_SCRIPT_FINISHED_LINE}"; then
 	echo "startup-script finished successfully (startup script finished line detected)"
 	exit 0
+elif echo "${ser_log}" | grep -qE "${STARTUP_SCRIPT_SERVICE_FINISHED_LINE}"; then
+	echo "startup-script finished successfully (startup script service finished line detected)"
+	exit 0
+# --- If we reached deadline, it's a timeout ---
 elif [[ now -ge deadline ]]; then
 	echo "startup-script timed out after ${TIMEOUT} seconds"
 	echo "${INSPECT_OUTPUT_TEXT}"
 	echo "${fetch_cmd}"
 	exit 1
+# --- All other cases are considered failure or invalid state ---
 else
-	echo "Invalid return status: '${STATUS}'"
+	echo "Invalid or undetermined startup script status. Last detected exit status: '${LAST_EXIT_STATUS}'"
 	echo "${INSPECT_OUTPUT_TEXT}"
 	echo "${fetch_cmd}"
-	exit 1
+	exit "${LAST_EXIT_STATUS}"
 fi
-
-exit "${STATUS}"

docs/network_storage.md

@@ -5,6 +5,7 @@ storage.
 
 The Toolkit contains modules that will **provision**:
 
+- [Google Cloud NetApp Volumes (GCP managed enterprise NFS and SMB)][netapp-volumes]
 - [Filestore (GCP managed NFS)][filestore]
 - [DDN EXAScaler lustre][ddn-exascaler] (Deprecated, removal on July 1, 2025)
 - [Managed Lustre][managed-lustre]
@@ -106,6 +107,7 @@ nfs-server | via USE | via USE | via USE | via STARTUP | via USE | via USE
 cloud-storage-bucket (GCS)| via USE | via USE | via USE | via STARTUP | via USE | via USE
 DDN EXAScaler lustre | via USE | via USE | via USE | Needs Testing | via USE | via USE
 Managed Lustre | via USE | Needs Testing | via USE | Needs Testing | Needs Testing |  Needs Testing
+netapp-volume | Needs Testing | Needs Testing | via USE | Needs Testing | Needs Testing | Needs Testing
   |  |   |   |   |   |  
 filestore (pre-existing) | via USE | via USE | via USE | via STARTUP | via USE | via USE
 nfs-server (pre-existing) | via USE | via USE | via USE | via STARTUP | via USE | via USE
@@ -129,3 +131,4 @@ GCS FUSE (pre-existing) | via USE | via USE | via USE | via STARTUP | via USE |
 [ddn-exascaler]: ../community/modules/file-system/DDN-EXAScaler/README.md
 [managed-lustre]: ../modules/file-system/managed-lustre/README.md
 [nfs-server]: ../community/modules/file-system/nfs-server/README.md
+[netapp-volumes]: ../modules/file-system/netapp-volume/README.md

examples/README.md

@@ -64,6 +64,7 @@ md_toc github examples/README.md | sed -e "s/\s-\s/ * /"
   * [xpk-n2-filestore](#xpk-n2-filestore--) ![community-badge] ![experimental-badge]
   * [gke-h4d](#gke-h4d-) ![core-badge]
   * [gke-g4](#gke-g4-) ![core-badge]
+  * [netapp-volumes.yaml](#netapp-volumesyaml--) ![core-badge]
 * [Blueprint Schema](#blueprint-schema)
 * [Writing an HPC Blueprint](#writing-an-hpc-blueprint)
   * [Blueprint Boilerplate](#blueprint-boilerplate)
@@ -1642,6 +1643,61 @@ This blueprint uses GKE to provision a Kubernetes cluster and a G4 node pool, al
 
 [gke-g4]: ../examples/gke-g4
 
+### [netapp-volumes.yaml] ![core-badge]
+
+This blueprint demonstrates how to provision NFS volumes as shared filesystems for compute VMs, using Google Cloud NetApp Volumes. It can be used as an alternative to FileStore in blueprints.
+
+NetApp Volumes is a first-party Google service that provides NFS and/or SMB shared file-systems to VMs. It offers advanced data management capabilities and highly scalable capacity and performance.
+
+NetApp Volume provides:
+
+* robust support for NFSv3, NFSv4.x and SMB 2.1 and 3.x
+* a [rich feature set][service-levels]
+* scalable [performance](https://cloud.google.com/netapp/volumes/docs/performance/performance-benchmarks)
+* FlexCache: Caching of ONTAP-based volumes to provide high-throughput and low latency read access to compute clusters of on-premises data
+* [Auto-tiering](https://cloud.google.com/netapp/volumes/docs/configure-and-use/volumes/manage-auto-tiering) of unused data to optimse cost
+
+Support for NetApp Volumes is split into two modules.
+
+* **netapp-storage-pool** provisions a [storage pool](https://cloud.google.com/netapp/volumes/docs/configure-and-use/storage-pools/overview). Storage pools are pre-provisioned storage capacity containers which host volumes. A pool also defines fundamental properties of all the volumes within, like the region, the attached network, the [service level][service-levels], CMEK encryption, Active Directory and LDAP settings.
+* **netapp-volume** provisions a [volume](https://cloud.google.com/netapp/volumes/docs/configure-and-use/volumes/overview) inside an existing storage pool. A volume is a file-system which is shared using NFS or SMB. It provides advanced data management capabilities.
+
+You can provision multiple volumes in a pool. For service levels Standard, Premium and Extreme the throughput capability depends on volume size and service level. Every GiB of provisioned volume space adds 16/64/128 KiBps of throughput capability.
+
+#### Steps to deploy the blueprint
+
+To provision the bluebrint, please run:
+
+```shell
+./gcluster create examples/netapp-volumes.yaml --vars "project_id=${GOOGLE_CLOUD_PROJECT}" --vars region=us-central1 --vars zone=us-central1-a
+./gcluster deploy netapp-volumes
+```
+
+After the blueprint deployed, you can login to the VM created:
+
+```shell
+gcloud compute ssh --zone "us-central1-a" "netapp-volumes-0" --project ${GOOGLE_CLOUD_PROJECT} --tunnel-through-iap
+```
+
+A NetApp Volumes volume is provisioned and mounted to /home in all the provisioned VMs. A home directory for your user is created automatically:
+
+```shell
+pwd
+df -h -t nfs
+```
+
+#### Clean Up
+To destroy all resources associated with creating the GKE cluster, run the following command:
+
+```sh
+./gcluster destroy netapp-volumes
+```
+
+[netapp-storage-pool]: ../netapp-storage-pool/README.md
+[service-levels]: https://cloud.google.com/netapp/volumes/docs/discover/service-levels
+[auto-tiering]: https://cloud.google.com/netapp/volumes/docs/configure-and-use/volumes/manage-auto-tiering
+[netapp-volumes.yaml]: ../examples/netapp-volumes.yaml
+
 ## Blueprint Schema
 
 Similar documentation can be found on