Merge pull request GoogleCloudPlatform#4768 from GoogleCloudPlatform/release-candidate

Release candidate: v1.69.0

Author: ankitkumar-quad

.github/workflows/multi-approvers.yml

@@ -15,15 +15,6 @@
 name: 'multi-approvers'
 
 on:
-  pull_request:
-    types:
-    - 'opened'
-    - 'edited'
-    - 'reopened'
-    - 'synchronize'
-    - 'ready_for_review'
-    - 'review_requested'
-    - 'review_request_removed'
   pull_request_review:
     types:
     - 'submitted'
@@ -40,6 +31,7 @@ concurrency:
 
 jobs:
   multi-approvers:
+    if: github.event_name == 'pull_request_review' && (github.event.review.state == 'approved' ||  github.event.action == 'dismissed')
     uses: 'abcxyz/pkg/.github/workflows/multi-approvers.yml@main'
     with:
       org-members-path: 'GoogleCloudPlatform/cluster-toolkit/develop/cluster-toolkit-writers.json'

cmd/root.go

@@ -53,7 +53,7 @@ HPC deployments on the Google Cloud Platform.`,
 				logging.Fatal("cmd.Help function failed: %s", err)
 			}
 		},
-		Version:     "v1.68.0",
+		Version:     "v1.69.0",
 		Annotations: annotation,
 	}
 )

…ity/gce-cos-nvidia-bug-report/Dockerfile …mmunity/cos-nvidia-bug-report/Dockerfilecommunity/gce-cos-nvidia-bug-report/Dockerfile renamed to community/cos-nvidia-bug-report/Dockerfile community/gce-cos-nvidia-bug-report/Dockerfile renamed to community/cos-nvidia-bug-report/Dockerfile

@@ -32,6 +32,7 @@ RUN apt-get update && \
     libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev \
     acpica-tools kmod usbutils procps nano python3-pip python3.11-venv
 
+RUN apt-get install -y mesa-utils dmidecode vulkan-tools ibutils x11-xserver-utils
 RUN wget https://www.python.org/ftp/python/3.13.1/Python-3.13.1.tgz
 
 RUN tar -xvf Python-3.13.1.tgz
@@ -51,4 +52,4 @@ RUN . .venv/bin/activate
 # Install Python dependencies
 RUN .venv/bin/pip install --no-cache-dir -r requirements.txt
 
-ENTRYPOINT [".venv/bin/python", "gce-cos-nvidia-bug-report.py"]
+ENTRYPOINT [".venv/bin/python", "cos-nvidia-bug-report.py"]

…nity/gce-cos-nvidia-bug-report/README.md community/cos-nvidia-bug-report/README.mdcommunity/gce-cos-nvidia-bug-report/README.md renamed to community/cos-nvidia-bug-report/README.md community/gce-cos-nvidia-bug-report/README.md renamed to community/cos-nvidia-bug-report/README.md

@@ -1,9 +1,9 @@
-# GCE COS NVIDIA Bug Report Collector
+# GCP COS NVIDIA Bug Report Collector
 
-![Platform: GCE/COS](https://img.shields.io/badge/Platform-GCE%2FCOS-green.svg)
+![Platform: GCP/COS](https://img.shields.io/badge/Platform-GCP%2FCOS-green.svg)
 
 A universal tool to simplify the generation of NVIDIA bug reports on Google
-Compute Engine (GCE) VMs that use the Container-Optimized OS (COS) guest
+Compute Platform (GCP) VMs that use the Container-Optimized OS (COS) guest
 operating system.
 
 This script provides a simple and reliable one-command experience to collect
@@ -14,11 +14,11 @@ to generate a more comprehensive report with deep hardware diagnostics.
 
 --------------------------------------------------------------------------------
 
-## 🤔 The Challenge: Getting GPU Bug Report on GCE with COS
+## 🤔 The Challenge: Getting GPU Bug Report on GCP with COS
 
 When troubleshooting GPU issues, the first step is often to generate an
-`nvidia-bug-report`. However, doing so on a **Google Compute Engine (GCE)
-virtual machine that uses Container-Optimized OS (COS) as its guest operating
+`nvidia-bug-report`. However, doing so on a **Google Compute Platform (GCP)
+that uses Container-Optimized OS (COS) as its guest operating
 system** could be a less trivial.
 
 COS is a minimal, security-hardened operating system from Google, designed
@@ -66,7 +66,7 @@ However, setting up MFT is a cumbersome process on COS:
 ## 💡 Our Solution: A Smart, All-in-One Collector
 
 This script eliminates all of the aforementioned complexity. It acts as a
-universal collector that simplifies bug report generation for all users on GCE
+universal collector that simplifies bug report generation for all users on GCP
 with COS.
 
 * **For all supported GPUs**, it automates the steps needed to generate a
@@ -80,8 +80,8 @@ docker command.
 
 ### ✨ Key Features
 
-* **Universal Collector for GCE COS**: A single, simple command to generate an
-    `nvidia-bug-report` on any supported GCE VM with the COS guest OS.
+* **Universal Collector for GCP COS**: A single, simple command to generate an
+    `nvidia-bug-report` on any supported GCP machine with the COS guest OS.
 * **Automatic MFT Enhancement**: For Blackwell and newer GPUs, the script
     automatically installs and configures the NVIDIA MFT suite to unlock deeper,
     more comprehensive hardware diagnostics.
@@ -92,14 +92,15 @@ docker command.
 
 Before running the script, ensure you have:
 
-1. A Google Compute Engine (GCE) GPU VM instance with **Container-Optimized OS
+1. A Google Compute Engine (GCE) GPU VM instance or a GKE node with **Container-Optimized OS
     (COS)** as its guest operating system.
 
-2. The GPU driver is installed on the VM instance.
+2. The GPU driver is installed on the GCE VM instance.
    * Please refer to
         [COS's official documentation page](\(https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus#install\))
         for more detail.
    * Sample commands to install the GPU driver and verify the installation:
+   * ***NOTE:*** For GKE the driver is already installed. Unless opted for [manual driver installation](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#create-gpu-pool-auto-drivers)
 
     ```bash
     # Install NVIDIA GPU Driver
@@ -151,7 +152,9 @@ Before running the script, ensure you have:
 This tool is designed to be run as a Docker container. The primary method of use
 is a single docker run command.
 
-Sample command to run on a VM with 8 GPUs:
+### To run in a GCE VM instance
+
+Sample command to run on a GCE VM with 8 GPUs:
 
 Note: If you have a different number of GPUs on your system, you may need to
 adjust the `--device /dev/nvidia<gpu_num>:/dev/nvidia<gpu_num>` in the docker
@@ -177,6 +180,22 @@ us-central1-docker.pkg.dev/gce-ai-infra/gce-cos-nvidia-bug-report-repo/gce-cos-n
 --gcs_bucket=${GCS_BUCKET}
 ```
 
+### To run in a GKE cluster
+Note: Update the below fields in the pod manifest `./bug-report-pod.yaml`
+
+| Fields | Description | Example |
+|:--------:|:--------:|:--------:|
+|  `<gke-node-name>`  |   GKE node which we want to target  |  gke-a4-2-a4-highgpu-8g-a4-p-b99cb17a-xrre  |
+|  `<number_of_gpus>`  |  Total number of GPUs on the targeted GPU node  |  A4 - 8, A4X - 4  |
+
+Note: Exporting the final bug reports to a GCS bucket is optional. If you do not
+intend to export it elsewhere, you may remove the `--gcs_bucket=${GCS_BUCKET}`
+at the end.
+
+```bash
+kubectl apply -f cluster-toolkit/community/cos-nvidia-bug-report/bug-report-pod.yaml
+```
+
 ### 📝 Example Output
 
 ```bash
@@ -274,7 +293,7 @@ Registry.
 We provide a convenient shell script to build and push your customized image to
 your own Artifact Registry.
 
-You can do so by invoking the `build-and-push-gce-cos-nvidia-bug-report.sh`
+You can do so by invoking the `build-and-push-cos-nvidia-bug-report.sh`
 script with the following parameters:
 
 | Flag | Description                                                     | Required |
@@ -288,7 +307,7 @@ script with the following parameters:
 Sample command:
 
 ```bash
-bash build-and-push-gce-cos-nvidia-bug-report.sh \
+bash build-and-push-cos-nvidia-bug-report.sh \
     -p ${PROJECT?} \
     -r ${ARTIFACT_REPO?} \
     -i "custom-bug-report-collector" \

…e-cos-nvidia-bug-report/app/constants.py …y/cos-nvidia-bug-report/app/constants.pycommunity/gce-cos-nvidia-bug-report/app/constants.py renamed to community/cos-nvidia-bug-report/app/constants.py community/gce-cos-nvidia-bug-report/app/constants.py renamed to community/cos-nvidia-bug-report/app/constants.py

@@ -0,0 +1,66 @@
+# Copyright 2025 "Google LLC"
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: Pod
+metadata:
+  name: cos-nvidia-bug-report
+spec:
+  restartPolicy: Never
+  nodeSelector:
+    kubernetes.io/hostname: <gke-node-name> # TODO: Replace with your GKE node name
+  tolerations:
+  - effect: NoSchedule
+    key: nvidia.com/gpu
+    operator: Exists
+  containers:
+  - name: bug-reporter-container
+    resources:
+      limits:
+        nvidia.com/gpu: "<number_of_gpus>" # TODO: Replace with number of GPUs on this machine type
+      requests:
+        nvidia.com/gpu: "<number_of_gpus>" # TODO: Replace with number of GPUs on this machine type
+    env:
+    - name: LD_LIBRARY_PATH
+      value: /usr/local/nvidia/lib64
+    image: us-central1-docker.pkg.dev/gce-ai-infra/gce-cos-nvidia-bug-report-repo/gce-cos-nvidia-bug-report:latest
+    imagePullPolicy: Always
+    args:
+    - --gcs_bucket=${GCS_BUCKET}
+    securityContext:
+      privileged: true
+    volumeMounts:
+    - name: host-etc
+      mountPath: /etc_host
+      readOnly: true
+    - name: host-tmp
+      mountPath: /tmp
+      readOnly: false
+    - mountPath: /usr/local/nvidia
+      name: nvidia
+      readOnly: true
+  hostNetwork: true
+  volumes:
+  - name: host-etc
+    hostPath:
+      path: /etc
+      type: Directory
+  - name: host-tmp
+    hostPath:
+      path: /tmp
+      type: Directory
+  - hostPath:
+      path: /home/kubernetes/bin/nvidia
+      type: Directory
+    name: nvidia

…-report/app/gce-cos-nvidia-bug-report.py …-bug-report/app/cos-nvidia-bug-report.pycommunity/gce-cos-nvidia-bug-report/app/gce-cos-nvidia-bug-report.py renamed to community/cos-nvidia-bug-report/app/cos-nvidia-bug-report.py community/gce-cos-nvidia-bug-report/app/gce-cos-nvidia-bug-report.py renamed to community/cos-nvidia-bug-report/app/cos-nvidia-bug-report.py

@@ -16,7 +16,7 @@
 set -e -u -o pipefail
 
 usage() {
-	echo >&2 "Usage: bash build-and-push-gce-cos-nvidia-bug-report.sh -p <PROJECT_ID> -r <REPO_NAME> -i <IMAGE_NAME> [-l <REGION>]"
+	echo >&2 "Usage: bash build-and-push-cos-nvidia-bug-report.sh -p <PROJECT_ID> -r <REPO_NAME> -i <IMAGE_NAME> [-l <REGION>]"
 	echo >&2 "This script builds a Docker image and pushes it to Google Artifact Registry."
 	echo >&2 ""
 	echo >&2 "Options:"
@@ -27,10 +27,10 @@ usage() {
 	echo >&2 "  -h    Display this help message."
 	echo >&2 ""
 	echo >&2 "Example (default region):"
-	echo >&2 "  bash build-and-push-gce-cos-nvidia-bug-report.sh -p gpu-test-project -r gce-cos-nvidia-bug-report-repo -i gce-cos-nvidia-bug-report"
+	echo >&2 "  bash build-and-push-cos-nvidia-bug-report.sh -p gpu-test-project -r gce-cos-nvidia-bug-report-repo -i gce-cos-nvidia-bug-report"
 	echo >&2 ""
 	echo >&2 "Example (specific region):"
-	echo >&2 "  bash build-and-push-gce-cos-nvidia-bug-report.sh -p gpu-test-project -r gce-cos-nvidia-bug-report-repo -i gce-cos-nvidia-bug-report -l us-east4"
+	echo >&2 "  bash build-and-push-cos-nvidia-bug-report.sh -p gpu-test-project -r gce-cos-nvidia-bug-report-repo -i gce-cos-nvidia-bug-report -l us-east4"
 	exit 1
 }
 

…s-nvidia-bug-report/app/requirements.txt …s-nvidia-bug-report/app/requirements.txtcommunity/gce-cos-nvidia-bug-report/app/requirements.txt renamed to community/cos-nvidia-bug-report/app/requirements.txt community/gce-cos-nvidia-bug-report/app/requirements.txt renamed to community/cos-nvidia-bug-report/app/requirements.txt

@@ -29,6 +29,6 @@ terraform {
   }
 
   provider_meta "google" {
-    module_name = "blueprints/terraform/hpc-toolkit:htcondor-execute-point/v1.68.0"
+    module_name = "blueprints/terraform/hpc-toolkit:htcondor-execute-point/v1.69.0"
   }
 }