feat(testing): cut over daily test suite to Kueue scheduling

[sudheer-quad]

This PR transitions the slurm-gcp-v6-rocky-8 daily test suite to Kueue scheduling in the hpc-toolkit-dev-2 project.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request migrates the slurm-gcp-v6-rocky-8 daily test suite to use Kueue for job scheduling within the hpc-toolkit-dev-2 project. The changes involve replacing the legacy build process with a containerized job execution model that leverages Kueue for queue management, retry logic, and better resource utilization. Additionally, the infrastructure code has been updated to support these new triggers and scheduling requirements.

Highlights

• Kueue Migration: Transitioned the slurm-gcp-v6-rocky-8 daily test suite to use Kueue scheduling for improved resource management.
• Infrastructure Updates: Added new helper scripts for monitoring Kueue jobs, checking for retriable errors, and handling zone capacity retries.
• Terraform Configuration: Updated infrastructure provisioning to support the migrated test suite, including new triggers and scheduling configurations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request migrates the slurm-gcp-v6-rocky8 daily integration test to run as a GKE Kueue job, introducing scripts to monitor Kueue jobs, wait for zone capacity, and check for retriable errors, alongside updating the Terraform provisioning configurations. Feedback on these changes highlights several issues: missing secret environment variables in the job manifest generation step, a lack of file existence verification in the retriable error check script, a missing set -e inside the subshell of the zone availability script, and the potential for the newly added backup file slurm-gcp-v6-rocky8_old.yaml to be accidentally discovered and scheduled by automated test scripts.

[gemini-code-assist]

The generate-job-manifest step references several triage environment variables (e.g., $$TRIAGE_GCS_BUCKET, $$TRIAGE_PROJECT_NUMBER, $$TRIAGE_INVOKER_SA, $$TRIAGE_CLOUD_RUN_URL) inside the cat <<EOF block. Because these variables are not included in the secretEnv list for this step, they will expand to empty strings when generating job.yaml.

Please add these secret environment variables to the generate-job-manifest step.

  secretEnv: ['GCLUSTER_GCS_PATH', 'SA_EMAIL', 'TRIAGE_GCS_BUCKET', 'TRIAGE_PROJECT_NUMBER', 'TRIAGE_INVOKER_SA', 'TRIAGE_CLOUD_RUN_URL']

[gemini-code-assist]

The script does not verify if the log file actually exists and is readable before running grep. If the file is missing or unreadable, grep will output an error to stderr and exit with status 2, which might be confusing or cause unexpected behavior in the calling pipeline.

Please add a check to ensure the file exists and is a regular file.

Suggested change

	LOG_FILE=$1
	if [ -z "$LOG_FILE" ]; then
	echo "Usage: $0 <log_file>" >&2
	exit 2
	fi
	LOG_FILE=$1
	if [ -z "$LOG_FILE" ]; then
	echo "Usage: $0 <log_file>" >&2
	exit 2
	fi
	if [ ! -f "$LOG_FILE" ]; then
	echo "Error: Log file '$LOG_FILE' does not exist or is not a regular file." >&2
	exit 2
	fi

[gemini-code-assist]

Inside the subshell, set -e is not explicitly enabled because set +e was called on the parent shell at line 22. If find_available_zone.sh does not internally set set -e and encounters a non-fatal command failure, the subshell will continue executing and write empty or incomplete values to /tmp/zone_export.sh, exiting with status 0.

Additionally, it is safer to double-quote the exported variables to prevent any potential word splitting or shell expansion issues.

Please enable set -e explicitly inside the subshell and quote the exported values.

Suggested change

	(
	source /workspace/tools/cloud-build/find_available_zone.sh
	# If it succeeds, these lines will execute and save the exports.
	echo "export ZONE=${ZONE}" >/tmp/zone_export.sh
	echo "export PROVISIONING_MODEL=${PROVISIONING_MODEL}" >>/tmp/zone_export.sh
	) 2>&1 | tee /tmp/zone_output.log
	(
	set -e
	source /workspace/tools/cloud-build/find_available_zone.sh
	# If it succeeds, these lines will execute and save the exports.
	echo "export ZONE=\"${ZONE}\"" >/tmp/zone_export.sh
	echo "export PROVISIONING_MODEL=\"${PROVISIONING_MODEL}\"" >>/tmp/zone_export.sh
	) 2>&1 | tee /tmp/zone_output.log