Skip to content

fix(slurm): fail over after GCE capacity errors #5909

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
reidlevesque wants to merge 3 commits into
base: develop
Choose a base branch
from
Draft

fix(slurm): fail over after GCE capacity errors #5909

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
46d94c6
fix(slurm): fail over after GCE capacity errors
reidlevesque Jul 6, 2026
5dbbb59
fix(slurm): harden capacity circuit state reads
reidlevesque Jul 6, 2026
8c3614c
fix(slurm): replay circuit transitions safely
reidlevesque Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions community/modules/scheduler/schedmd-slurm-gcp-v6-controller/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,7 @@ limitations under the License.
| <a name="input_bucket_dir"></a> [bucket\_dir](#input\_bucket\_dir) | Bucket directory for cluster files to be put into. If not specified, then one will be chosen based on slurm\_cluster\_name. | `string` | `null` | no |
| <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket.<br/>Ignored when 'create\_bucket' is true. | `string` | `null` | no |
| <a name="input_can_ip_forward"></a> [can\_ip\_forward](#input\_can\_ip\_forward) | Enable IP forwarding, for NAT instances for example. | `bool` | `false` | no |
| <a name="input_capacity_circuit_breaker"></a> [capacity\_circuit\_breaker](#input\_capacity\_circuit\_breaker) | Controls automatic failover away from a GCE nodeset after a capacity error.<br/>When enabled, powered-down nodes in the affected nodeset are drained while<br/>a small probe set is retried after an exponential cooldown. A successful<br/>probe makes the nodeset schedulable again. If no workload can use the probe<br/>set, the nodeset is automatically reset after a bounded grace period that<br/>accounts for Slurm's ResumeTimeout. | <pre>object({<br/> enabled = optional(bool, false)<br/> initial_cooldown_seconds = optional(number, 300)<br/> max_cooldown_seconds = optional(number, 1800)<br/> probe_count = optional(number, 1)<br/> })</pre> | `{}` | no |
| <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters) | <pre>object({<br/> no_comma_params = optional(bool, false)<br/> private_data = optional(list(string))<br/> scheduler_parameters = optional(list(string))<br/> resume_rate = optional(number)<br/> resume_timeout = optional(number)<br/> suspend_rate = optional(number)<br/> suspend_timeout = optional(number)<br/> slurmd_timeout = optional(number)<br/> unkillable_step_timeout = optional(number)<br/> topology_plugin = optional(string)<br/> topology_param = optional(string)<br/> tree_width = optional(number)<br/> prolog_flags = optional(string)<br/> switch_type = optional(string)<br/> })</pre> | `{}` | no |
| <a name="input_cloudsql"></a> [cloudsql](#input\_cloudsql) | Use this database instead of the one on the controller.<br/> server\_ip : Address of the database server.<br/> user : The user to access the database as.<br/> password : The password, given the user, to access the given database. (sensitive)<br/> db\_name : The database to access.<br/> user\_managed\_replication : The list of location and (optional) kms\_key\_name for secret | <pre>object({<br/> server_ip = string<br/> user = string<br/> password = string # sensitive<br/> db_name = string<br/> user_managed_replication = optional(list(object({<br/> location = string<br/> kms_key_name = optional(string)<br/> })), [])<br/> })</pre> | `null` | no |
Expand Down
1 change: 1 addition & 0 deletions ...modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ No modules.
| <a name="input_backup_controller_key_timeout"></a> [backup\_controller\_key\_timeout](#input\_backup\_controller\_key\_timeout) | The time in seconds for the backup controller to wait for the shared key to become available. | `number` | `300` | no |
| <a name="input_bucket_dir"></a> [bucket\_dir](#input\_bucket\_dir) | Bucket directory for cluster files to be put into. | `string` | `null` | no |
| <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket to use. | `string` | n/a | yes |
| <a name="input_capacity_circuit_breaker"></a> [capacity\_circuit\_breaker](#input\_capacity\_circuit\_breaker) | Controls automatic nodeset failover after GCE capacity errors. | <pre>object({<br/> enabled = optional(bool, false)<br/> initial_cooldown_seconds = optional(number, 300)<br/> max_cooldown_seconds = optional(number, 1800)<br/> probe_count = optional(number, 1)<br/> })</pre> | `{}` | no |
| <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Default behavior defined in scripts/conf.py | <pre>object({<br/> no_comma_params = optional(bool, false)<br/> private_data = optional(list(string))<br/> scheduler_parameters = optional(list(string))<br/> resume_rate = optional(number)<br/> resume_timeout = optional(number)<br/> suspend_rate = optional(number)<br/> suspend_timeout = optional(number)<br/> slurmd_timeout = optional(number)<br/> unkillable_step_timeout = optional(number)<br/> topology_plugin = optional(string)<br/> topology_param = optional(string)<br/> tree_width = optional(number)<br/> prolog_flags = optional(string)<br/> switch_type = optional(string)<br/> })</pre> | `{}` | no |
| <a name="input_cloudsql_secret"></a> [cloudsql\_secret](#input\_cloudsql\_secret) | Secret URI to cloudsql secret. | `string` | `null` | no |
Expand Down
13 changes: 8 additions & 5 deletions community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,12 @@ locals {
slurm_key_mount = var.slurm_key_mount

# slurm conf
prolog_scripts = [for k, v in google_storage_bucket_object.prolog_scripts : k]
epilog_scripts = [for k, v in google_storage_bucket_object.epilog_scripts : k]
task_prolog_scripts = [for k, v in google_storage_bucket_object.task_prolog_scripts : k]
task_epilog_scripts = [for k, v in google_storage_bucket_object.task_epilog_scripts : k]
cloud_parameters = var.cloud_parameters
prolog_scripts = [for k, v in google_storage_bucket_object.prolog_scripts : k]
epilog_scripts = [for k, v in google_storage_bucket_object.epilog_scripts : k]
task_prolog_scripts = [for k, v in google_storage_bucket_object.task_prolog_scripts : k]
task_epilog_scripts = [for k, v in google_storage_bucket_object.task_epilog_scripts : k]
cloud_parameters = var.cloud_parameters
capacity_circuit_breaker = var.capacity_circuit_breaker

# hybrid
hybrid = var.enable_hybrid
Expand Down Expand Up @@ -186,6 +187,7 @@ locals {
"tools/gpu-test",
"tools/task-epilog",
"tools/task-prolog",
"capacity_circuit.py",
"conf.py",
"conf_v2411.py",
"file_cache.py",
Expand All @@ -212,6 +214,7 @@ locals {
"tools/gpu-test",
"tools/task-epilog",
"tools/task-prolog",
"capacity_circuit.py",
"conf.py",
"conf_v2411.py",
"file_cache.py",
Expand Down
Loading
Loading