fix(slurm): fail over after GCE capacity errors

Author: reidlevesque

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/README.md

@@ -395,6 +395,7 @@ limitations under the License.
 | <a name="input_bucket_dir"></a> [bucket\_dir](#input\_bucket\_dir) | Bucket directory for cluster files to be put into. If not specified, then one will be chosen based on slurm\_cluster\_name. | `string` | `null` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket.<br/>Ignored when 'create\_bucket' is true. | `string` | `null` | no |
 | <a name="input_can_ip_forward"></a> [can\_ip\_forward](#input\_can\_ip\_forward) | Enable IP forwarding, for NAT instances for example. | `bool` | `false` | no |
+| <a name="input_capacity_circuit_breaker"></a> [capacity\_circuit\_breaker](#input\_capacity\_circuit\_breaker) | Controls automatic failover away from a GCE nodeset after a capacity error.<br/>When enabled, powered-down nodes in the affected nodeset are drained while<br/>a small probe set is retried after an exponential cooldown. A successful<br/>probe makes the nodeset schedulable again. If no workload can use the probe<br/>set, the nodeset is automatically reset after a bounded grace period that<br/>accounts for Slurm's ResumeTimeout. | <pre>object({<br/>    enabled                  = optional(bool, false)<br/>    initial_cooldown_seconds = optional(number, 300)<br/>    max_cooldown_seconds     = optional(number, 1800)<br/>    probe_count              = optional(number, 1)<br/>  })</pre> | `{}` | no |
 | <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
 | <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters) | <pre>object({<br/>    no_comma_params         = optional(bool, false)<br/>    private_data            = optional(list(string))<br/>    scheduler_parameters    = optional(list(string))<br/>    resume_rate             = optional(number)<br/>    resume_timeout          = optional(number)<br/>    suspend_rate            = optional(number)<br/>    suspend_timeout         = optional(number)<br/>    slurmd_timeout          = optional(number)<br/>    unkillable_step_timeout = optional(number)<br/>    topology_plugin         = optional(string)<br/>    topology_param          = optional(string)<br/>    tree_width              = optional(number)<br/>    prolog_flags            = optional(string)<br/>    switch_type             = optional(string)<br/>  })</pre> | `{}` | no |
 | <a name="input_cloudsql"></a> [cloudsql](#input\_cloudsql) | Use this database instead of the one on the controller.<br/>  server\_ip : Address of the database server.<br/>  user      : The user to access the database as.<br/>  password  : The password, given the user, to access the given database. (sensitive)<br/>  db\_name   : The database to access.<br/>  user\_managed\_replication : The list of location and (optional) kms\_key\_name for secret | <pre>object({<br/>    server_ip = string<br/>    user      = string<br/>    password  = string # sensitive<br/>    db_name   = string<br/>    user_managed_replication = optional(list(object({<br/>      location     = string<br/>      kms_key_name = optional(string)<br/>    })), [])<br/>  })</pre> | `null` | no |

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/README.md

@@ -69,6 +69,7 @@ No modules.
 | <a name="input_backup_controller_key_timeout"></a> [backup\_controller\_key\_timeout](#input\_backup\_controller\_key\_timeout) | The time in seconds for the backup controller to wait for the shared key to become available. | `number` | `300` | no |
 | <a name="input_bucket_dir"></a> [bucket\_dir](#input\_bucket\_dir) | Bucket directory for cluster files to be put into. | `string` | `null` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket to use. | `string` | n/a | yes |
+| <a name="input_capacity_circuit_breaker"></a> [capacity\_circuit\_breaker](#input\_capacity\_circuit\_breaker) | Controls automatic nodeset failover after GCE capacity errors. | <pre>object({<br/>    enabled                  = optional(bool, false)<br/>    initial_cooldown_seconds = optional(number, 300)<br/>    max_cooldown_seconds     = optional(number, 1800)<br/>    probe_count              = optional(number, 1)<br/>  })</pre> | `{}` | no |
 | <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
 | <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Default behavior defined in scripts/conf.py | <pre>object({<br/>    no_comma_params         = optional(bool, false)<br/>    private_data            = optional(list(string))<br/>    scheduler_parameters    = optional(list(string))<br/>    resume_rate             = optional(number)<br/>    resume_timeout          = optional(number)<br/>    suspend_rate            = optional(number)<br/>    suspend_timeout         = optional(number)<br/>    slurmd_timeout          = optional(number)<br/>    unkillable_step_timeout = optional(number)<br/>    topology_plugin         = optional(string)<br/>    topology_param          = optional(string)<br/>    tree_width              = optional(number)<br/>    prolog_flags            = optional(string)<br/>    switch_type             = optional(string)<br/>  })</pre> | `{}` | no |
 | <a name="input_cloudsql_secret"></a> [cloudsql\_secret](#input\_cloudsql\_secret) | Secret URI to cloudsql secret. | `string` | `null` | no |

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/main.tf

@@ -68,11 +68,12 @@ locals {
     slurm_key_mount = var.slurm_key_mount
 
     # slurm conf
-    prolog_scripts      = [for k, v in google_storage_bucket_object.prolog_scripts : k]
-    epilog_scripts      = [for k, v in google_storage_bucket_object.epilog_scripts : k]
-    task_prolog_scripts = [for k, v in google_storage_bucket_object.task_prolog_scripts : k]
-    task_epilog_scripts = [for k, v in google_storage_bucket_object.task_epilog_scripts : k]
-    cloud_parameters    = var.cloud_parameters
+    prolog_scripts           = [for k, v in google_storage_bucket_object.prolog_scripts : k]
+    epilog_scripts           = [for k, v in google_storage_bucket_object.epilog_scripts : k]
+    task_prolog_scripts      = [for k, v in google_storage_bucket_object.task_prolog_scripts : k]
+    task_epilog_scripts      = [for k, v in google_storage_bucket_object.task_epilog_scripts : k]
+    cloud_parameters         = var.cloud_parameters
+    capacity_circuit_breaker = var.capacity_circuit_breaker
 
     # hybrid
     hybrid                        = var.enable_hybrid
@@ -186,6 +187,7 @@ locals {
     "tools/gpu-test",
     "tools/task-epilog",
     "tools/task-prolog",
+    "capacity_circuit.py",
     "conf.py",
     "conf_v2411.py",
     "file_cache.py",
@@ -212,6 +214,7 @@ locals {
     "tools/gpu-test",
     "tools/task-epilog",
     "tools/task-prolog",
+    "capacity_circuit.py",
     "conf.py",
     "conf_v2411.py",
     "file_cache.py",