Adding unkillable_step_timeout (UnkillableStepTimeut) as a default parameter to the H4D blueprint.

It has been observed that for large jobs on 32 node+ clusters, we often hit the QueuePair (QP) limit, leading to SIGKILL and subsequent cleaning of QPs.
The process of QP cleaning can take upto 12 minutes currently, which is more than twice the default UnkillableStepTimeout of 300 seconds.
This breach of timeout results in the affected nodes being marked as "DRAIN" rendering them unusable for any future jobs without explicit restart operations.

With this change we are providing an additional parameter within the blueprint to alter this value based on the workload and usage.
The change was tested on a 32 node cluster that ran Intel MPI benchmark and created >900K QPs before being killed. All the nodes successfully returned to "IDLE" state after QP cleaning

An earlier PR added SlurmdTimeout to the blueprint to mitigate this issue but intensive testing revealed that to be an incorrect approach, hence removing the slurmd_timeout(SlurmdTimeout)
parameter from the blueprint to avoid unnecessary clutter. Users can still specify slurmd_timeout in blueprints since it is still accepted as a valid blueprint param.

Author: Neelabh94

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/README.md

@@ -313,7 +313,7 @@ limitations under the License.
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket.<br/>Ignored when 'create\_bucket' is true. | `string` | `null` | no |
 | <a name="input_can_ip_forward"></a> [can\_ip\_forward](#input\_can\_ip\_forward) | Enable IP forwarding, for NAT instances for example. | `bool` | `false` | no |
 | <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
-| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters) | <pre>object({<br/>    no_comma_params      = optional(bool, false)<br/>    private_data         = optional(list(string))<br/>    scheduler_parameters = optional(list(string))<br/>    resume_rate          = optional(number)<br/>    resume_timeout       = optional(number)<br/>    suspend_rate         = optional(number)<br/>    suspend_timeout      = optional(number)<br/>    slurmd_timeout       = optional(number)<br/>    topology_plugin      = optional(string)<br/>    topology_param       = optional(string)<br/>    tree_width           = optional(number)<br/>    prolog_flags         = optional(string)<br/>    switch_type          = optional(string)<br/>  })</pre> | `{}` | no |
+| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters) | <pre>object({<br/>    no_comma_params         = optional(bool, false)<br/>    private_data            = optional(list(string))<br/>    scheduler_parameters    = optional(list(string))<br/>    resume_rate             = optional(number)<br/>    resume_timeout          = optional(number)<br/>    suspend_rate            = optional(number)<br/>    suspend_timeout         = optional(number)<br/>    slurmd_timeout          = optional(number)<br/>    unkillable_step_timeout = optional(number)<br/>    topology_plugin         = optional(string)<br/>    topology_param          = optional(string)<br/>    tree_width              = optional(number)<br/>    prolog_flags            = optional(string)<br/>    switch_type             = optional(string)<br/>  })</pre> | `{}` | no |
 | <a name="input_cloudsql"></a> [cloudsql](#input\_cloudsql) | Use this database instead of the one on the controller.<br/>  server\_ip : Address of the database server.<br/>  user      : The user to access the database as.<br/>  password  : The password, given the user, to access the given database. (sensitive)<br/>  db\_name   : The database to access.<br/>  user\_managed\_replication : The list of location and (optional) kms\_key\_name for secret | <pre>object({<br/>    server_ip = string<br/>    user      = string<br/>    password  = string # sensitive<br/>    db_name   = string<br/>    user_managed_replication = optional(list(object({<br/>      location     = string<br/>      kms_key_name = optional(string)<br/>    })), [])<br/>  })</pre> | `null` | no |
 | <a name="input_compute_startup_script"></a> [compute\_startup\_script](#input\_compute\_startup\_script) | DEPRECATED: `compute_startup_script` has been deprecated.<br/>Use `startup_script` of nodeset module instead. | `any` | `null` | no |
 | <a name="input_compute_startup_scripts_timeout"></a> [compute\_startup\_scripts\_timeout](#input\_compute\_startup\_scripts\_timeout) | The timeout (seconds) applied to each startup script in compute nodes. If<br/>any script exceeds this timeout, then the instance setup process is considered<br/>failed and handled accordingly.<br/><br/>NOTE: When set to 0, the timeout is considered infinite and thus disabled. | `number` | `300` | no |

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/README.md

@@ -68,7 +68,7 @@ No modules.
 | <a name="input_bucket_dir"></a> [bucket\_dir](#input\_bucket\_dir) | Bucket directory for cluster files to be put into. | `string` | `null` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket to use. | `string` | n/a | yes |
 | <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
-| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Default behavior defined in scripts/conf.py | <pre>object({<br/>    no_comma_params      = optional(bool, false)<br/>    private_data         = optional(list(string))<br/>    scheduler_parameters = optional(list(string))<br/>    resume_rate          = optional(number)<br/>    resume_timeout       = optional(number)<br/>    suspend_rate         = optional(number)<br/>    suspend_timeout      = optional(number)<br/>    slurmd_timeout       = optional(number)<br/>    topology_plugin      = optional(string)<br/>    topology_param       = optional(string)<br/>    tree_width           = optional(number)<br/>    prolog_flags         = optional(string)<br/>    switch_type          = optional(string)<br/>  })</pre> | `{}` | no |
+| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Default behavior defined in scripts/conf.py | <pre>object({<br/>    no_comma_params         = optional(bool, false)<br/>    private_data            = optional(list(string))<br/>    scheduler_parameters    = optional(list(string))<br/>    resume_rate             = optional(number)<br/>    resume_timeout          = optional(number)<br/>    suspend_rate            = optional(number)<br/>    suspend_timeout         = optional(number)<br/>    slurmd_timeout          = optional(number)<br/>    unkillable_step_timeout = optional(number)<br/>    topology_plugin         = optional(string)<br/>    topology_param          = optional(string)<br/>    tree_width              = optional(number)<br/>    prolog_flags            = optional(string)<br/>    switch_type             = optional(string)<br/>  })</pre> | `{}` | no |
 | <a name="input_cloudsql_secret"></a> [cloudsql\_secret](#input\_cloudsql\_secret) | Secret URI to cloudsql secret. | `string` | `null` | no |
 | <a name="input_compute_startup_scripts_timeout"></a> [compute\_startup\_scripts\_timeout](#input\_compute\_startup\_scripts\_timeout) | The timeout (seconds) applied to each script in compute\_startup\_scripts. If<br/>any script exceeds this timeout, then the instance setup process is considered<br/>failed and handled accordingly.<br/><br/>NOTE: When set to 0, the timeout is considered infinite and thus disabled. | `number` | `300` | no |
 | <a name="input_controller_network_attachment"></a> [controller\_network\_attachment](#input\_controller\_network\_attachment) | SelfLink for NetworkAttachment to be attached to the controller, if any. | `string` | `null` | no |

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/scripts/conf.py

@@ -133,6 +133,7 @@ def get(key, default):
         "SuspendRate": get("suspend_rate", 0),
         "SuspendTimeout": get("suspend_timeout", 300),
         "SlurmdTimeout": get("slurmd_timeout", 300),
+        "UnkillableStepTimeout": get("unkillable_step_timeout", 300),
         "TreeWidth": get("tree_width", default_tree_width),
         "JobSubmitPlugins": "lua" if any_tpu else None,
         "TopologyPlugin": topology_plugin(lkp),

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/scripts/tests/test_conf.py

@@ -101,6 +101,7 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=0
 SuspendTimeout=300
 SlurmdTimeout=300
+UnkillableStepTimeout=300
 TreeWidth=128
 TopologyPlugin=topology/tree
 TopologyParam=SwitchAsNodeRank"""),
@@ -114,6 +115,7 @@ def test_dict_to_conf(value: dict, want: str):
                 "resume_timeout": None,
                 "suspend_rate": None,
                 "suspend_timeout": None,
+                "unkillable_step_timeout": None,
                 "slurmd_timeout": None,
                 "topology_plugin": None,
                 "topology_param": None,
@@ -129,6 +131,7 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=0
 SuspendTimeout=300
 SlurmdTimeout=300
+UnkillableStepTimeout=300
 TreeWidth=128
 TopologyPlugin=topology/tree
 TopologyParam=SwitchAsNodeRank"""),
@@ -151,7 +154,8 @@ def test_dict_to_conf(value: dict, want: str):
                 "suspend_rate": 3,
                 "suspend_timeout": 4,
                 "slurmd_timeout": 5,
-                "tree_width": 6,
+                "unkillable_step_timeout": 6,
+                "tree_width": 7,
                 "topology_plugin": "guess",
                 "topology_param": "yellow",
             },
@@ -166,7 +170,8 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=3
 SuspendTimeout=4
 SlurmdTimeout=5
-TreeWidth=6
+UnkillableStepTimeout=6
+TreeWidth=7
 TopologyPlugin=guess
 TopologyParam=yellow"""),
         (TstCfg(
@@ -187,6 +192,7 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=0
 SuspendTimeout=300
 SlurmdTimeout=300
+UnkillableStepTimeout=300
 TreeWidth=128
 TopologyPlugin=topology/tree
 TopologyParam=SwitchAsNodeRank"""),

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/variables.tf

@@ -339,19 +339,20 @@ variable "nodeset_tpu" {
 variable "cloud_parameters" {
   description = "cloud.conf options. Default behavior defined in scripts/conf.py"
   type = object({
-    no_comma_params      = optional(bool, false)
-    private_data         = optional(list(string))
-    scheduler_parameters = optional(list(string))
-    resume_rate          = optional(number)
-    resume_timeout       = optional(number)
-    suspend_rate         = optional(number)
-    suspend_timeout      = optional(number)
-    slurmd_timeout       = optional(number)
-    topology_plugin      = optional(string)
-    topology_param       = optional(string)
-    tree_width           = optional(number)
-    prolog_flags         = optional(string)
-    switch_type          = optional(string)
+    no_comma_params         = optional(bool, false)
+    private_data            = optional(list(string))
+    scheduler_parameters    = optional(list(string))
+    resume_rate             = optional(number)
+    resume_timeout          = optional(number)
+    suspend_rate            = optional(number)
+    suspend_timeout         = optional(number)
+    slurmd_timeout          = optional(number)
+    unkillable_step_timeout = optional(number)
+    topology_plugin         = optional(string)
+    topology_param          = optional(string)
+    tree_width              = optional(number)
+    prolog_flags            = optional(string)
+    switch_type             = optional(string)
   })
   default  = {}
   nullable = false

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/variables.tf

@@ -456,19 +456,20 @@ EOD
 variable "cloud_parameters" {
   description = "cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters)"
   type = object({
-    no_comma_params      = optional(bool, false)
-    private_data         = optional(list(string))
-    scheduler_parameters = optional(list(string))
-    resume_rate          = optional(number)
-    resume_timeout       = optional(number)
-    suspend_rate         = optional(number)
-    suspend_timeout      = optional(number)
-    slurmd_timeout       = optional(number)
-    topology_plugin      = optional(string)
-    topology_param       = optional(string)
-    tree_width           = optional(number)
-    prolog_flags         = optional(string)
-    switch_type          = optional(string)
+    no_comma_params         = optional(bool, false)
+    private_data            = optional(list(string))
+    scheduler_parameters    = optional(list(string))
+    resume_rate             = optional(number)
+    resume_timeout          = optional(number)
+    suspend_rate            = optional(number)
+    suspend_timeout         = optional(number)
+    slurmd_timeout          = optional(number)
+    unkillable_step_timeout = optional(number)
+    topology_plugin         = optional(string)
+    topology_param          = optional(string)
+    tree_width              = optional(number)
+    prolog_flags            = optional(string)
+    switch_type             = optional(string)
   })
   default  = {}
   nullable = false

examples/hpc-slurm-h4d.yaml

@@ -127,4 +127,4 @@ deployment_groups:
     settings:
       enable_controller_public_ips: true
       cloud_parameters:
-        slurmd_timeout: 900
+        unkillable_step_timeout: 900