Merge pull request #4601 from Neelabh94/step_timeout

Adding unkillable_step_timeout (UnkillableStepTimeout) to the H4D blueprint.

Author: Neelabh94

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/README.md

@@ -313,7 +313,7 @@ limitations under the License.
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket.<br/>Ignored when 'create\_bucket' is true. | `string` | `null` | no |
 | <a name="input_can_ip_forward"></a> [can\_ip\_forward](#input\_can\_ip\_forward) | Enable IP forwarding, for NAT instances for example. | `bool` | `false` | no |
 | <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
-| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters) | <pre>object({<br/>    no_comma_params      = optional(bool, false)<br/>    private_data         = optional(list(string))<br/>    scheduler_parameters = optional(list(string))<br/>    resume_rate          = optional(number)<br/>    resume_timeout       = optional(number)<br/>    suspend_rate         = optional(number)<br/>    suspend_timeout      = optional(number)<br/>    slurmd_timeout       = optional(number)<br/>    topology_plugin      = optional(string)<br/>    topology_param       = optional(string)<br/>    tree_width           = optional(number)<br/>    prolog_flags         = optional(string)<br/>    switch_type          = optional(string)<br/>  })</pre> | `{}` | no |
+| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters) | <pre>object({<br/>    no_comma_params         = optional(bool, false)<br/>    private_data            = optional(list(string))<br/>    scheduler_parameters    = optional(list(string))<br/>    resume_rate             = optional(number)<br/>    resume_timeout          = optional(number)<br/>    suspend_rate            = optional(number)<br/>    suspend_timeout         = optional(number)<br/>    slurmd_timeout          = optional(number)<br/>    unkillable_step_timeout = optional(number)<br/>    topology_plugin         = optional(string)<br/>    topology_param          = optional(string)<br/>    tree_width              = optional(number)<br/>    prolog_flags            = optional(string)<br/>    switch_type             = optional(string)<br/>  })</pre> | `{}` | no |
 | <a name="input_cloudsql"></a> [cloudsql](#input\_cloudsql) | Use this database instead of the one on the controller.<br/>  server\_ip : Address of the database server.<br/>  user      : The user to access the database as.<br/>  password  : The password, given the user, to access the given database. (sensitive)<br/>  db\_name   : The database to access.<br/>  user\_managed\_replication : The list of location and (optional) kms\_key\_name for secret | <pre>object({<br/>    server_ip = string<br/>    user      = string<br/>    password  = string # sensitive<br/>    db_name   = string<br/>    user_managed_replication = optional(list(object({<br/>      location     = string<br/>      kms_key_name = optional(string)<br/>    })), [])<br/>  })</pre> | `null` | no |
 | <a name="input_compute_startup_script"></a> [compute\_startup\_script](#input\_compute\_startup\_script) | DEPRECATED: `compute_startup_script` has been deprecated.<br/>Use `startup_script` of nodeset module instead. | `any` | `null` | no |
 | <a name="input_compute_startup_scripts_timeout"></a> [compute\_startup\_scripts\_timeout](#input\_compute\_startup\_scripts\_timeout) | The timeout (seconds) applied to each startup script in compute nodes. If<br/>any script exceeds this timeout, then the instance setup process is considered<br/>failed and handled accordingly.<br/><br/>NOTE: When set to 0, the timeout is considered infinite and thus disabled. | `number` | `300` | no |

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/README.md

@@ -68,7 +68,7 @@ No modules.
 | <a name="input_bucket_dir"></a> [bucket\_dir](#input\_bucket\_dir) | Bucket directory for cluster files to be put into. | `string` | `null` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Name of GCS bucket to use. | `string` | n/a | yes |
 | <a name="input_cgroup_conf_tpl"></a> [cgroup\_conf\_tpl](#input\_cgroup\_conf\_tpl) | Slurm cgroup.conf template file path. | `string` | `null` | no |
-| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Default behavior defined in scripts/conf.py | <pre>object({<br/>    no_comma_params      = optional(bool, false)<br/>    private_data         = optional(list(string))<br/>    scheduler_parameters = optional(list(string))<br/>    resume_rate          = optional(number)<br/>    resume_timeout       = optional(number)<br/>    suspend_rate         = optional(number)<br/>    suspend_timeout      = optional(number)<br/>    slurmd_timeout       = optional(number)<br/>    topology_plugin      = optional(string)<br/>    topology_param       = optional(string)<br/>    tree_width           = optional(number)<br/>    prolog_flags         = optional(string)<br/>    switch_type          = optional(string)<br/>  })</pre> | `{}` | no |
+| <a name="input_cloud_parameters"></a> [cloud\_parameters](#input\_cloud\_parameters) | cloud.conf options. Default behavior defined in scripts/conf.py | <pre>object({<br/>    no_comma_params         = optional(bool, false)<br/>    private_data            = optional(list(string))<br/>    scheduler_parameters    = optional(list(string))<br/>    resume_rate             = optional(number)<br/>    resume_timeout          = optional(number)<br/>    suspend_rate            = optional(number)<br/>    suspend_timeout         = optional(number)<br/>    slurmd_timeout          = optional(number)<br/>    unkillable_step_timeout = optional(number)<br/>    topology_plugin         = optional(string)<br/>    topology_param          = optional(string)<br/>    tree_width              = optional(number)<br/>    prolog_flags            = optional(string)<br/>    switch_type             = optional(string)<br/>  })</pre> | `{}` | no |
 | <a name="input_cloudsql_secret"></a> [cloudsql\_secret](#input\_cloudsql\_secret) | Secret URI to cloudsql secret. | `string` | `null` | no |
 | <a name="input_compute_startup_scripts_timeout"></a> [compute\_startup\_scripts\_timeout](#input\_compute\_startup\_scripts\_timeout) | The timeout (seconds) applied to each script in compute\_startup\_scripts. If<br/>any script exceeds this timeout, then the instance setup process is considered<br/>failed and handled accordingly.<br/><br/>NOTE: When set to 0, the timeout is considered infinite and thus disabled. | `number` | `300` | no |
 | <a name="input_controller_network_attachment"></a> [controller\_network\_attachment](#input\_controller\_network\_attachment) | SelfLink for NetworkAttachment to be attached to the controller, if any. | `string` | `null` | no |

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/scripts/conf.py

@@ -133,6 +133,7 @@ def get(key, default):
         "SuspendRate": get("suspend_rate", 0),
         "SuspendTimeout": get("suspend_timeout", 300),
         "SlurmdTimeout": get("slurmd_timeout", 300),
+        "UnkillableStepTimeout": get("unkillable_step_timeout", 300),
         "TreeWidth": get("tree_width", default_tree_width),
         "JobSubmitPlugins": "lua" if any_tpu else None,
         "TopologyPlugin": topology_plugin(lkp),

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/scripts/tests/test_conf.py

@@ -101,6 +101,7 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=0
 SuspendTimeout=300
 SlurmdTimeout=300
+UnkillableStepTimeout=300
 TreeWidth=128
 TopologyPlugin=topology/tree
 TopologyParam=SwitchAsNodeRank"""),
@@ -114,6 +115,7 @@ def test_dict_to_conf(value: dict, want: str):
                 "resume_timeout": None,
                 "suspend_rate": None,
                 "suspend_timeout": None,
+                "unkillable_step_timeout": None,
                 "slurmd_timeout": None,
                 "topology_plugin": None,
                 "topology_param": None,
@@ -129,6 +131,7 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=0
 SuspendTimeout=300
 SlurmdTimeout=300
+UnkillableStepTimeout=300
 TreeWidth=128
 TopologyPlugin=topology/tree
 TopologyParam=SwitchAsNodeRank"""),
@@ -151,7 +154,8 @@ def test_dict_to_conf(value: dict, want: str):
                 "suspend_rate": 3,
                 "suspend_timeout": 4,
                 "slurmd_timeout": 5,
-                "tree_width": 6,
+                "unkillable_step_timeout": 6,
+                "tree_width": 7,
                 "topology_plugin": "guess",
                 "topology_param": "yellow",
             },
@@ -166,7 +170,8 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=3
 SuspendTimeout=4
 SlurmdTimeout=5
-TreeWidth=6
+UnkillableStepTimeout=6
+TreeWidth=7
 TopologyPlugin=guess
 TopologyParam=yellow"""),
         (TstCfg(
@@ -187,6 +192,7 @@ def test_dict_to_conf(value: dict, want: str):
 SuspendRate=0
 SuspendTimeout=300
 SlurmdTimeout=300
+UnkillableStepTimeout=300
 TreeWidth=128
 TopologyPlugin=topology/tree
 TopologyParam=SwitchAsNodeRank"""),

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/modules/slurm_files/variables.tf

@@ -339,19 +339,20 @@ variable "nodeset_tpu" {
 variable "cloud_parameters" {
   description = "cloud.conf options. Default behavior defined in scripts/conf.py"
   type = object({
-    no_comma_params      = optional(bool, false)
-    private_data         = optional(list(string))
-    scheduler_parameters = optional(list(string))
-    resume_rate          = optional(number)
-    resume_timeout       = optional(number)
-    suspend_rate         = optional(number)
-    suspend_timeout      = optional(number)
-    slurmd_timeout       = optional(number)
-    topology_plugin      = optional(string)
-    topology_param       = optional(string)
-    tree_width           = optional(number)
-    prolog_flags         = optional(string)
-    switch_type          = optional(string)
+    no_comma_params         = optional(bool, false)
+    private_data            = optional(list(string))
+    scheduler_parameters    = optional(list(string))
+    resume_rate             = optional(number)
+    resume_timeout          = optional(number)
+    suspend_rate            = optional(number)
+    suspend_timeout         = optional(number)
+    slurmd_timeout          = optional(number)
+    unkillable_step_timeout = optional(number)
+    topology_plugin         = optional(string)
+    topology_param          = optional(string)
+    tree_width              = optional(number)
+    prolog_flags            = optional(string)
+    switch_type             = optional(string)
   })
   default  = {}
   nullable = false

community/modules/scheduler/schedmd-slurm-gcp-v6-controller/variables.tf

@@ -456,19 +456,20 @@ EOD
 variable "cloud_parameters" {
   description = "cloud.conf options. Defaults inherited from [Slurm GCP repo](https://github.com/GoogleCloudPlatform/slurm-gcp/blob/master/terraform/slurm_cluster/modules/slurm_files/README_TF.md#input_cloud_parameters)"
   type = object({
-    no_comma_params      = optional(bool, false)
-    private_data         = optional(list(string))
-    scheduler_parameters = optional(list(string))
-    resume_rate          = optional(number)
-    resume_timeout       = optional(number)
-    suspend_rate         = optional(number)
-    suspend_timeout      = optional(number)
-    slurmd_timeout       = optional(number)
-    topology_plugin      = optional(string)
-    topology_param       = optional(string)
-    tree_width           = optional(number)
-    prolog_flags         = optional(string)
-    switch_type          = optional(string)
+    no_comma_params         = optional(bool, false)
+    private_data            = optional(list(string))
+    scheduler_parameters    = optional(list(string))
+    resume_rate             = optional(number)
+    resume_timeout          = optional(number)
+    suspend_rate            = optional(number)
+    suspend_timeout         = optional(number)
+    slurmd_timeout          = optional(number)
+    unkillable_step_timeout = optional(number)
+    topology_plugin         = optional(string)
+    topology_param          = optional(string)
+    tree_width              = optional(number)
+    prolog_flags            = optional(string)
+    switch_type             = optional(string)
   })
   default  = {}
   nullable = false

examples/hpc-slurm-h4d.yaml

@@ -127,4 +127,4 @@ deployment_groups:
     settings:
       enable_controller_public_ips: true
       cloud_parameters:
-        slurmd_timeout: 900
+        unkillable_step_timeout: 900