add support for template values in `scontrol` command

[boegel]

I would like to look into adding support for using a templated control command, like using this in the bot configuration (app.cfg):

scontrol_command = /usr/bin/scontrol --clusters=%(cluster)s

We have a use case for this at HPC-UGent, where our Slurm setup consists of a set of separate clusters (not partitions).

This is not a problem for the squeue command, where we can use squeue --clusters=ALL to query all jobs across all clusters, but for scontrol we need to specify a specific cluster via the --clusters option (ALL doesn't work there).

To harvest the value to be used for the %(cluster_name)s template, the way we run the squeue command should be changed a bit.

We currently run something like:

/usr/bin/squeue --long --noheader --user $USER

and then harvest the job ID, job state, and job (state) reason from that (by extracting the 0th/4th/8th columns).

We can selectively harvest more fields and also using a better parsing approach by using something like:

/usr/bin/squeue --long --noheader --user $USER --Format JobId,Cluster,Partition,State,Reason

In the case of HPC-UGent, our bot is configured to use "/usr/bin/squeue --clusters=ALL ..." instead of "/usr/bin/squeue ...".

That way we also control the order of the fields it reports, which is better than assuming that the value at index 4 corresponds to job state.

In this case, 2nd column will provide extra info (cluster) which can be used to complete the potentially templated scontrol command. Likewise, 3rd column would be value for %(partition)s template.

[trz42]

Overall, I think this should be doable. There may be a few pitfalls though.

• The name of the cluster may be empty. $\rightarrow$ we could add the cluster at the last output element Maybe cluster name is always there. Was mixing it up with something else.
• Some values may be so long, so there's no whitespace separating values anymore. See example below

  squeue --long --noheader --user bot --Format JobId,Cluster,Partition,State,Reason
  61488               aws-rocky88-202310  x86-64-intel-caslakeRUNNING             None
  61487               aws-rocky88-202310  x86-64-intel-icelakeRUNNING             None
  

[boegel]

We could add a safeguard if cluster name is not there. And even then, you would simply end up with an scontrol command that's invalid. If you don't use the %(cluster_name)s template value, there's no problem if that field is empty.

For the 2nd problem, you can specify the width of each field. For example, to use 100 characters for each field:

$ squeue --clusters=ALL --long --noheader --user $USER --Format JobId:100,Cluster:100,Partition:100,State:100,Reason:100
53521781                                                                                            doduo                                                                                               doduo                                                                                               RUNNING                                                                                             None
53478249                                                                                            doduo                                                                                               doduo                                                                                               RUNNING                                                                                             None
20170935                                                                                            donphan                                                                                             donphan                                                                                             RUNNING                                                                                             None

You can also add a "suffix" to ensure you have a delimiter:

$ squeue --clusters=ALL --long --noheader --user $USER --Format JobId:100@,Cluster:100@,Partition:100@,State:100@,Reason:100@
53521781                                                                                            @doduo                                                                                               @doduo                                                                                               @RUNNING                                                                                             @None                                                                                                @
53478249                                                                                            @doduo                                                                                               @doduo                                                                                               @RUNNING                                                                                             @None                                                                                                @
20170935                                                                                            @donphan                                                                                             @donphan                                                                                             @RUNNING                                                                                             @None                                                                                                @

$ squeue --clusters=ALL --long --noheader --user $USER --Format JobId:100@,Cluster:100@,Partition:100@,State:100@,Reason:100@ | cut -f2 -d@
doduo
doduo
donphan

[casparvl]

As discussed, another option would be to use a wrapper for scontrol. That would mean no modification to the bot code is required. My main concern would be if those changes would introduce things that are rather system-specific - if it can be kept generic, I have no objections :)

Anyway, the wrapper should take the jobid argument, and first call squeue to figure out the cluster name. Then, it should call the 'real' scontrol with the --cluster=<cluster> argument`.

[laraPPr]

First attemp. With the comments above taken into account.
See:

• support for template values in scontrol command  #327

Not sure if adding Python placeholder values in the bot configuration file is the best way. But this pr resolves our issue and I do not think it will impact other bots.

[trz42]

Fixed by #327