You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
+24-17Lines changed: 24 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,31 +4,35 @@
4
4
If you are new to using HPC resources and would like to learn about the principles of using the `SLURM` scheduler for submitting batch jobs, please refer to [this section](../13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx). This section focuses on the specifics of the Torch cluster and assumes familiarity with the tutorial.
5
5
:::
6
6
7
-
## Partitions on Torch
8
-
`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`. Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`).
7
+
## Partitions
8
+
9
+
`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`([QualityOfService](https://slurm.schedmd.com/qos.html)).
9
10
10
11
:::tip Partitions
11
12
Do not specify partitions manually, except for preemption which is described later.
12
13
:::
13
14
14
-
Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions.
15
+
## Resource limits and restrictions
16
+
Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`).
17
+
18
+
Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions. Refer to the section below for details on preemptible jobs.
15
19
16
20
## Job Submission on Torch
17
-
As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory). The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
21
+
As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory) needed for the job. Requesting too many resources can prevent your job from being scheduled within an adequate time. The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
18
22
19
-
:::warning Low GPU Utilization Policy
23
+
:::danger Low GPU Utilization Policy
20
24
Jobs with low GPU utilization will be automatically canceled. The exact threshold is TBD, but enforcement will be very aggressive.
21
25
:::
22
26
23
27
## Preemptible jobs on Torch
24
28
On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs.
25
29
26
-
:::info Preemption Policy
30
+
:::warning Preemption Policy
27
31
Jobs become eligible for preemption after 1 hour of runtime. Jobs will not be canceled within the first hour.
28
32
:::
29
33
30
34
The preemption order is:
31
-
- Stakeholder jobs have highest priority, can preempt GPU jobs from public users or other stakeholders
35
+
- Stakeholder jobs (highest priority), these can preempt GPU jobs from public users or other stakeholders
32
36
- GPU jobs can preempt CPU-only jobs running on GPU nodes
33
37
- Partition Assignment Order
34
38
- PI stakeholder partitions
@@ -39,26 +43,29 @@ Applies separately to both GPU types: L40S first, then H200
39
43
40
44
41
45
To allow jobs in both normal and preemption partitions:
42
-
```sh
46
+
```
43
47
#SBATCH --comment="preemption=yes;requeue=true"
44
48
```
45
49
Jobs in stakeholder partitions will not be canceled, but those in preemption partitions may be. Canceled jobs will be re-queued automatically with `requeue=true`. To use only preemption partitions:
0 commit comments