You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
+18-18Lines changed: 18 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
# Submitting Jobs on Torch
2
2
3
3
:::tip [Beginner tutorial available]
4
-
If you are new to using HPC resources and would like to learn about the principles of using the `SLURM` scheduler for submitting batch jobs, please refer to cthis section](../13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx). This section focuses on the specifics of the Torch cluster and assumes familiarity with the tutorial.
4
+
If you are new to using HPC resources and would like to learn about the principles of using the `SLURM` scheduler for submitting batch jobs, please refer to [this section](../13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx). This section focuses on the specifics of the Torch cluster and assumes familiarity with the tutorial.
5
5
:::
6
6
7
7
@@ -11,38 +11,38 @@ Instead of running interactive jobs on Login nodes, users can run interactive jo
11
11
12
12
13
13
## Partition Management
14
-
Partitions control stakeholder resource access.
15
-
No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition QoS.
16
-
Jobs within the same partition cannot exceed their assigned resources (QOSGrpGRES).
17
-
User GPU Quotas
18
-
Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (QOSMaxGRESPerUser).
14
+
- Partitions control stakeholder resource access.
15
+
- No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition QoS.
16
+
- Jobs within the same partition cannot exceed their assigned resources (QOSGrpGRES).
17
+
- User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (QOSMaxGRESPerUser).
19
18
## Job Submission
20
-
Do not specify partitions manually.
21
-
Only request the compute resources (e.g., GPUs, CPUs, memory).
22
-
Scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
19
+
-Do not specify partitions manually.
20
+
-Only request the compute resources (e.g., GPUs, CPUs, memory).
21
+
-Scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
23
22
24
23
:::warning [Low GPU Utilization Policy]
25
24
Jobs with low GPU utilization will be automatically cancelled.
26
25
The exact threshold is TBD, but enforcement will be very aggressive.
27
26
:::
28
27
29
-
New Slurm configuration introduces preemption partitions enabling:
30
-
Non-stakeholders to temporarily use stakeholder resources
31
-
One stakeholder group to temporarily use another group’s resources
32
-
Stakeholders retain normal access to their own resources.
33
-
If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (cancelled) once stakeholders submit new jobs.
34
-
Public users are allowed to use stakeholder resources only with preemption partitions.
28
+
## New Slurm configuration introduces preemption partitions enabling:
29
+
-Non-stakeholders to temporarily use stakeholder resources
30
+
-One stakeholder group to temporarily use another group’s resources
31
+
-Stakeholders retain normal access to their own resources.
32
+
-If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (cancelled) once stakeholders submit new jobs.
33
+
-Public users are allowed to use stakeholder resources only with preemption partitions.
35
34
36
35
:::info [Pre-Emption Policy]
37
36
Jobs become eligible for pre-emption after 1 hour of runtime.
38
37
Jobs will not be cancelled within the first hour.
38
+
:::
39
39
40
-
How to Enable Pre-Emption
41
-
To allow jobs in both normal and pre-emption partitions:
40
+
## How to Enable Pre-Emption
41
+
-To allow jobs in both normal and pre-emption partitions:
42
42
#SBATCH --comment="preemption=yes;requeue=true"
43
43
Jobs in stakeholder partitions will not be cancelled, but those in pre-emption partitions may be.
44
44
Cancelled jobs will be requeued automatically with requeue=true
0 commit comments