Skip to content

Commit b50fb4a

Browse files
committed
formatting tweak
modified: docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
1 parent 3050fae commit b50fb4a

1 file changed

Lines changed: 24 additions & 17 deletions

File tree

docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md

Lines changed: 24 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,31 +4,35 @@
44
If you are new to using HPC resources and would like to learn about the principles of using the `SLURM` scheduler for submitting batch jobs, please refer to [this section](../13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx). This section focuses on the specifics of the Torch cluster and assumes familiarity with the tutorial.
55
:::
66

7-
## Partitions on Torch
8-
`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`. Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`).
7+
## Partitions
8+
9+
`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`([QualityOfService](https://slurm.schedmd.com/qos.html)).
910

1011
:::tip Partitions
1112
Do not specify partitions manually, except for preemption which is described later.
1213
:::
1314

14-
Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions.
15+
## Resource limits and restrictions
16+
Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`).
17+
18+
Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions. Refer to the section below for details on preemptible jobs.
1519

1620
## Job Submission on Torch
17-
As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory). The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
21+
As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory) needed for the job. Requesting too many resources can prevent your job from being scheduled within an adequate time. The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
1822

19-
:::warning Low GPU Utilization Policy
23+
:::danger Low GPU Utilization Policy
2024
Jobs with low GPU utilization will be automatically canceled. The exact threshold is TBD, but enforcement will be very aggressive.
2125
:::
2226

2327
## Preemptible jobs on Torch
2428
On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs.
2529

26-
:::info Preemption Policy
30+
:::warning Preemption Policy
2731
Jobs become eligible for preemption after 1 hour of runtime. Jobs will not be canceled within the first hour.
2832
:::
2933

3034
The preemption order is:
31-
- Stakeholder jobs have highest priority, can preempt GPU jobs from public users or other stakeholders
35+
- Stakeholder jobs (highest priority), these can preempt GPU jobs from public users or other stakeholders
3236
- GPU jobs can preempt CPU-only jobs running on GPU nodes
3337
- Partition Assignment Order
3438
- PI stakeholder partitions
@@ -39,26 +43,29 @@ Applies separately to both GPU types: L40S first, then H200
3943

4044

4145
To allow jobs in both normal and preemption partitions:
42-
```sh
46+
```
4347
#SBATCH --comment="preemption=yes;requeue=true"
4448
```
4549
Jobs in stakeholder partitions will not be canceled, but those in preemption partitions may be. Canceled jobs will be re-queued automatically with `requeue=true`. To use only preemption partitions:
46-
```sh
50+
```
4751
#SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true"
4852
```
4953
Jobs with preemption partitions only might be allowed to use more resources
5054

51-
## Enable GPU MPS
52-
Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, allows multiple GPU jobs to share a single GPU concurrently
53-
```sh
55+
## Advanced options
56+
57+
### GPU MPS
58+
Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, as this allows multiple GPU jobs to share a single GPU concurrently by:
59+
```
5460
#SBATCH --comment="gpu_mps=yes"
5561
```
56-
Mount Memory as Disk
57-
Create a small RAM disk for fast I/O operations
58-
```sh
62+
### RAM disk
63+
A portion of the RAM available can be mounted as a disk for fast `I/O` operations:
64+
```
5965
#SBATCH --comment="ram_disk=1GB"
6066
```
61-
Example with preemption, GPU MPS, and RAM disk enabled:
62-
```sh
67+
### GPU MPS & RAM Disk in a preemptible job
68+
Both of these can be combined with preemption as shown:
69+
```
6370
#SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true;gpu_mps=yes;ram_disk=1GB"
6471
```

0 commit comments

Comments
 (0)