Skip to content

Commit cbe6a06

Browse files
committed
formatting tweak
modified: docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
1 parent 3050fae commit cbe6a06

1 file changed

Lines changed: 14 additions & 10 deletions

File tree

docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,30 +5,34 @@ If you are new to using HPC resources and would like to learn about the principl
55
:::
66

77
## Partitions on Torch
8-
`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`. Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`).
8+
`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`[QualityOfService](https://slurm.schedmd.com/qos.html).
99

1010
:::tip Partitions
1111
Do not specify partitions manually, except for preemption which is described later.
1212
:::
1313

14+
### Resource limits and restrictions
15+
Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`).
16+
17+
1418
Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions.
1519

1620
## Job Submission on Torch
1721
As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory). The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
1822

19-
:::warning Low GPU Utilization Policy
23+
:::danger Low GPU Utilization Policy
2024
Jobs with low GPU utilization will be automatically canceled. The exact threshold is TBD, but enforcement will be very aggressive.
2125
:::
2226

2327
## Preemptible jobs on Torch
2428
On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs.
2529

26-
:::info Preemption Policy
30+
:::warning Preemption Policy
2731
Jobs become eligible for preemption after 1 hour of runtime. Jobs will not be canceled within the first hour.
2832
:::
2933

3034
The preemption order is:
31-
- Stakeholder jobs have highest priority, can preempt GPU jobs from public users or other stakeholders
35+
- Stakeholder jobs (highest priority), these can preempt GPU jobs from public users or other stakeholders
3236
- GPU jobs can preempt CPU-only jobs running on GPU nodes
3337
- Partition Assignment Order
3438
- PI stakeholder partitions
@@ -39,26 +43,26 @@ Applies separately to both GPU types: L40S first, then H200
3943

4044

4145
To allow jobs in both normal and preemption partitions:
42-
```sh
46+
```
4347
#SBATCH --comment="preemption=yes;requeue=true"
4448
```
4549
Jobs in stakeholder partitions will not be canceled, but those in preemption partitions may be. Canceled jobs will be re-queued automatically with `requeue=true`. To use only preemption partitions:
46-
```sh
50+
```
4751
#SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true"
4852
```
4953
Jobs with preemption partitions only might be allowed to use more resources
5054

5155
## Enable GPU MPS
52-
Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, allows multiple GPU jobs to share a single GPU concurrently
53-
```sh
56+
Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, as this allows multiple GPU jobs to share a single GPU concurrently by:
57+
```
5458
#SBATCH --comment="gpu_mps=yes"
5559
```
5660
Mount Memory as Disk
5761
Create a small RAM disk for fast I/O operations
58-
```sh
62+
```
5963
#SBATCH --comment="ram_disk=1GB"
6064
```
6165
Example with preemption, GPU MPS, and RAM disk enabled:
62-
```sh
66+
```
6367
#SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true;gpu_mps=yes;ram_disk=1GB"
6468
```

0 commit comments

Comments
 (0)