Skip to content

Commit 9a9d7d9

Browse files
committed
added info on max jobs per user
1 parent 5fccc19 commit 9a9d7d9

1 file changed

Lines changed: 17 additions & 1 deletion

File tree

docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,28 @@ Non-stakeholders to temporarily use stakeholder resources (a stakeholder group t
2828
:::
2929

3030
## Job Submission on Torch
31-
As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory) needed for the job. Requesting too many resources can prevent your job from being scheduled within an adequate time. The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
31+
As stated in the tutorial, be sure to only request the compute resources (e.g., GPUs, CPUs, memory) needed for the job. Requesting too many resources can prevent your job from being scheduled within an adequate time. The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests.
3232

3333
:::danger Low GPU Utilization Policy
3434
Jobs with low GPU utilization will be automatically canceled. The exact threshold is TBD, but enforcement will be very aggressive.
3535
:::
3636

37+
:::info Maximum number of jobs a user can submit
38+
This depends on the requested wall time.<br />
39+
You can check it with the following command:
40+
```bash
41+
[NetID@torch-login-b-1 ~]$ sacctmgr show qos cpu48,cpu168,gpu48,gpu168,interactive format=name%16,maxTresPU%32,MaxWall,MaxSubmitJobsPerUser
42+
Name MaxTRESPU MaxWall MaxSubmitPU
43+
---------------- -------------------------------- ----------- -----------
44+
cpu168 cpu=1000,mem=2000G 7-00:00:00 500
45+
cpu48 cpu=3000,mem=6000G 2-00:00:00 5000
46+
gpu168 gres/gpu=4 7-00:00:00 50
47+
gpu48 gres/gpu=16 2-00:00:00 2000
48+
interactive cpu=16,mem=60G 06:00:00 20
49+
```
50+
The values above are just an example and they may change without notice, so please check within your account for the most up to date results.
51+
:::
52+
3753
## Preemptible jobs on Torch
3854
On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs.
3955

0 commit comments

Comments
 (0)