Merge pull request #363 from KempnerInstitute/user-group-quota

Add global GPU caps to cluster usage policies

Author: mmshad

kempner_computing_handbook/s1_high_performance_computing/efficient_use_of_resources/fair_use_and_prioritization_policies.md

@@ -13,6 +13,8 @@ One common misconception around fairshare is that a given user or lab should hav
 
 A fairshare score is computed for each fairshare group (SLURM account) based on the prior usage of the cluster by that group and their share of the cluster (the fraction of the cluster the account has been granted). Note that for some accounts, such as lab accounts, the share is determined by the lab. Individual users do not have their own subshares of the lab share. This means that if a lab has collectively been overusing the cluster, individual members of that lab may have a low fairshare score even if they were not running jobs. 
 
+Fairshare affects job priority, not hard limits. Separately, the number of GPUs a single user or account can hold at once is capped; see the GPU caps in [Cluster Usage Policies](../kempner_cluster/kempner_policies_for_responsible_use.md).
+
 
 ## Job Priority
 When you submit a job, a priority number will be calculated for that job, which determines the position of the job in the pending queue. This priority number is calculated based on a few factors: your fairshare score and the job age. This secondary factor ensures that jobs that have been waiting for a while to run are increasing in priority over time. This means that jobs from accounts with low fairshare will eventually run. You can see the Pending queue of jobs for a partition (such as `kempner`) by running `showq -o -p <partitionname>`. 

kempner_computing_handbook/s1_high_performance_computing/kempner_cluster/kempner_policies_for_responsible_use.md

@@ -17,20 +17,19 @@ The cluster should never be used for CPU-only jobs. There are four types of Kemp
 - 16 cores and 240 GB per GPU for `kempner` partition, and 
 - 24 cores and 360 GB per GPU for `kempner_h100` partition.  
 - 16 cores and 360 GB per GPU for `kempner_h200` partition, and 
-- 16 cores and 190 GB per GPU for `kempner_rtx` partition.  
+- 16 cores and 180 GB per GPU for `kempner_rtx` partition.  
 
 ::::{important}
-Users should not submit jobs that utilize more than the following cluster resources or that run for extended periods. Given the current resources, the limits in each partition are:
+Total GPU usage across the `kempner`, `kempner_h100`, `kempner_h200`, and `kempner_rtx` partitions is capped for each user and each Kempner account:
 
-- 8 A100 GPUs across up to 4 nodes in the `kempner` partition, and a 2-day runtime limit.
-- 8 H100 GPUs across up to 4 nodes in the `kempner_h100` partition, and a 2-day runtime limit.
-- 8 H200 GPUs across up to 4 nodes in the `kempner_h200` partition, and a 2-day runtime limit.
-- 8 RTX6000 GPUs across up to 4 nodes in the `kempner_rtx` partition, and a 2-day runtime limit.
+- Each **user** may hold at most **16 GPUs** at once, summed across these four partitions.
+- Each **account** (for example, `kempner_<lab>_lab`) may hold at most **96 GPUs** at once, summed across these four partitions.
+- The `kempner_undergrads` account has a lower cap of **4 GPUs** at once across these four partitions.
 
-See the following guidelines for jobs that exceed this limit.
+Each of these partitions also has a **2-day runtime limit** per job. The GPU caps apply to concurrent usage regardless of how it is split across jobs or partitions. On the [Kempner requeue](https://docs.rc.fas.harvard.edu/kb/kempner-partitions/) partition and on reservation (priority) partitions, the per-account cap can be exceeded by submitting under a non-Kempner account (for example, `sham_lab` instead of `kempner_sham_lab`), since usage on non-Kempner accounts does not count against the Kempner group cap.
 ::::
 
-If your job exceeds this predefined limit, please adhere to the following guidelines:
+If you need more capacity than these caps allow, please adhere to the following guidelines:
 
 - Use the [Kempner requeue](https://docs.rc.fas.harvard.edu/kb/kempner-partitions/) partition. This allows jobs to be preempted by other jobs and then restarted, thus allowing large jobs to run without disrupting access for the rest of the community.
 - If not using the requeue partition, only submit jobs when there is significant excess capacity and actively monitor the cluster (hourly) to ensure that there are still resources available for other users.