Merge pull request #307 from KempnerInstitute/iss_url

fix url issues

Author: Naeemkh

.github/workflows/linkcheck.yml

@@ -25,6 +25,10 @@ jobs:
           --no-progress
           --max-retries 2
           --accept 200,203,204,206,429
+          --scheme http
+          --scheme https
+          --exclude '^https?://vdi\.rc\.fas\.harvard\.edu/?$'
+          --exclude '^https?://(www\.)?cs\.toronto\.edu/~kriz/cifar\.html$'
           'kempner_computing_handbook/**/*.md'
           'README.md'
           'CONTRIBUTING.md'

kempner_computing_handbook/s1_high_performance_computing/development_and_runtime_envs/software_module_and_environment_management.md

@@ -44,4 +44,4 @@ Each module is defined by a script that runs when the module is loaded. This scr
 
 These changes to the terminal environment will persist until `module unload <module_name>` or `module purge` is called. They act independently of other modifications to environment variables, namely virtual environments.
 
-You can read more about modules at the [Lmod documentation](https://lmod.readthedocs.io/).
+You can read more about modules at the [Lmod documentation](https://lmod.readthedocs.io/en/latest/).

kempner_computing_handbook/s1_high_performance_computing/efficient_use_of_resources/fair_use_and_prioritization_policies.md

@@ -5,7 +5,7 @@
 
 The Kempner cluster uses a system called fairshare to determine prioritization and which jobs run when. The fairshare algorithm prioritizes a balanced allocation of resources, aimed at facilitating the timely completion of tasks from various user groups. This means that jobs, particularly those that are resource intensive or are being run in labs with high recent usage, may not run immediately or on demand. 
 
-As an approved user of the Kempner cluster, you will be part of one or more [fairshare groups (called slurm accounts)](resource_management:understanding_slurm:slurm_accounts), through which you receive a share of the cluster resources.  For more information, please see the [Cluster Governance guidelines](https://sites.harvard.edu/kempner/computing/).
+As an approved user of the Kempner cluster, you will be part of one or more [fairshare groups (called slurm accounts)](resource_management:understanding_slurm:slurm_accounts), through which you receive a share of the cluster resources.  For more information, please see the [Cluster Governance guidelines](https://kempnerinstitute.harvard.edu/kempner-community/).
 
 ```{tip}
 One common misconception around fairshare is that a given user or lab should have access at any given moment to exactly "their share" of the cluster. Instead, fairshare guarantees access to resources averaged over a period of several months. At any given moment, a user or lab may be using a greater or lesser amount of their available share.

kempner_computing_handbook/s1_high_performance_computing/kempner_cluster/kempner_policies_for_responsible_use.md

@@ -33,7 +33,7 @@ If your job exceeds this predefined limit, please adhere to the following guidel
 - Use a run-time limit to prevent large jobs from running for a long period of time unexpectedly.
 - Consider running the job with a smaller number of GPUs over a longer time-period.
 
-If none of the above strategies will work, users should submit a reservation request (see [Cluster Governance Guideline](https://sites.harvard.edu/kempner/computing/)) so that large projects can be planned and communicated to the community, limiting disruptions for other users.  
+If none of the above strategies will work, users should submit a reservation request (see [Cluster Governance Guideline](https://kempnerinstitute.harvard.edu/kempner-community/)) so that large projects can be planned and communicated to the community, limiting disruptions for other users.  
 
 
 ::::{warning}

kempner_computing_handbook/s1_high_performance_computing/kempner_cluster/overview_of_kempner_cluster.md

@@ -46,17 +46,17 @@ This is a diagram of the GPU racks and network core.
 
 ## Can I buy into the Kempner Institute AI Cluster?
 
-If you have a Kempner Institute affiliation, you may request to contribute funds towards a Kempner cluster purchase. If you contribute funds we will increase your priority on the cluster to reflect the proportion of the cluster you have contributed. Please see the Cluster Governance guidelines for more information: https://sites.harvard.edu/kempner/computing/
+If you have a Kempner Institute affiliation, you may request to contribute funds towards a Kempner cluster purchase. If you contribute funds we will increase your priority on the cluster to reflect the proportion of the cluster you have contributed. Please see the Cluster Governance guidelines for more information: https://kempnerinstitute.harvard.edu/kempner-community/
 
 ## How is priority set on the Kempner Institute AI Cluster?
 
-Priority is set through Fairshare scheduling.   For more information, please see the Cluster Governance guidelines: https://sites.harvard.edu/kempner/computing/
+Priority is set through Fairshare scheduling.   For more information, please see the Cluster Governance guidelines: https://kempnerinstitute.harvard.edu/kempner-community/
 
 
 ## Can I reserve GPUs on the Kempner Institute AI Cluster?
 
-Generally we hope to have enough GPUs available on the cluster for most use cases.  However, for situations in which you need a large number of GPUs that are dedicated for a period of time, or need a smaller subset for a longer period of time, you can make a reservation request. Please see the full governance guidelines for further details: https://sites.harvard.edu/kempner/computing/
+Generally we hope to have enough GPUs available on the cluster for most use cases.  However, for situations in which you need a large number of GPUs that are dedicated for a period of time, or need a smaller subset for a longer period of time, you can make a reservation request. Please see the full governance guidelines for further details: https://kempnerinstitute.harvard.edu/kempner-community/
 
 ## Where is the Kempner Institute AI Cluster located physically?
 
-The cluster is located at the [Massachusetts Green High-Performance Computing Center](https://www.mghpcc.org) (MGHPCC) in Holyoke, MA. This is a state-of-the-art datacenter that is shared among the Boston-area universities. It runs off of hydropower, making it one of the “greenest” computing centers in the world. FASRC also provides co-located [storage resources](https://docs.rc.fas.harvard.edu/kb/storage-service-center/) at MGHPCC.
+The cluster is located at the [Massachusetts Green High-Performance Computing Center](https://mghpcc.org) (MGHPCC) in Holyoke, MA. This is a state-of-the-art datacenter that is shared among the Boston-area universities. It runs off of hydropower, making it one of the “greenest” computing centers in the world. FASRC also provides co-located [storage resources](https://docs.rc.fas.harvard.edu/kb/data-storage-workflow-rdm/) at MGHPCC.

kempner_computing_handbook/s1_high_performance_computing/storage_and_data_transfer/shared_data_repository.md

@@ -30,13 +30,13 @@ We will develop a web interface later for data and model discovery.
 ---
 
 - **OpenAI**
-    - Path: `$MODEL_PATH/gpt2` (see on [HuggingFace](https://huggingface.co/gpt2))
+    - Path: `$MODEL_PATH/gpt2` (see on [HuggingFace](https://huggingface.co/openai-community/gpt2))
         - Size: 4.5 M
 
 --- 
 
 - **Google**
-    - Path: `$MODEL_PATH/t5-base` (see on [HuggingFace](https://huggingface.co/t5-base))
+    - Path: `$MODEL_PATH/t5-base` (see on [HuggingFace](https://huggingface.co/google-t5/t5-base))
         - Size: 3.4 M
 
 ## The current list of ML datasets

kempner_computing_handbook/s1_high_performance_computing/storage_and_data_transfer/understanding_storage_options.md

@@ -93,7 +93,7 @@ The following table summarizes the details of the default scratch storage (`nets
 | **Use Cases**       | Optimized for Varieties of Workflows including High I/O AI Workflows |
 
 
-The following table summarizes the storage options available on the cluster, visit [data storage](https://www.rc.fas.harvard.edu/services/data-storage/) for more information.
+The following table summarizes the storage options available on the cluster, visit [data storage](https://www.rc.fas.harvard.edu/data-storage/) for more information.
 
 ```{figure} figures/png/storage_table_20240324.png
 ---

kempner_computing_handbook/s3_ai_workflows/distributed_inference.md

@@ -17,10 +17,10 @@ The following models have been tested and configured for distributed inference u
 
 | Model            | Model Size | Hugging Face                                                     | Deployment Guide                                                                                                                  |
 |------------------|------------|------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
-| Llama 3.1        | 70B        | [HF Link](https://huggingface.co/meta-llama/Llama-3.1-70B)       | [Llama 3.1 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/README_Llama3.1.md)               |
-| Llama 3.1        | 405B       | [HF Link](https://huggingface.co/meta-llama/Llama-3.1-405B)      | [Llama 3.1 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/README_Llama3.1.md)               |
-| DeepSeek-R1      | 671B       | [HF Link](https://huggingface.co/deepseek-ai/DeepSeek-R1)        | [DeepSeek-R1 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/README_DeepSeekR1.md)           |
-| DeepSeek-R1-0528 | 671B       | [HF Link](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)   | [DeepSeek-R1-0528 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/README_DeepSeekR1-0528.md) |
+| Llama 3.1        | 70B        | [HF Link](https://huggingface.co/meta-llama/Llama-3.1-70B)       | [Llama 3.1 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/workflows/Llama-3.1-70B_multinode-server/README.md)               |
+| Llama 3.1        | 405B       | [HF Link](https://huggingface.co/meta-llama/Llama-3.1-405B)      | [Llama 3.1 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/workflows/Llama-3.1-405B_multinode-server/README.md)               |
+| DeepSeek-R1      | 671B       | [HF Link](https://huggingface.co/deepseek-ai/DeepSeek-R1)        | [DeepSeek-R1 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/workflows/DeepSeek-R1_multinode-server/README.md)           |
+| DeepSeek-R1-0528 | 671B       | [HF Link](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)   | [DeepSeek-R1-0528 Deployment](https://github.com/KempnerInstitute/distributed-inference-vllm/blob/main/workflows/DeepSeek-R1-0528_multinode-server/README.md) |
 
 Each model's deployment page includes:
 

kempner_computing_handbook/s3_ai_workflows/nemo_workflow.md

@@ -4,7 +4,7 @@ This section documents NVIDIA NeMo Workflows, designed for training and finetuni
 
 ## Overview
 
-This section provides a modular, cluster-ready workflow built on top of the [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) framework. It supports:
+This section provides a modular, cluster-ready workflow built on top of the [NVIDIA NeMo](https://github.com/NVIDIA-NeMo/NeMo) framework. It supports:
 
 - Pretraining and finetuning of LLMs such as GPT2 and Llama 3
 - Scalable distributed training utilizing SLURM

kempner_computing_handbook/s5_ai_scaling_and_engineering/experiment_management/logging_and_monitoring.md

@@ -40,4 +40,4 @@ for epoch in range(epochs):
     wandb.log({"accuracy": acc, "loss": loss})
 ```
 
-Access your project and view your runs on [wandb.ai](https://wandb.ai/home) website. Additionally, invite any collaborators you have to your project at this page. Refer to the [W&B Quickstart](https://docs.wandb.ai/quickstart/) for more details.
+Access your project and view your runs on [wandb.ai](https://wandb.ai/home) website. Additionally, invite any collaborators you have to your project at this page. Refer to the [W&B Quickstart](https://docs.wandb.ai/models/quickstart) for more details.