Merge pull request #282 from NYU-RTS/Amanda-dong

Amanda-dong · web-flow · commit 3fd4f3d4d093 · 2026-02-15T14:27:05.000-05:00
Update LLM inference docs
diff --git a/docs/hpc/08_ml_ai_hpc/llm_inference/05_run_hf_model.md b/docs/hpc/08_ml_ai_hpc/llm_inference/05_run_hf_model.md
@@ -22,13 +22,13 @@ Some of the following steps can require significant resources, so we'll move to
 
 ### Copy appropriate overlay file to the project directory
 ```bash
-[NetID@cm001 llm_example]$ cp -rp /scratch/work/public/overlay-fs-ext3/overlay-50G-10M.ext3.gz .
+[NetID@cm001 llm_example]$ cp -rp /share/apps/overlay-fs-ext3/overlay-50G-10M.ext3.gz .
 [NetID@cm001 llm_example]$ gunzip overlay-50G-10M.ext3.gz
 ```
 
 ### Launch Singularity container in read/write mode
 ```bash
-[NetID@cm001 llm_example]$ singularity exec --overlay overlay-50G-10M.ext3:rw /scratch/work/public/singularity/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash
+[NetID@cm001 llm_example]$ apptainer exec --fakeroot --overlay overlay-50G-10M.ext3:rw /share/apps/images/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash
 ```
 
 ### Install miniconda in the container
diff --git a/docs/hpc/08_ml_ai_hpc/llm_inference/07_vLLM.md b/docs/hpc/08_ml_ai_hpc/llm_inference/07_vLLM.md
diff --git a/docs/hpc/08_ml_ai_hpc/llm_inference/overview.md b/docs/hpc/08_ml_ai_hpc/llm_inference/overview.md
@@ -0,0 +1,23 @@
+# Overview
+This directory provides two primary pathways for deploying and running Large Language Models (LLMs) on the NYU Torch/Greene cluster: Hugging Face Transformers (for research/experimentation) and vLLM (for high-performance serving).
+
+## 1. Basic Inference (Hugging Face)
+This method is ideal for feature extraction, embeddings, or small-scale batch processing. It relies on a persistent Singularity environment using an .ext3 overlay.
+
+**Workflow:**
+Environment: Launch an Apptainer container with a read/write overlay.
+Persistence: Install Conda/Miniforge and libraries directly into /ext3.
+Execution: Use AutoModel to load weights and perform a forward pass.
+Key File: huggingface.py
+Ideal for: Extracting last_hidden_state (embeddings) or sentiment classification.
+
+## 2. High-Performance Serving (vLLM)
+vLLM is the recommended tool for production-level throughput and low-latency inference. It utilizes PagedAttention to manage memory efficiently.
+
+**Why vLLM?**
+Speed: higher throughput than standard backends on Torch.
+Compatibility: Drop-in replacement for OpenAI API.
+
+**Deployment Options:**
+Online: Use vllm serve to start an HTTP server accessible via curl or OpenAI clients.
+Offline: Use the LLM class within Python for processing large datasets without a server.
