Merge pull request #560 from saiyam1814/add-hami-isolation-lab

Add Lab 7: GPU isolation on k3s without the GPU Operator

Author: hami-robot[bot]

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-isolation-k3s.md

@@ -19,4 +19,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto';
 
 <LabCardGridAuto />
 
-每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。
+每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md

@@ -41,6 +41,11 @@ module.exports = {
           id: "labs/hami-vllm",
           customProps: { level: "Intermediate", duration: "about 45 minutes" },
         },
+        {
+          type: "doc",
+          id: "labs/hami-isolation-k3s",
+          customProps: { level: "Intermediate", duration: "about 45 minutes" },
+        },
       ],
     },
   ],

sidebars-tutorials.js

@@ -0,0 +1,28 @@
+# A third Pod whose memory slice fits the card when it is empty, but NOT alongside
+# the two 8000 MiB slices that hami-share-a / hami-share-b already hold. It stays
+# Pending with CardInsufficientMemory - proving the device plugin and scheduler
+# account the card's memory as one shared, finite budget.
+#
+# Apply this ONLY after hami-share-a and hami-share-b are Running.
+#
+# Sizing (IMPORTANT - adjust for your card): set gpumem ABOVE (card_total - 2*slice)
+# so it cannot fit beside the two slices, and BELOW card_total so it WOULD fit on an
+# empty card. 90000 MiB works for a 96 GB RTX PRO 6000 (97887 MiB) with two 8000 MiB
+# slices held (~82000 free). For a 48 GB card use ~45000; for a 24 GB card with two
+# 4000 MiB slices, use ~20000.
+# Read the card size first:
+#   kubectl exec hami-share-a -- nvidia-smi --query-gpu=memory.total --format=csv
+apiVersion: v1
+kind: Pod
+metadata:
+  name: hami-oversubscribe
+spec:
+  restartPolicy: Never
+  containers:
+    - name: cuda
+      image: nvidia/cuda:12.4.1-devel-ubuntu22.04
+      command: ["bash", "-c", "nvidia-smi; sleep infinity"]
+      resources:
+        limits:
+          nvidia.com/gpu: 1
+          nvidia.com/gpumem: 90000

tutorials/labs/examples/07-hami-isolation-k3s/oversubscribe-pending.yaml

@@ -0,0 +1,35 @@
+# Two Pods, each asking for one physical GPU and an 8000 MiB memory slice. On a
+# 96 GB RTX PRO 6000 (or a 48 GB A6000) both fit with plenty of headroom, so both
+# land on the one GPU. On a 24 GB card use ~4000 MiB instead.
+#
+# The image is the CUDA devel image because the memory probe compiles a tiny CUDA
+# allocator with nvcc inside the Pod. It is large to pull; that is expected.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: hami-share-a
+spec:
+  restartPolicy: Never
+  containers:
+    - name: cuda
+      image: nvidia/cuda:12.4.1-devel-ubuntu22.04
+      command: ["bash", "-c", "nvidia-smi; sleep infinity"]
+      resources:
+        limits:
+          nvidia.com/gpu: 1
+          nvidia.com/gpumem: 8000
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: hami-share-b
+spec:
+  restartPolicy: Never
+  containers:
+    - name: cuda
+      image: nvidia/cuda:12.4.1-devel-ubuntu22.04
+      command: ["bash", "-c", "nvidia-smi; sleep infinity"]
+      resources:
+        limits:
+          nvidia.com/gpu: 1
+          nvidia.com/gpumem: 8000