Skip to content

Commit 4f6ffb7

Browse files
committed
docs: scope chart-e2e vs operator-e2e; note soft-slicing in architecture
- chart-e2e: clarify it validates the chart contract (install/startup/image/ version), not feature/behavioral e2e - architecture.md: note the device-plugin allocator soft-slicing runtime isolation in Stage 2 - spec: align task descriptions with the inline-clone packaging (no submodule) and record the GPU-less injection-e2e blocker as a next-spec follow-up The sliced soft-slicing injection e2e (CASE 6) is intentionally NOT added here: GPU-less simulation can't drive the device-plugin Allocate (detector is hardware-gated), so it's deferred to a spec that adds a detector simulation mode. Signed-off-by: thxCode <thxcode0824@gmail.com>
1 parent fc33a06 commit 4f6ffb7

3 files changed

Lines changed: 117 additions & 71 deletions

File tree

.claude/skills/gpustack-operator-chart-e2e/SKILL.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,21 @@
11
---
22
name: gpustack-operator-chart-e2e
3-
description: "Verify the GPUStack Operator **Helm chart** end to end on a reachable local cluster (k3s / docker-desktop): build & load the dev image, `helm install` the chart, assert the worker/device-manager roll out, the versions are consistent (image tag ↔ bundled chart tgz ↔ `gpustack-operator --version`), then `helm uninstall` and assert zero leftovers. Proactively offer this when a branch ahead of main changes the chart, the in-cluster app-installation code, or the image build. Examples: \"verify the chart installs and uninstalls cleanly\", \"does helm install of the operator work\", \"check the chart version is right\", \"test my chart change end to end\", \"did my kuberess change break the runtime install\"."
3+
description: "Verify the GPUStack Operator **Helm chart** end to end on a reachable local cluster (k3s / docker-desktop): build & load the dev image, `helm install` the chart, assert the worker/device-manager roll out, the versions are consistent (image tag ↔ bundled chart tgz ↔ `gpustack-operator --version`), then `helm uninstall` and assert zero leftovers. SCOPE: this validates the **chart contract** — chart changes, install/startup, image build, and version consistency — and is **NOT a feature/behavioral e2e**; for deep scheduling-chain behavior use `gpustack-operator-e2e` instead. Proactively offer this when a branch ahead of main changes the chart, the in-cluster app-installation code, or the image build. Examples: \"verify the chart installs and uninstalls cleanly\", \"does helm install of the operator work\", \"check the chart version is right\", \"test my chart change end to end\", \"did my kuberess change break the runtime install\"."
44
allowed-tools: "Read, Bash(bash .claude/skills/_e2e-lib/scripts/preflight.sh*), Bash(bash .claude/skills/_e2e-lib/scripts/assert-core.sh*), Bash(bash .claude/skills/gpustack-operator-chart-e2e/cases/case-1.sh*), Bash(kubectl get*), Bash(kubectl cluster-info*), Bash(kubectl version*), Bash(kubectl config current-context), Bash(helm status*), Bash(helm list*), Bash(helm template*), Bash(helm lint*), Bash(git diff*), Bash(git rev-parse*), Bash(command -v*)"
55
model: sonnet
66
---
77

88
# GPUStack Operator — Helm chart E2E verification
99

1010
Install the operator **from its Helm chart** onto a **local** cluster and verify it installs, runs, and
11-
uninstalls cleanly — with the **right version** baked in. This is the chart/version counterpart to the
12-
`gpustack-operator-e2e` skill (which exercises the deep NFD → Worker → Kueue scheduling-chain behavior);
13-
keep the deep behavioral assertions there and the install/uninstall + version contract here.
11+
uninstalls cleanly — with the **right version** baked in.
12+
13+
**Scope — chart contract only, not feature e2e.** This skill is for validating **chart changes, install,
14+
startup/rollout, the image build, and the version contract** (image tag ↔ bundled chart tgz ↔
15+
`gpustack-operator --version`). It is the chart/version counterpart to the `gpustack-operator-e2e` skill,
16+
which exercises the deep NFD → Worker → Kueue scheduling-chain behavior. **Do not** add feature/behavioral
17+
assertions here — keep the deep behavioral e2e (scheduling chain) in `gpustack-operator-e2e`, and keep only
18+
install/uninstall + version-contract assertions here.
1419

1520
This skill **mutates a cluster**. Hard rules:
1621

docs/architecture.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,10 @@ where `prefix` is `feature.gpustack.ai/` — so the device labels live under the
148148

149149
The NodeFeature object is owned by the Node; the DM also reports a `Devices` custom resource named after the node (owned by that NodeFeature) and keeps monitoring devices, re-detecting whenever the device set or health changes.
150150

151+
Alongside detection, the DM runs the **device-plugin allocator** (`pkg/devicemanager/allocator/<mfr>`, on `pkg/deviceplugin`): it registers per-mode resources (exclusive / shared / sliced) with the kubelet and, on `Allocate`, returns the container injection. For **exclusive/shared** this is just the device-visibility env (`NVIDIA_VISIBLE_DEVICES` / `ASCEND_VISIBLE_DEVICES` / …). For **sliced (soft slicing)** it additionally applies **runtime isolation**: the container is started with a vendor preload library — NVIDIA HAMi-core `libvgpu.so`, Ascend vcann-rt `libvruntime.so` — activated via `/etc/ld.so.preload`, plus per-container quota derived from the slice request (NVIDIA env `CUDA_DEVICE_SM_LIMIT` / `CUDA_DEVICE_MEMORY_LIMIT_*`; Ascend an `npu_info.config` carrying `aicore-quota` / `memory-quota`). The preload libraries are compiled into the operator image per runtime version (cloned inline at pinned commits — no git submodule — and built in the `xbuild-nvidia-cuda-*` / `xbuild-ascend-cann-*` Dockerfile stages, with the build scripts under `pack/gpustack-operator/external/{nvidia,ascend}`) and staged onto the host (`/var/lib/gpustack/operator/lib`) by a device-manager **init container**; the allocator mounts the matching library + a per-pod working directory into each sliced container and reclaims those directories once their pods are gone.
152+
153+
A node hosts a single driver/runtime stack per manufacturer, so every `DevicesGroup` of a given manufacturer on that node shares the same driver and runtime version (same root/source). The per-runtime-version library subdir (`cuda-<major>` / `cann-<major>-<family>`) the allocator picks from the first allocated card is therefore correct for every card in a sliced allocation — a single sliced container can never span mismatched CUDA/CANN majors in practice. The allocator nonetheless guards against it **defensively** (NVIDIA rejects a sliced allocation spanning different CUDA majors; Ascend rejects a multi-card sliced allocation, since vcann-rt's `npu_info.config` models a single physical NPU), so any future regression fails the `Allocate` loudly instead of silently mounting an incompatible library.
154+
151155
### Stage 3: GPUStack Operator Worker (WK)
152156

153157
The WK's `NodeFeatureReconciler` (`pkg/worker/controllers/worker/nodefeature.go`) watches Nodes and reports a NodeFeature object named `${NODE_NAME}-gpustack-worker`, whose labels are built by `nodefeature.ConstructNodeCapacityLabels`.

0 commit comments

Comments
 (0)