Skip to content

Commit 08d5373

Browse files
flomedjaflomedja
committed
docs(observability): document GPU monitoring templates and NVSentinel
 Add two pages to the fleet observability section for ENGPLAT-834: - GPU monitoring templates: summary of the bundled Argo CD templates (cert-manager, GPU Operator, Fleet Intelligence Agent, NVSentinel), deployment order, the two DCGM metric families with the double-count warning, per-pod GPU attribution, and stack teardown - NVSentinel GPU monitoring: deploy guide with UI and YAML flows, what dry-run does and does not do, full parameter reference including Janitor providers, and cleanup of the retained MongoDB PVCs Link both pages from the section overview and the Argo CD templates page, and shift later pages down in the sidebar.
1 parent c2dc818 commit 08d5373

7 files changed

Lines changed: 506 additions & 4 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,4 @@ hack/cli-drift/cli-drift
2828
hack/cli-drift/bin
2929
hack/config-drift/config-drift
3030
hack/config-drift/bin
31+
.serena/

‎platform/maintenance/observability/deploy-with-argocd.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,8 @@ their resource name through `spec.templateRef.name`.
4141
| Fleet Observability: Grafana | `fleet-observability-grafana` | Grafana with the fleet, cluster, and GPU dashboards pre-provisioned | `vcluster-platform` |
4242
| Fleet Observability: Cluster Collector | `cluster-collector` | OpenTelemetry Collectors | `observability` |
4343

44-
The integration also ships templates for cert-manager and GPU monitoring, which are
45-
outside the scope of this guide.
44+
The integration also ships templates for cert-manager and GPU monitoring; see
45+
[GPU monitoring templates](./gpu-monitoring-templates.mdx).
4646

4747
Because the templates are cluster-scoped, you don't copy them into each project. You
4848
reference them by name and pass per-deployment values through `spec.parameters`. For the

‎platform/maintenance/observability/gpu-monitoring-templates.mdx‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
---
2+
title: Monitor GPUs with the bundled Argo CD templates
3+
sidebar_label: GPU monitoring templates
4+
sidebar_position: 7
5+
description: Overview of the GPU monitoring Argo CD templates that vCluster Platform ships, the order to deploy them in, which DCGM metric family each produces, and how to tear the stack down.
6+
---
7+
8+
When the [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled,
9+
vCluster Platform ships a set of `ArgoCDApplicationTemplate` resources for GPU
10+
monitoring. Together they deploy the NVIDIA GPU stack into a cluster that runs GPU
11+
nodes, typically a tenant cluster with private nodes: drivers and telemetry through the
12+
GPU Operator, node-level GPU health reporting through the Fleet Intelligence Agent, and
13+
fault detection and remediation through NVSentinel.
14+
15+
This page summarizes the templates, the order to deploy them in, and the metric
16+
behaviors to know before you build queries on top of them. For the NVSentinel deploy
17+
guide and parameter reference, see
18+
[Monitor GPU nodes with NVSentinel](./nvsentinel-gpu-monitoring.mdx).
19+
20+
## Bundled GPU templates
21+
22+
In an `ArgoCDApplication`, reference each template by its resource name through
23+
`spec.templateRef.name`. The Platform UI lists them by display name. For the general
24+
template and application model, see
25+
[Deploy applications](../../integrations/argocd/deploy-applications.mdx).
26+
27+
| Template (Platform UI) | `templateRef.name` | Deploys | Default namespace |
28+
|---|---|---|---|
29+
| cert-manager | `cert-manager` | cert-manager, for the internal TLS certificates NVSentinel components need | `cert-manager` |
30+
| NVIDIA GPU Operator | `nvidia-gpu-operator` | NVIDIA drivers, device plugin, DCGM, and dcgm-exporter | `gpu-operator` |
31+
| Fleet Intelligence Agent | `fleet-intelligence-agent` | NVIDIA's gpud-based node agent that reports GPU health and DCGM metrics | `fleet-intelligence` |
32+
| NVSentinel | `nvsentinel` | GPU fault detection, quarantine, and remediation | `nvsentinel` |
33+
34+
## Deployment order
35+
36+
The applications depend on each other at runtime, so deploy them in this order:
37+
38+
1. **cert-manager**, if you plan to enable NVSentinel components that need internal TLS,
39+
such as the MongoDB store or the Janitor. See
40+
[the NVSentinel prerequisites](./nvsentinel-gpu-monitoring.mdx) for exactly which
41+
components require it.
42+
2. **NVIDIA GPU Operator.** Everything else builds on it: it installs the drivers and
43+
device plugin, and runs the DCGM service that NVSentinel reads GPU health from, plus
44+
dcgm-exporter for GPU metrics.
45+
3. **Fleet Intelligence Agent**, optionally, for node-level GPU health reporting. The
46+
bundled GPU dashboard reads some panels from its metrics.
47+
4. **NVSentinel**, to detect and remediate GPU faults. See
48+
[Monitor GPU nodes with NVSentinel](./nvsentinel-gpu-monitoring.mdx).
49+
50+
## GPU metrics and attribution
51+
52+
When the GPU templates are deployed, two separate exporters report metrics for the same
53+
GPUs, and by default those metrics do not show which pod used a GPU. Both behaviors are
54+
intentional. Keep them in mind when you write your own queries or bill tenants for
55+
their GPU usage based on the bundled dashboards.
56+
57+
### Two DCGM metric families
58+
59+
The same physical GPU is reported by two exporters when the full GPU stack is deployed:
60+
61+
| Source | Deployed by | Metric family | Example |
62+
|---|---|---|---|
63+
| dcgm-exporter | NVIDIA GPU Operator template | UPPERCASE `DCGM_FI_DEV_*` | `DCGM_FI_DEV_GPU_UTIL` |
64+
| gpud | Fleet Intelligence Agent template | lowercase `dcgm_fi_dev_*` | `dcgm_fi_dev_gpu_util` |
65+
66+
The bundled GPU dashboard mixes both families: some panels read the dcgm-exporter series
67+
and others read the Fleet Intelligence Agent series.
68+
69+
:::warning Pick one metric family per query
70+
Aggregating across both families counts every GPU twice, for example with a regular
71+
expression matcher that is case-insensitive or that spans both spellings. When you write
72+
your own PromQL, pick one family per query.
73+
:::
74+
75+
### Per-pod GPU attribution is off by default
76+
77+
The NVIDIA GPU Operator template ships dcgm-exporter with
78+
`dcgmExporterEnablePodLabels: "false"` and `dcgmExporterEnablePodUID: "false"`. GPU
79+
metrics then carry node, GPU, UUID, and `modelName` labels but **no pod or namespace
80+
label**, so GPU usage cannot be attributed to a workload:
81+
82+
```text
83+
DCGM_FI_DEV_GPU_UTIL{gpu="0", UUID="GPU-9d9d49d7-...", modelName="Tesla T4", Hostname="ip-192-168-75-24"} 0
84+
```
85+
86+
To attribute GPU usage to workloads and bill tenants for it, set both parameters to
87+
`"true"` on the NVIDIA GPU Operator application and let it sync.
88+
89+
:::note Benign crashloop on startup
90+
With pod labels enabled, dcgm-exporter can restart a few times right after rollout while
91+
its ServiceAccount token is projected, logging
92+
`Failed to get in-cluster config, pod labels will not be available`. It heals on its own
93+
within a few restarts; no action is needed.
94+
:::
95+
96+
## Tear down the GPU stack
97+
98+
Deleting the `ArgoCDApplication` resources prunes the workloads, but some artifacts are
99+
retained by design and stay in the cluster until you remove them: NVSentinel's MongoDB
100+
PVCs, the cert-manager CRDs (`keepCRDs: true`), and the destination namespaces, which
101+
stay `Active` with zero pods.
102+
103+
First remove NVSentinel's retained data and namespace, described in
104+
[Uninstall and clean up NVSentinel](./nvsentinel-gpu-monitoring.mdx#uninstall-and-cleanup).
105+
Then remove the remaining GPU namespaces:
106+
107+
```bash title="Clean up GPU stack namespaces"
108+
kubectl delete namespace gpu-operator fleet-intelligence
109+
```
110+
111+
### Remove cert-manager entirely (optional)
112+
113+
CRDs are cluster-wide type definitions, so they cannot be scoped to the GPU templates:
114+
the only choice is to keep them or remove them for the whole cluster. Run this step only
115+
when cert-manager was installed solely for the GPU stack and nothing else in the cluster
116+
uses it. Deleting a CRD cascade-deletes every resource of that type in every namespace.
117+
118+
```bash title="Remove cert-manager CRDs and namespace"
119+
# Review what exists first
120+
kubectl get crd -o name | grep -E '\.cert-manager\.io$'
121+
122+
kubectl delete crd \
123+
certificates.cert-manager.io \
124+
certificaterequests.cert-manager.io \
125+
challenges.acme.cert-manager.io \
126+
clusterissuers.cert-manager.io \
127+
issuers.cert-manager.io \
128+
orders.acme.cert-manager.io
129+
130+
kubectl delete namespace cert-manager
131+
```
132+
133+
None of these carry blocking finalizers, so deletion completes cleanly.
134+
135+
## Related pages
136+
137+
- [Monitor GPU nodes with NVSentinel](./nvsentinel-gpu-monitoring.mdx) for the NVSentinel
138+
deploy guide, dry-run semantics, and parameter reference.
139+
- [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx) for the metrics
140+
backend, Grafana, and edge collector templates.
141+
- [Query fleet metrics](./query-fleet-metrics.mdx) for scoped PromQL access to GPU
142+
metrics.

‎platform/maintenance/observability/labels-reference.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: Fleet Observability labels reference
33
sidebar_label: Labels reference
4-
sidebar_position: 7
4+
sidebar_position: 9
55
description: Reference for vcluster_platform labels stamped by the Fleet Observability Write Gateway.
66
---
77

0 commit comments

Comments
 (0)