diff --git a/.github/styles/Google/Headings.yml b/.github/styles/Google/Headings.yml index e46b69e695..69c16e2a2c 100644 --- a/.github/styles/Google/Headings.yml +++ b/.github/styles/Google/Headings.yml @@ -66,4 +66,7 @@ exceptions: - Helm - FIPS - SSO - - App Project \ No newline at end of file + - App Project + - GPU + - DCGM + - NVSentinel \ No newline at end of file diff --git a/.gitignore b/.gitignore index c5d6237aad..872da1cd10 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,4 @@ hack/cli-drift/cli-drift hack/cli-drift/bin hack/config-drift/config-drift hack/config-drift/bin +.serena/ diff --git a/.vale.ini b/.vale.ini index a4a8fa3ddf..d8056f0c8e 100644 --- a/.vale.ini +++ b/.vale.ini @@ -87,6 +87,18 @@ BasedOnStyles = Loft, Google, Vale Vale.Terms = NO Google.Headings = NO +# "Terminate" here always means the cloud/Kubernetes action of destroying a node or +# instance (matches the actual API names, e.g. ec2:TerminateInstances), not ending a +# session, so Google.WordList's stop/exit/cancel/end suggestion would be inaccurate. +# "GCP service account name" is a real GCP resource-name field, not a login username, +# so the account-name-to-username substitution is also a false positive. Scope the +# whole rule off for this page rather than editing the shared word list, since inline +# per-instance vale toggles proved unreliable across this page's wrapped list items +# and table cells. +[**/observability/nvsentinel-gpu-monitoring.mdx] +BasedOnStyles = Loft, Google, Vale +Google.WordList = NO + # Auto-generated by scripts/generate-platform-cli-index.js from # vcluster/cli/vcluster_platform*.md. The regenerate-instructions comment # references that file path in plain text, and command descriptions are diff --git a/platform/maintenance/observability/configure-edge-collectors.mdx b/platform/maintenance/observability/configure-edge-collectors.mdx index 51fbc17381..f71ba1cdbc 100644 --- a/platform/maintenance/observability/configure-edge-collectors.mdx +++ b/platform/maintenance/observability/configure-edge-collectors.mdx @@ -3,22 +3,148 @@ title: Configure edge collectors sidebar_label: Configure collectors sidebar_position: 5 description: Configure OpenTelemetry Collectors to send fleet metrics to the vCluster observability Write Gateway. -sidebar_class_name: pro --- +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import Flow, { Step } from "@site/src/components/Flow"; +import Button from "@site/src/components/Button"; +import Label from "@site/src/components/Label"; + Edge collectors are OpenTelemetry Collectors that scrape or receive metrics near the workloads, then send OTLP to the Write Gateway. Each collector request must include a bearer token and scope headers that match a `metrics-writer` access key. -:::tip Deploy collectors with Argo CD -If you use the Argo CD integration, you can deploy the collectors from a bundled template -instead of configuring them by hand. See -[Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). This manual path is the -only option for vCluster Standalone tenant clusters, which can't use the Argo CD -integration. +Deploy collectors once the gateway is running and a `metrics-writer` key is in place, as +described in [Install the observability gateway](./install-observability-gateway.mdx) and +[Configure metrics access](./configure-metrics-access.mdx). + +## Deploy collectors with Argo CD + +Collectors run on every tenant cluster and control plane cluster you want metrics from. +If you use the [Argo CD integration](../../integrations/argocd/overview.mdx), deploy one +`cluster-collector` `ArgoCDApplication` to each cluster instead of configuring the +OpenTelemetry Collector by hand. + +:::note vCluster Standalone tenant clusters +The out-of-the-box collector path relies on the Argo CD integration, which vCluster +Standalone tenant clusters can't use. There is no bundled collector for them. Configure a +collector manually instead, as described below. ::: -## Gateway endpoints +Each collector authenticates to the gateway with a `metrics-writer` bearer token. Create a +Secret named `otel-otlp-auth` in the collector namespace before you deploy, or pass the +token through the `bearerToken` parameter. + +```yaml title="otel-otlp-auth.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: otel-otlp-auth + namespace: observability +type: Opaque +stringData: + authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY" +``` + +Create the writer key first, as described in +[Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster) +or [Create a writer key for a control plane cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-control-plane-cluster), +then deploy the collectors. The walkthrough below deploys to a tenant cluster. For a +control plane cluster, swap `destination.cluster` in for `destination.virtualCluster`, as +described at the end of this section. + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="fleet-edge-collectors.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-edge-collectors + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet edge collectors" + destination: + virtualCluster: + name: my-vcluster # replace with your tenant cluster name + target: vCluster # deploy into the tenant cluster + templateRef: + name: cluster-collector + parameters: + destinationNamespace: observability + otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 + project: default +``` + + + + + + + Select your project from the projects dropdown at the top of the left navigation bar. + + + Navigate to and open the tenant cluster you want to collect from. + + + Select the tab and click . + + + In the section, enter a such as `Fleet edge collectors`. + + + Select to deploy into the tenant cluster itself. + + + From the dropdown, select the template. + + + Fill the . Set to `observability` and to the Write Gateway endpoint. Use the in-cluster Service endpoint for collectors in the Platform cluster, or the gateway LoadBalancer address for collectors in a tenant cluster elsewhere. + + + Click to add the Argo CD Application to the tenant cluster. + + + + + + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `observability` | Namespace for the collectors. | +| `project` | `default` | Argo CD project that owns the application. | +| `otlpEndpoint` | Fleet Write Gateway Service | OTLP HTTP endpoint the collectors push to. Use the in-cluster Service endpoint (`:4318`) for collectors in the Platform cluster, or the gateway LoadBalancer address (`https://:4318`) for collectors elsewhere. See [Expose the gateway with a LoadBalancer](./install-observability-gateway.mdx#expose-the-gateway-with-a-loadbalancer). | +| `bearerToken` | _(empty)_ | `metrics-writer` bearer token. When empty, the collectors read the `otel-otlp-auth` Secret. | +| `otlpCaData` | _(empty)_ | PEM CA data used to verify the gateway serving certificate. | +| `otlpInsecureSkipVerify` | `false` | Skip gateway TLS verification. Use only for test gateways. | + +The collectors stamp the request scope from the destination, so target a tenant cluster +for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set +`destination.cluster` instead of `destination.virtualCluster` to target a control plane +cluster. Don't set `vcluster_platform_*` labels in collector configuration; the Write +Gateway owns that namespace. See [Protect platform labels](#protect-platform-labels) for +the label rules, and [Gateway endpoints](#gateway-endpoints) below for the scope headers +the template sets on your behalf. + +On a cluster with the GPU stack deployed, this template also scrapes GPU metrics without +extra configuration. See +[The cluster-collector template scrapes GPU metrics automatically](./gpu-observability-templates.mdx#the-cluster-collector-template-scrapes-gpu-metrics-automatically). + +## Configure collectors manually + +To configure a collector yourself instead of using the bundled template, or to customize +behavior the template doesn't expose as a parameter, wire it up directly. + +### Gateway endpoints Inside the Platform cluster, the fleet gateway Service is: @@ -52,7 +178,7 @@ endpoint instead of the Service name, for example to enable it and retrieve the address. The exporter `endpoint` and TLS settings in the examples below are otherwise unchanged; only the host differs. -## Store the writer key +### Store the writer key Create a Secret for the collector, using a `metrics-writer` access key from [Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster) @@ -72,7 +198,7 @@ stringData: Mount or project this Secret into your collector pod and expose it as an environment variable or file. -## Tenant cluster collector +### Tenant cluster collector Use tenant scope headers when a collector writes metrics for a specific tenant cluster. @@ -97,7 +223,7 @@ service: The `metrics-writer` key must include a matching `spec.scope.virtualClusters` entry. -## Control plane cluster collector +### Control plane cluster collector Use cluster scope headers when a collector writes metrics for a control plane cluster. @@ -121,7 +247,7 @@ service: The `metrics-writer` key must include a matching `spec.scope.clusters` entry. -## Scrape custom targets +### Scrape custom targets Use the collector's `prometheus` receiver to scrape exporters such as kubelet, cAdvisor, DCGM exporter, or application endpoints. The gateway doesn't require a diff --git a/platform/maintenance/observability/configure-metrics-access.mdx b/platform/maintenance/observability/configure-metrics-access.mdx index 07bb7a9a77..28d35d7963 100644 --- a/platform/maintenance/observability/configure-metrics-access.mdx +++ b/platform/maintenance/observability/configure-metrics-access.mdx @@ -3,7 +3,6 @@ title: Configure metrics access sidebar_label: Configure access sidebar_position: 4 description: Create metrics-writer and metrics-reader access keys for the Fleet Observability gateway. -sidebar_class_name: pro --- import InterpolatedCodeBlock from "@site/src/components/InterpolatedCodeBlock"; @@ -178,7 +177,7 @@ recommended for collector configurations. :::note Argo CD collectors set these headers for you When you deploy collectors with the `cluster-collector` template as described in -[Deploy collectors to each cluster](./deploy-with-argocd.mdx#deploy-collectors-to-each-cluster), the template sets +[Deploy collectors with Argo CD](./configure-edge-collectors.mdx#deploy-collectors-with-argo-cd), the template sets these scope headers automatically from the deployment destination. You only set them by hand when you configure a collector manually. ::: diff --git a/platform/maintenance/observability/deploy-with-argocd.mdx b/platform/maintenance/observability/deploy-with-argocd.mdx deleted file mode 100644 index 823184b300..0000000000 --- a/platform/maintenance/observability/deploy-with-argocd.mdx +++ /dev/null @@ -1,359 +0,0 @@ ---- -title: Deploy Fleet Observability with Argo CD -sidebar_label: Argo CD templates -sidebar_position: 3 -description: Use the default Argo CD Application Templates that vCluster Platform ships to deploy the fleet metrics backend, Grafana, and edge collectors through GitOps. -sidebar_class_name: pro ---- - -import Tabs from "@theme/Tabs"; -import TabItem from "@theme/TabItem"; -import Flow, { Step } from "@site/src/components/Flow"; -import Button from "@site/src/components/Button"; -import Label from "@site/src/components/Label"; - -When the [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled, -vCluster Platform ships a set of default `ArgoCDApplicationTemplate` resources for Fleet -Observability. You reference these templates from `ArgoCDApplication` resources to deploy the -metrics backend, Grafana, and edge collectors through GitOps, instead of installing and -wiring them up by hand. - -The templates are an alternative to the manual setup in -[Install the observability gateway](./install-observability-gateway.mdx) and -[Configure edge collectors](./configure-edge-collectors.mdx). The observability gateway -itself is still deployed from an observability connector Secret, as described on the -install page. - -:::info Prerequisites -The bundled templates are installed only when the Argo CD integration is enabled and your -license includes Fleet Observability. You also need an Argo CD connector registered on the -target cluster. See [Connect to Argo CD](../../integrations/argocd/connect-argocd.mdx). -::: - -## Bundled templates - -Platform installs the following cluster-scoped `ArgoCDApplicationTemplate` resources. The -Platform UI lists them by display name. In an `ArgoCDApplication`, you reference them by -their resource name through `spec.templateRef.name`. - -| Template (Platform UI) | `templateRef.name` | Deploys | Default namespace | -|---|---|---|---| -| Fleet Observability: Prometheus | `fleet-observability-prometheus` | Prometheus configured as an OTLP metrics backend | `vcluster-platform` | -| Fleet Observability: Grafana | `fleet-observability-grafana` | Grafana with the fleet, cluster, and GPU dashboards pre-provisioned | `vcluster-platform` | -| Fleet Observability: Cluster Collector | `cluster-collector` | OpenTelemetry Collectors | `observability` | - -The integration also ships templates for cert-manager and GPU monitoring, which are -outside the scope of this guide. - -Because the templates are cluster-scoped, you don't copy them into each project. You -reference them by name and pass per-deployment values through `spec.parameters`. For the -general template and application model, see -[Deploy applications](../../integrations/argocd/deploy-applications.mdx). - -## How the pieces fit together - -The full setup assembles four pieces in order: - -1. `fleet-observability-prometheus` deploys the metrics backend on the control plane - cluster. -2. The observability connector Secret deploys the Write Gateway and Query Proxy and points - them at that backend. -3. `cluster-collector` deploys collectors that push OTLP metrics to the Write Gateway. -4. `fleet-observability-grafana` deploys Grafana with datasources that read through the - Query Proxy. - -## Deploy to the control plane cluster - -The metrics backend and Grafana run on the control plane cluster, alongside the gateway. -Deploy them once, before creating the connector. - -### Deploy the metrics backend - -Deploy an `ArgoCDApplication` that references `fleet-observability-prometheus`. This -deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform -namespace on your control plane cluster. - - - - -Create the `ArgoCDApplication` object in your project namespace: - -```yaml title="fleet-metrics-backend.yaml" -apiVersion: management.loft.sh/v1 -kind: ArgoCDApplication -metadata: - name: fleet-metrics-backend - namespace: p-my-project # replace with your project namespace -spec: - displayName: "Fleet metrics backend" - destination: - cluster: - name: loft-cluster # replace with your control plane cluster name - templateRef: - name: fleet-observability-prometheus - parameters: - destinationNamespace: vcluster-platform - prometheusStorageSize: 20Gi - project: default -``` - - - - - - - Click in the left sidebar under the infrastructure section, then click the name of the control plane cluster to open its detail page. - - - Select the tab and click . - - - In the section, enter a such as `Fleet metrics backend`. - - - From the dropdown, select the template. - - - Fill the . Keep as `vcluster-platform`, and adjust the storage size or chart version if needed. - - - Click . - - - - - - -| Parameter | Default | Description | -|---|---|---| -| `destinationNamespace` | `vcluster-platform` | Namespace for the Prometheus backend. | -| `project` | `default` | Argo CD project that owns the application. | -| `repoURL` | Prometheus community Helm repo | Helm repository for the Prometheus chart. | -| `version` | `29.14.0` | Prometheus Helm chart version. | -| `prometheusStorageSize` | `20Gi` | Persistent volume size for Prometheus storage. | -| `metricsBackendTLSSecret` | _(empty)_ | Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named `metrics-backend-mtls`. | - -The observability connector you create on the -[install page](./install-observability-gateway.mdx) must use these endpoints when the -template is deployed in its default namespace: - -```yaml -stringData: - metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp - metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local - metricsBackendCertSecretName: metrics-backend-mtls -``` - -The `/api/v1/otlp` path is required for OTLP ingestion. If you change -`destinationNamespace`, replace `vcluster-platform` in both endpoints. If you supply -`metricsBackendTLSSecret`, use that Secret name instead of `metrics-backend-mtls`. - -### Deploy Grafana - -Grafana is optional. Deploy it when you want the pre-provisioned fleet dashboards and -Platform-authenticated access. Reference `fleet-observability-grafana`. - - - - -Create the `ArgoCDApplication` object in your project namespace: - -```yaml title="fleet-grafana.yaml" -apiVersion: management.loft.sh/v1 -kind: ArgoCDApplication -metadata: - name: fleet-grafana - namespace: p-my-project # replace with your project namespace -spec: - displayName: "Fleet Grafana" - destination: - cluster: - name: loft-cluster # replace with your control plane cluster name - templateRef: - name: fleet-observability-grafana - parameters: - destinationNamespace: vcluster-platform - platformHost: platform.example.com - project: default -``` - - - - - - - Click in the left sidebar under the infrastructure section, then click the name of the control plane cluster to open its detail page. - - - Select the tab and click . - - - In the section, enter a such as `Fleet Grafana`. - - - From the dropdown, select the template. - - - Fill the . Set to your Platform host. - - - Click . - - - - - - -| Parameter | Default | Description | -|---|---|---| -| `destinationNamespace` | `vcluster-platform` | Namespace for Grafana. | -| `project` | `default` | Argo CD project that owns the application. | -| `platformHost` | Platform host | Platform host used to authenticate Grafana users. | -| `grafanaAdminGroup` | _(empty)_ | Group mapped to the Grafana admin role. | -| `grafanaCertSecret` | _(empty)_ | TLS Secret for Grafana serving. When empty, Platform self-signs a Secret named `grafana-server-tls`. | - -Configure the fleet observability connector so Platform can proxy and embed the bundled -Grafana dashboards. These values match the template defaults: - -```yaml -stringData: - grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local - grafanaCertSecretName: grafana-server-tls - fleetObservabilityDashboardUid: vcluster-fleet-observability -``` - -If you change `destinationNamespace`, update `grafanaUrl`. If you supply -`grafanaCertSecret`, use that Secret name instead of `grafana-server-tls`. - -## Deploy collectors to each cluster - -Collectors run on the clusters to observe. Deploy one to each cluster once the gateway is -running and a `metrics-writer` key is in place. - -The `cluster-collector` template deploys OpenTelemetry Collectors that push OTLP metrics -to the Write Gateway. Deploy it to each tenant cluster you want to collect from. - -:::note vCluster Standalone tenant clusters -The out-of-the-box collector path relies on the Argo CD integration, which vCluster -Standalone tenant clusters can't use. There is no bundled collector for them. Deploy a -collector manually instead, as described in -[Configure edge collectors](./configure-edge-collectors.mdx). -::: - -Each collector authenticates to the gateway with a `metrics-writer` bearer token. Create a -Secret named `otel-otlp-auth` in the collector namespace before you deploy, or pass the -token through the `bearerToken` parameter. - -```yaml title="otel-otlp-auth.yaml" -apiVersion: v1 -kind: Secret -metadata: - name: otel-otlp-auth - namespace: observability -type: Opaque -stringData: - authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY" -``` - -Create the writer key first, as described in -[Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster) -or [Create a writer key for a control plane cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-control-plane-cluster), -then deploy the collectors: - - - - -Create the `ArgoCDApplication` object in your project namespace: - -```yaml title="fleet-edge-collectors.yaml" -apiVersion: management.loft.sh/v1 -kind: ArgoCDApplication -metadata: - name: fleet-edge-collectors - namespace: p-my-project # replace with your project namespace -spec: - displayName: "Fleet edge collectors" - destination: - virtualCluster: - name: my-vcluster # replace with your tenant cluster name - target: vCluster # deploy into the tenant cluster - templateRef: - name: cluster-collector - parameters: - destinationNamespace: observability - otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 - project: default -``` - - - - - - - Select your project from the projects dropdown at the top of the left navigation bar. - - - Navigate to and open the tenant cluster you want to collect from. - - - Select the tab and click . - - - In the section, enter a such as `Fleet edge collectors`. - - - Select to deploy into the tenant cluster itself. - - - From the dropdown, select the template. - - - Fill the . Set to `observability` and to the Write Gateway endpoint. Use the in-cluster Service endpoint for collectors in the Platform cluster, or the gateway LoadBalancer address for collectors in a tenant cluster elsewhere. - - - Click to add the Argo CD Application to the tenant cluster. - - - - - - -| Parameter | Default | Description | -|---|---|---| -| `destinationNamespace` | `observability` | Namespace for the collectors. | -| `project` | `default` | Argo CD project that owns the application. | -| `otlpEndpoint` | Fleet Write Gateway Service | OTLP HTTP endpoint the collectors push to. Use the in-cluster Service endpoint (`:4318`) for collectors in the Platform cluster, or the gateway LoadBalancer address (`https://:4318`) for collectors elsewhere. See [Expose the gateway with a LoadBalancer](./install-observability-gateway.mdx#expose-the-gateway-with-a-loadbalancer). | -| `bearerToken` | _(empty)_ | `metrics-writer` bearer token. When empty, the collectors read the `otel-otlp-auth` Secret. | -| `otlpCaData` | _(empty)_ | PEM CA data used to verify the gateway serving certificate. | -| `otlpInsecureSkipVerify` | `false` | Skip gateway TLS verification. Use only for test gateways. | - -The collectors stamp the request scope from the destination, so target a tenant cluster -for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set -`destination.cluster` instead of `destination.virtualCluster` to target a control plane -cluster. Don't set `vcluster_platform_*` labels in collector configuration; the Write -Gateway owns that namespace. See [Configure edge collectors](./configure-edge-collectors.mdx) for the -scope headers and label rules. - -## Verify - -Once the applications sync, verify the backend and collectors with the checks in -[Install the observability gateway](./install-observability-gateway.mdx#verify-the-gateway) -and [Configure edge collectors](./configure-edge-collectors.mdx#verify-writes). diff --git a/platform/maintenance/observability/gpu-observability-templates.mdx b/platform/maintenance/observability/gpu-observability-templates.mdx new file mode 100644 index 0000000000..cdb3c9cc7c --- /dev/null +++ b/platform/maintenance/observability/gpu-observability-templates.mdx @@ -0,0 +1,173 @@ +--- +title: Deploy GPU observability with Argo CD +sidebar_label: GPU observability templates +sidebar_position: 7 +description: Overview of the GPU observability Argo CD templates that vCluster Platform ships, the order to deploy them in, which DCGM metric family each produces, and how to tear the stack down. +--- + +When the [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled, +vCluster Platform ships a set of `ArgoCDApplicationTemplate` resources for GPU +observability. Together they deploy the NVIDIA GPU stack into a cluster that runs GPU +nodes, typically a tenant cluster with private nodes. The GPU Operator provides drivers +and telemetry, the Fleet Intelligence Agent provides node-level GPU health reporting, and +NVSentinel provides fault detection and remediation. A fourth template, cert-manager, +supplies the internal TLS certificates some of those components need. + +This page summarizes the templates, the order to deploy them in, and the metric +behaviors to know before you build queries on top of them. For the NVSentinel deploy +guide and parameter reference, see +[Observe GPU nodes with NVSentinel](./nvsentinel-gpu-observability.mdx). + +## Bundled GPU templates + +In an `ArgoCDApplication`, reference each template by its resource name through +`spec.templateRef.name`. The Platform UI lists them by display name. For the general +template and application model, see +[Deploy applications](../../integrations/argocd/deploy-applications.mdx). + +| Template (Platform UI) | `templateRef.name` | Deploys | Default namespace | +|---|---|---|---| +| cert-manager | `cert-manager` | cert-manager, for the internal TLS certificates NVSentinel components need | `cert-manager` | +| NVIDIA GPU Operator | `nvidia-gpu-operator` | NVIDIA drivers, device plugin, DCGM, and dcgm-exporter | `gpu-operator` | +| Fleet Intelligence Agent | `fleet-intelligence-agent` | NVIDIA's gpud-based node agent that reports GPU health and DCGM metrics | `fleet-intelligence` | +| NVSentinel | `nvsentinel` | GPU fault detection, quarantine, and remediation | `nvsentinel` | + +## Deployment order + +The applications depend on each other at runtime, so deploy them in this order: + +1. **cert-manager**, if you plan to enable NVSentinel components that need internal TLS, + such as the MongoDB store or the Janitor. See + [the NVSentinel prerequisites](./nvsentinel-gpu-observability.mdx) for exactly which + components require it. +2. **NVIDIA GPU Operator.** Everything else builds on it. It installs the drivers and + device plugin, and runs the DCGM service that NVSentinel reads GPU health from, plus + dcgm-exporter for GPU metrics. +3. **Fleet Intelligence Agent**, optionally, for node-level GPU health reporting. The + bundled GPU dashboard reads some panels from its metrics. +4. **NVSentinel**, to detect and remediate GPU faults. See + [Observe GPU nodes with NVSentinel](./nvsentinel-gpu-observability.mdx). + +## GPU metrics and attribution + +When the GPU templates are deployed, two separate exporters report metrics for the same +GPUs, and by default those metrics don't show which pod used a GPU. Both behaviors are +intentional. Keep them in mind when you write your own queries or bill tenants for +their GPU usage based on the bundled dashboards. + +### Two DCGM metric families + +The same physical GPU is reported by two exporters when the full GPU stack is deployed: + +| Source | Deployed by | Metric family | Example | +|---|---|---|---| +| dcgm-exporter | NVIDIA GPU Operator template | UPPERCASE `DCGM_FI_DEV_*` | `DCGM_FI_DEV_GPU_UTIL` | +| gpud | Fleet Intelligence Agent template | lowercase `dcgm_fi_dev_*` | `dcgm_fi_dev_gpu_util` | + +The bundled GPU dashboard mixes both families. Some panels read the dcgm-exporter series +and others read the Fleet Intelligence Agent series. + +:::warning Pick one metric family per query +Aggregating across both families counts every GPU twice, for example with a regular +expression matcher that is case-insensitive or that spans both spellings. When you write +your own PromQL, pick one family per query. +::: + +### Per-pod GPU attribution is off by default + +The NVIDIA GPU Operator template ships dcgm-exporter with +`dcgmExporterEnablePodLabels: "false"` and `dcgmExporterEnablePodUID: "false"`. GPU +metrics then carry node, GPU, UUID, and `modelName` labels but **no pod or namespace +label**, so GPU usage can't be attributed to a workload: + +```text +DCGM_FI_DEV_GPU_UTIL{gpu="0", UUID="GPU-9d9d49d7-...", modelName="Tesla T4", Hostname="ip-192-168-75-24"} 0 +``` + +To attribute GPU usage to workloads and bill tenants for it, set both parameters to +`"true"` on the NVIDIA GPU Operator application and let it sync. + +:::note Benign crashloop on startup +With pod labels enabled, dcgm-exporter can restart a few times right after rollout while +its ServiceAccount token is projected, logging +`Failed to get in-cluster config, pod labels will not be available`. It heals on its own +within a few restarts. No action is needed. +::: + +## The cluster-collector template scrapes GPU metrics automatically + +If you deploy edge collectors with the `cluster-collector` template, described in +[Deploy collectors with Argo CD](./configure-edge-collectors.mdx#deploy-collectors-with-argo-cd), +it also scrapes the GPU stack. No extra collector configuration is needed. Alongside its +per-node DaemonSet, the template deploys a cluster-singleton collector preconfigured with +scrape jobs for: + +- The Fleet Intelligence Agent, in the `fleet-intelligence` namespace. +- The NVIDIA GPU Operator's dcgm-exporter, in the `gpu-operator` namespace. +- NVSentinel's GPU Health Monitor, in the `nvsentinel` namespace. + +These namespaces are hardcoded in the collector template. They aren't derived from the +GPU templates' `destinationNamespace` parameters, so this only works when you keep those +parameters at their defaults, shown in +[Bundled GPU templates](#bundled-gpu-templates) above. If you deploy a GPU template into a +non-default namespace, its metrics stop reaching Fleet Observability through this +collector. If you skip a component, such as the optional Fleet Intelligence Agent, its +scrape job simply finds no targets. + +## Tear down the GPU stack + +Deleting the `ArgoCDApplication` resources prunes the workloads, but some artifacts are +retained by design and stay in the cluster until you remove them. These are NVSentinel's +MongoDB PVCs, the cert-manager CRDs (`keepCRDs: true`), and the destination namespaces, +which stay `Active` with zero pods. + +Target the tenant cluster's `kubectl` context for every command below. CRDs and +namespaces are cluster-wide, and running these against the wrong cluster, such as your +control plane cluster, can remove cert-manager or a namespace that other workloads there +still depend on. + +First remove NVSentinel's retained data and namespace, described in +[Uninstall and clean up NVSentinel](./nvsentinel-gpu-observability.mdx#uninstall-and-cleanup). +Then remove the remaining GPU namespaces: + +```bash title="Clean up GPU stack namespaces" +TENANT_CLUSTER_CONTEXT=my-vcluster +kubectl --context "$TENANT_CLUSTER_CONTEXT" delete namespace gpu-operator fleet-intelligence +``` + +### Remove cert-manager entirely (optional) + +CRDs are cluster-wide type definitions, so they can't be scoped to the GPU templates. +The only choice is to keep them or remove them for the whole cluster. Run this step only +when cert-manager was installed solely for the GPU stack and nothing else in the cluster +uses it. Deleting a CRD cascade-deletes every resource of that type in every namespace. + +```bash title="Remove cert-manager CRDs and namespace" +TENANT_CLUSTER_CONTEXT=my-vcluster + +# Review what exists first +kubectl --context "$TENANT_CLUSTER_CONTEXT" get crd -o name | grep -E '\.cert-manager\.io$' + +kubectl --context "$TENANT_CLUSTER_CONTEXT" delete crd \ + certificates.cert-manager.io \ + certificaterequests.cert-manager.io \ + challenges.acme.cert-manager.io \ + clusterissuers.cert-manager.io \ + issuers.cert-manager.io \ + orders.acme.cert-manager.io + +kubectl --context "$TENANT_CLUSTER_CONTEXT" delete namespace cert-manager +``` + +None of these carry blocking finalizers, so deletion completes cleanly. + +## Related pages + +- [Observe GPU nodes with NVSentinel](./nvsentinel-gpu-observability.mdx) for the NVSentinel + deploy guide, dry-run semantics, and parameter reference. +- [Install the observability gateway](./install-observability-gateway.mdx) for the + metrics backend and gateway setup. +- [Configure edge collectors](./configure-edge-collectors.mdx) for deploying collectors + with Argo CD or by hand. +- [Query fleet metrics](./query-fleet-metrics.mdx) for scoped PromQL access to GPU + metrics. diff --git a/platform/maintenance/observability/install-observability-gateway.mdx b/platform/maintenance/observability/install-observability-gateway.mdx index db7f058eba..ecc41f8957 100644 --- a/platform/maintenance/observability/install-observability-gateway.mdx +++ b/platform/maintenance/observability/install-observability-gateway.mdx @@ -1,9 +1,8 @@ --- title: Install the observability gateway sidebar_label: Install gateway -sidebar_position: 2 +sidebar_position: 3 description: Create the Fleet Observability connector that deploys the Write Gateway and Query Proxy. -sidebar_class_name: pro --- import Tabs from "@theme/Tabs"; @@ -34,10 +33,102 @@ Before you create the connector, prepare: :::note The observability gateway doesn't install or size the metrics backend for you. Bring your -own backend, or deploy the bundled Prometheus backend out of the box with the Argo CD -integration. See [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). +own, or deploy the bundled Prometheus backend with Argo CD as described below. ::: +## Deploy the metrics backend with Argo CD + +If you use the [Argo CD integration](../../integrations/argocd/overview.mdx), deploy the +bundled Prometheus backend with the `fleet-observability-prometheus` +`ArgoCDApplicationTemplate` instead of installing and sizing a backend by hand. The backend +runs on the [local cluster](../../introduction/architecture.mdx#connected-clusters-and-agents), +the cluster where Platform itself is installed. Deploy it once, before you create the +connector below. + +Deploy an `ArgoCDApplication` that references `fleet-observability-prometheus`. This +deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform +namespace on the local cluster. + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="fleet-metrics-backend.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-metrics-backend + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet metrics backend" + destination: + cluster: + name: loft-cluster # the reserved name for Platform's local cluster; don't change this + templateRef: + name: fleet-observability-prometheus + parameters: + destinationNamespace: vcluster-platform + prometheusStorageSize: 20Gi + project: default +``` + + + + + + + Click in the left sidebar under the infrastructure section, then click the local cluster (named `loft-cluster` by default) to open its detail page. + + + Select the tab and click . + + + In the section, enter a such as `Fleet metrics backend`. + + + From the dropdown, select the template. + + + Fill the . Keep as `vcluster-platform`, and adjust the storage size or chart version if needed. + + + Click . + + + + + + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `vcluster-platform` | Namespace for the Prometheus backend. | +| `project` | `default` | Argo CD project that owns the application. | +| `repoURL` | Prometheus community Helm repo | Helm repository for the Prometheus chart. | +| `version` | `29.14.0` | Prometheus Helm chart version. | +| `prometheusStorageSize` | `20Gi` | Persistent volume size for Prometheus storage. | +| `metricsBackendTLSSecret` | _(empty)_ | Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named `metrics-backend-mtls`. | + +The connector you create below must use these endpoints when the template is deployed in +its default namespace: + +```yaml +stringData: + metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp + metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local + metricsBackendCertSecretName: metrics-backend-mtls +``` + +The `/api/v1/otlp` path is required for OTLP ingestion. If you change +`destinationNamespace`, replace `vcluster-platform` in both endpoints. If you supply +`metricsBackendTLSSecret`, use that Secret name instead of `metrics-backend-mtls`. + ## Create the connector Secret Create the Fleet Observability connector in the Platform namespace. The connector ID can be @@ -139,6 +230,19 @@ Platform reconciles the following resources: | `grafanaCertSecretName` | No | Grafana serving certificate Secret. Platform uses its `ca.crt` to trust the Grafana upstream. | | `fleetObservabilityDashboardUid` | No | UID of the fleet dashboard embedded by the Fleet Observability page. | +:::important Bring-your-own Grafana requirements +When you point `grafanaUrl` at a Grafana you manage instead of the bundled Grafana, that +Grafana must: + +- **Serve from the `/grafana` subpath.** Set `server.root_url` and + `server.serve_from_sub_path` so Grafana works behind the Platform `/grafana/` reverse + proxy. See + [`serve_from_sub_path`](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#serve_from_sub_path). +- **Allow embedding.** Set `security.allow_embedding = true` so the Fleet Observability page + and each tenant cluster's Observability tab can embed the dashboards in an iframe. See + [`allow_embedding`](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#allow_embedding). +::: + ## Customize the gateway serving certificate The gateway always serves OTLP and query traffic over TLS. When you don't specify a diff --git a/platform/maintenance/observability/labels-reference.mdx b/platform/maintenance/observability/labels-reference.mdx index a6e9a8e532..f5a8b6e861 100644 --- a/platform/maintenance/observability/labels-reference.mdx +++ b/platform/maintenance/observability/labels-reference.mdx @@ -1,9 +1,8 @@ --- title: Fleet Observability labels reference sidebar_label: Labels reference -sidebar_position: 7 +sidebar_position: 9 description: Reference for vcluster_platform labels stamped by the Fleet Observability Write Gateway. -sidebar_class_name: pro --- The Write Gateway owns the full `vcluster_platform_*` label namespace. These labels are diff --git a/platform/maintenance/observability/nvsentinel-gpu-observability.mdx b/platform/maintenance/observability/nvsentinel-gpu-observability.mdx new file mode 100644 index 0000000000..272c6813ef --- /dev/null +++ b/platform/maintenance/observability/nvsentinel-gpu-observability.mdx @@ -0,0 +1,371 @@ +--- +title: Observe GPU nodes with NVSentinel +sidebar_label: NVSentinel GPU observability +sidebar_position: 8 +description: Deploy NVIDIA NVSentinel with the bundled Argo CD template to detect, quarantine, and remediate GPU node faults, and understand dry-run mode and teardown cleanup. +--- + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import Flow, { Step } from "@site/src/components/Flow"; +import Button from "@site/src/components/Button"; +import Label from "@site/src/components/Label"; + +[NVSentinel](https://github.com/NVIDIA/NVSentinel) is NVIDIA's open source, Kubernetes-native +service for GPU fault handling. It detects hardware and software faults on GPU nodes through +DCGM, system logs, and cloud provider maintenance events. It can respond by cordoning nodes, +draining workloads, and triggering node reboots or terminations. + +When the [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled, vCluster +Platform ships an `ArgoCDApplicationTemplate` for NVSentinel alongside the other +[GPU observability templates](./gpu-observability-templates.mdx). You reference the template from an `ArgoCDApplication` to deploy +NVSentinel into a cluster that runs GPU nodes, typically a tenant cluster with private +nodes. NVSentinel metrics then flow through the Fleet Observability pipeline like any other +workload metrics and appear on the bundled GPU dashboards. + +## Prerequisites + +- The [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled. +- An Argo CD connector is registered on the target cluster. See [Connect to Argo CD](../../integrations/argocd/connect-argocd.mdx). +- The [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) is deployed in the same cluster, per the [deployment order](./gpu-observability-templates.mdx#deployment-order). +- cert-manager is deployed, if any component you enable needs internal TLS. See [When you need cert-manager](#when-you-need-cert-manager). +- A default StorageClass exists, or one is named in the template parameters, if any component you enable needs the MongoDB store. +- Prometheus Operator CRDs exist, if you enable the PodMonitor. + +## What the template deploys + +The template deploys the upstream `nvsentinel` Helm chart from `oci://ghcr.io/nvidia/nvsentinel`. +In an `ArgoCDApplication`, reference it through `spec.templateRef.name: nvsentinel`. The +Platform UI lists it as **NVSentinel**. + +NVSentinel is composed of independent components that you enable per deployment: + +| Component | Default | What it does | +|---|---|---| +| GPU Health Monitor | `true` | Watches GPU hardware through DCGM for thermal issues, ECC errors, and XID events. | +| Syslog Health Monitor | `true` | Scans system logs on each node for fault patterns. | +| NIC Health Monitor | `false` | Watches network interface health. | +| CSP Health Monitor | `false` | Pulls maintenance events from cloud provider APIs. | +| Fault Quarantine | `false` | Cordons and taints nodes with detected faults. | +| Node Drainer | `false` | Evicts workloads from quarantined nodes. | +| Fault Remediation | `false` | Creates maintenance resources to recover failing nodes after drain completes. | +| Janitor | `false` | Executes the node reboots and terminations that Fault Remediation requests. | +| Health Events Analyzer | `false` | Analyzes patterns across health events and produces recommended actions. | +| MongoDB Store | `false` | Persistent event storage; the coordination layer for the remediation components. | + +With the defaults, NVSentinel only observes health events and metrics. Nothing acts on nodes. The remediation pipeline (Fault Quarantine, Node Drainer, Fault Remediation, Janitor) is opt-in. + +### Component dependencies the template resolves for you + +Some components can't start without others. The template enables the dependencies +automatically, so the deployed application can differ from the toggles you set: + +- **MongoDB store** turns on whenever Fault Quarantine, Node Drainer, Fault Remediation, + or Health Events Analyzer is enabled, even if you leave **Enable MongoDB Store** off. + Those components coordinate through MongoDB change streams and crash without it. +- **Janitor** turns on whenever Fault Remediation is enabled, because Fault Remediation + can't reboot or terminate nodes without it. +- The Janitor's default provider is `kind`, which only **simulates** reboots and + terminations. Set the **Janitor Provider** parameter to your cloud provider (or + `generic` for bare metal) before relying on real remediation. See + [Janitor provider parameters](#janitor-provider-parameters). + +:::note +Node Drainer is the exception. It needs Fault Quarantine to have anything to drain, but +the template doesn't turn on Fault Quarantine for you the way it does for the +dependencies above. Set both `enableFaultQuarantine` and `enableNodeDrainer` to `"true"` +yourself, as the [deploy example](#deploy-nvsentinel) does. Enabling Node Drainer alone +deploys a drainer with no quarantined nodes to evict from. +::: + +### When you need cert-manager + +With the observe-only defaults, NVSentinel runs without cert-manager. It becomes required when you enable any of the following: + +- **The internal MongoDB store.** TLS/mTLS is enabled by default, and cert-manager creates the server, client, and CA certificates. Fault Quarantine, Node Drainer, and Health Events Analyzer turn the MongoDB store on automatically, so they indirectly require cert-manager too. +- **The Janitor or Fault Remediation.** The Janitor uses TLS certificates for its validating webhook and its provider gRPC endpoint. +- **Chart features that aren't exposed as template parameters**, such as preflight checks (the mutating admission webhook needs TLS, and cert-manager is the default provider) and internal PostgreSQL with the chart's TLS configuration. + +## Deploy NVSentinel + +Deploy NVSentinel into the cluster whose GPU nodes it should watch. For a tenant cluster +with private GPU nodes, target the tenant cluster itself. + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="nvsentinel.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: nvsentinel + namespace: p-my-project # replace with your project namespace +spec: + displayName: "NVSentinel" + destination: + virtualCluster: + name: my-vcluster # replace with your tenant cluster name + target: vCluster # deploy into the tenant cluster + templateRef: + name: nvsentinel + parameters: + destinationNamespace: nvsentinel + project: default + dryRun: "true" # keep dry run on until you have verified detection + enableFaultQuarantine: "true" + enableNodeDrainer: "true" +``` + + + + + + + Select your project from the projects dropdown at the top of the left navigation bar. + + + Navigate to and open the tenant cluster with the GPU nodes to monitor. + + + Select the tab and click . + + + In the section, enter a such as `NVSentinel`. + + + Select to deploy into the tenant cluster itself. + + + From the dropdown, select the template. + + + Fill the . Keep enabled for the first rollout, and enable the components you need. The DCGM defaults match a standard GPU Operator install. + + + Click . + + + + + + +## Dry run + +The template defaults to `dryRun: "true"`. As with any dry run, NVSentinel runs the full +detection and decision pipeline but skips changes to node state or workloads. **Dry run still writes real node annotations and labels, and still increments the metrics that feed the bundled GPU dashboard.** + +When a fault is detected in dry-run mode, NVSentinel: + +- Logs the intended action, for example `Cordoning node`. +- Writes its own annotations and the `dgxc.nvidia.com/nvsentinel-state` label on the node to track the quarantine decision. +- Increments quarantine metrics such as `fault_quarantine_current_quarantined_nodes` and `fault_quarantine_cordons_applied_total`, which feed the **Quarantined Nodes** panel on the bundled GPU dashboard. + +During a dry run, NVSentinel doesn't: + +- Cordon the node. `kubectl get node -o jsonpath='{.spec.unschedulable}'` stays empty and the node keeps accepting workloads. +- Evict workloads. +- Reboot or terminate nodes. + +Because the metrics increment but nodes are not cordoned or evicted during a dry run, both the node object and the dashboard can look like a genuine quarantine happened. + +Use dry run to validate detection end to end. Inject or wait for a health event, confirm the logs, node annotations, and metrics, and then set `dryRun: "false"` to let NVSentinel cordon, drain, and remediate for real. + +Before turning dry run off, make sure the Janitor provider is set to a real provider, since the default `kind` provider only simulates reboots and terminations. + +## Uninstall and cleanup + +Deleting the NVSentinel `ArgoCDApplication` prunes the workloads, but two artifacts are +retained by design and stay in the cluster until you remove them: + +- **MongoDB PVCs.** The three `datadir-mongodb-*` PVCs (10Gi each by default, 30Gi + total) survive uninstall so health-event history isn't destroyed by accident. +- **Namespace.** The destination namespace (`nvsentinel`) stays `Active` with zero pods. + Deleting it also removes the namespaced cert-manager resources (Certificates and + Issuers) that NVSentinel created. + +```bash title="Clean up retained NVSentinel artifacts" +# MongoDB data (destroys health-event history) +kubectl delete pvc -n nvsentinel datadir-mongodb-0 datadir-mongodb-1 datadir-mongodb-2 + +kubectl delete namespace nvsentinel +``` + +Neither carries blocking finalizers, so deletion completes cleanly. + +:::warning +Deleting the MongoDB PVCs permanently deletes NVSentinel's health-event history. +::: + +To remove the rest of the GPU stack, including the retained cert-manager CRDs, see +[Tear down the GPU stack](./gpu-observability-templates.mdx#tear-down-the-gpu-stack). + +## Parameters + +The template groups parameters by component. All boolean parameters are passed as strings +(`"true"` / `"false"`). + +### Core parameters + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `nvsentinel` | Namespace NVSentinel deploys into. | +| `project` | `default` | Argo CD project that owns the application. | +| `version` | `v1.13.0` | NVSentinel Helm chart version. | +| `repoURL` | `oci://ghcr.io/nvidia/nvsentinel` | Chart repository. Change to use a mirror or private registry. | +| `releaseName` | `nvsentinel` | Helm release name. | +| `chart` | `nvsentinel` | Chart name pulled from the repository. | +| `imageTag` | `v1.13.0` | NVSentinel image tag. | +| `dryRun` | `true` | Log and record intended actions without cordoning, draining, or remediating. See [Dry run](#dry-run). | + +### DCGM parameters + +| Parameter | Default | Description | +|---|---|---| +| `dcgmEndpoint` | `nvidia-dcgm.gpu-operator.svc` | DCGM service the GPU Health Monitor reads from. The default matches a standard GPU Operator install. | +| `dcgmPort` | `5555` | DCGM service port. | + +### Component toggles + +| Parameter | Default | Description | +|---|---|---| +| `enableGpuHealthMonitor` | `true` | GPU hardware health through DCGM. | +| `enableSyslogHealthMonitor` | `true` | System log fault patterns. | +| `enableNicHealthMonitor` | `false` | Network interface health. | +| `enableCspHealthMonitor` | `false` | Cloud provider maintenance events. | +| `enableFaultQuarantine` | `false` | Cordon and taint faulty nodes. Turns on the MongoDB store automatically. | +| `enableNodeDrainer` | `false` | Evict workloads from quarantined nodes. Turns on the MongoDB store automatically, but not Fault Quarantine. [Enable that yourself too](#component-dependencies-the-template-resolves-for-you). | +| `enableFaultRemediation` | `false` | Create maintenance resources to recover failing nodes. Turns on the MongoDB store and the Janitor automatically. | +| `enableJanitor` | `false` | Execute reboots and terminations. Turns on automatically with Fault Remediation. | +| `enableHealthEventsAnalyzer` | `false` | Analyze patterns across health events. Turns on the MongoDB store automatically. | + +### MongoDB store parameters + +| Parameter | Default | Description | +|---|---|---| +| `enableMongodbStore` | `false` | Enable the MongoDB datastore explicitly. Turns on automatically with the components listed above. | +| `mongodbStorageClass` | _(empty)_ | StorageClass for the MongoDB PVCs. Empty uses the cluster default. | +| `mongodbStorageSize` | `10Gi` | Volume size per MongoDB replica. The chart runs three replicas, so the default provisions 30Gi total. | + +### Node labeler parameters + +| Parameter | Default | Description | +|---|---|---| +| `assumeDriverInstalled` | `false` | Enable when NVIDIA drivers are preinstalled on the host rather than deployed by GPU Operator driver pods (`driver.enabled=false`). The labeler then marks all GPU nodes as having the driver installed instead of watching for driver pods. Without it there are no driver pods to detect, so the syslog and metadata DaemonSets never start on those nodes. | + +### Metrics and operations parameters + +| Parameter | Default | Description | +|---|---|---| +| `enablePodMonitor` | `false` | Create a PodMonitor for NVSentinel metrics. Requires Prometheus Operator CRDs. | +| `podMonitorInterval` | `30s` | PodMonitor scrape interval. | +| `enableNetworkPolicy` | `true` | Restrict ingress to metrics ports. Disable if it blocks a same-namespace webhook such as cert-manager. | +| `enableTracing` | `false` | Enable OpenTelemetry tracing. | +| `tracingEndpoint` | _(empty)_ | OTLP gRPC endpoint (`host:port`) for trace ingestion. Required when tracing is enabled. | +| `enableAuditLogging` | `false` | Enable audit logging. | + +### Janitor provider parameters + +The Janitor talks to your infrastructure provider to reboot and terminate nodes. Pick the +provider with `janitorProvider` and fill the matching provider section. The `kind` and +`kwok` providers only simulate actions and are meant for test clusters. + +| Parameter | Default | Description | +|---|---|---| +| `janitorProvider` | `kind` | One of `kind` or `kwok` (both simulated), `aws`, `gcp`, `azure`, `oci`, `nebius`, or `generic` (bare metal or on-prem). | +| `janitorServiceAccountAnnotationKey` | _(empty)_ | Annotation key on the Janitor ServiceAccount for cloud identity, for example `eks.amazonaws.com/role-arn` or `iam.gke.io/gcp-service-account`. Set together with the value. | +| `janitorServiceAccountAnnotationValue` | _(empty)_ | Value for the annotation key above, for example the IAM role ARN or GCP service account email. | + +Select the tab for your Janitor provider: + + + + +| Parameter | Default | Description | +|---|---|---| +| `janitorAwsRegion` | _(empty)_ | AWS region of the EKS cluster (for example `us-east-1`). The Janitor uses this one region for all EC2 calls, so every node must live in it. Multiple availability zones within the region are fine. | +| `janitorAwsAccountId` | _(empty)_ | AWS account ID (12 digits), required for IRSA. | +| `janitorAwsIamRoleName` | _(empty)_ | IAM role name for the Janitor ServiceAccount. The role needs `ec2:RebootInstances` and `ec2:TerminateInstances`. | + + + + +| Parameter | Default | Description | +|---|---|---| +| `janitorGcpProject` | _(empty)_ | GCP project ID of the GKE cluster. Optional; the Janitor auto-detects each node's project from its `providerID`. | +| `janitorGcpZone` | _(empty)_ | GCP zone where cluster nodes run (for example `us-central1-a`). Optional; auto-detected per node from its `providerID`. | +| `janitorGcpServiceAccount` | _(empty)_ | GCP service account name for Workload Identity, without the `@project.iam.gserviceaccount.com` suffix. Needs `compute.instances.reset`. | + + + + +| Parameter | Default | Description | +|---|---|---| +| `janitorAzureSubscriptionId` | _(empty)_ | Azure subscription ID (UUID). | +| `janitorAzureResourceGroup` | _(empty)_ | Resource group of the AKS cluster. Derived per node from its `providerID`, so set it only if detection fails. | +| `janitorAzureLocation` | _(empty)_ | Azure region of the AKS cluster (for example `eastus`). Derived per node from its `providerID`, so not normally required. | +| `janitorAzureClientId` | _(empty)_ | Managed identity client ID (UUID) for Workload Identity. The identity needs the Virtual Machine Contributor role. | + + + + +| Parameter | Default | Description | +|---|---|---| +| `janitorOciRegion` | _(empty)_ | OCI region of the OKE cluster (for example `us-ashburn-1`). One Janitor works in a single region, so all nodes must be in it. | +| `janitorOciCompartment` | _(empty)_ | Compartment OCID containing cluster resources. | +| `janitorOciCredentialsFile` | _(empty)_ | Path to an OCI credentials file. Leave empty to use workload identity. | +| `janitorOciProfile` | `DEFAULT` | Profile name in the credentials file. | +| `janitorOciPrincipalId` | _(empty)_ | Principal OCID for Workload Identity (the dynamic group or service principal OCID). | + + + + +| Parameter | Default | Description | +|---|---|---| +| `janitorNebiusServiceAccountKeySecret` | _(empty)_ | Name of an existing Kubernetes Secret holding the Nebius service account key as `sa-credentials.json`. The service account needs `compute.instances.stop` and `compute.instances.start`. Zone and region are auto-detected per node. | + + + + +The `generic` provider reboots nodes with a privileged Job scheduled onto the node. It +can't terminate instances. + +| Parameter | Default | Description | +|---|---|---| +| `janitorGenericRebootImage` | _(empty)_ | Container image for the reboot Job. Empty uses the chart default (busybox). | +| `janitorGenericUseSysrqReboot` | `false` | Use Linux Magic SysRq for an immediate kernel reboot instead of a `chroot /host reboot`. Enable only when nodes allow SysRq through `/proc/sysrq-trigger`. | +| `janitorGenericRebootJobNamespace` | _(empty)_ | Namespace for reboot Jobs. Empty uses the release namespace. | +| `janitorGenericRebootJobTTLSeconds` | _(empty)_ | TTL in seconds for completed reboot Jobs. Empty uses the chart default (3600). | +| `janitorGenericImagePullSecrets` | _(empty)_ | Comma-separated image pull Secret names for the reboot Job. | + + + + +## Related pages + +- [GPU observability templates](./gpu-observability-templates.mdx) for the full GPU template + stack, deployment order, DCGM metric families, and per-pod GPU attribution. +- [Install the observability gateway](./install-observability-gateway.mdx) for the + metrics backend and gateway setup. +- [Configure edge collectors](./configure-edge-collectors.mdx) for deploying collectors + with Argo CD or by hand. +- [Query fleet metrics](./query-fleet-metrics.mdx) for scoped PromQL access to NVSentinel + and DCGM metrics. +- [Troubleshooting](./troubleshooting.mdx) for gateway and collector failures. diff --git a/platform/maintenance/observability/overview.mdx b/platform/maintenance/observability/overview.mdx index 579c127775..7791ccc15d 100644 --- a/platform/maintenance/observability/overview.mdx +++ b/platform/maintenance/observability/overview.mdx @@ -3,7 +3,6 @@ title: Fleet Observability sidebar_label: Overview sidebar_position: 1 description: Monitor tenant clusters across your vCluster Platform fleet with an observability gateway, scoped access keys, and platform-owned metric labels. -sidebar_class_name: pro --- import FleetObservabilityArchitecture from '@site/static/media/diagrams/fleet-observability-architecture.svg'; @@ -23,17 +22,11 @@ shared nodes, private nodes, or connected clusters. ## Get started -The default setup is out of the box and driven from the Platform UI. Three steps: - -1. **Deploy the metrics backend and Grafana.** Use the bundled Argo CD templates to deploy - Prometheus and Grafana. See [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). -2. **Create the observability connector.** In the Platform UI, open **Connectors** and - connect your backend on the **Observability** tab. Platform then deploys the gateway. See - [Install the observability gateway](./install-observability-gateway.mdx). -3. **Deploy the edge collectors.** Create a `metrics-writer` access key, then deploy the - bundled collector template to each cluster to collect from. See - [Configure metrics access](./configure-metrics-access.mdx) and - [Deploy collectors to each cluster](./deploy-with-argocd.mdx#deploy-collectors-to-each-cluster). +Follow the [Fleet Observability quickstart](./quickstart.mdx) to deploy the default +pipeline end to end. It covers the metrics backend, gateway, access keys, and edge +collectors, using the bundled Argo CD templates. For other backends, manual collector +configuration, or control plane cluster targets, see the individual setup pages in the +page guide below. :::info Scope Fleet Observability is currently intended for platform administrators. It provides the @@ -92,10 +85,10 @@ You must provide a backend that can import OTLP metrics and serve the Prometheus API. You must also configure the edge collectors that send metrics to the gateway. If you enable the [Argo CD integration](../../integrations/argocd/overview.mdx), -Platform ships default `ArgoCDApplicationTemplate` resources for the metrics backend -and edge collectors. You can leverage these templates to create the `ArgoCDApplication` -resources that deploy the backend and collectors, instead of wiring them up by hand. -See [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). +Platform ships default `ArgoCDApplicationTemplate` resources for the metrics backend, +Grafana, and edge collectors. You can leverage these templates to create the +`ArgoCDApplication` resources that deploy them, instead of wiring them up by hand. Each +setup page below covers its own Argo CD template alongside the manual path. ## Security model @@ -130,11 +123,13 @@ The bundled components are the default. Replace any of them as your environment | Task | Page | |---|---| -| Deploy the metrics backend and Grafana with Argo CD | [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx) | -| Create the connector and deploy the gateway | [Install the observability gateway](./install-observability-gateway.mdx) | +| Deploy the default pipeline end to end with Argo CD | [Fleet Observability quickstart](./quickstart.mdx) | +| Deploy the metrics backend, then create the connector and gateway | [Install the observability gateway](./install-observability-gateway.mdx) | | Create scoped metrics access keys | [Configure metrics access](./configure-metrics-access.mdx) | | Send metrics from collectors to the gateway | [Configure edge collectors](./configure-edge-collectors.mdx) | -| Query metrics through Grafana or PromQL | [Query fleet metrics](./query-fleet-metrics.mdx) | +| Deploy Grafana and query metrics through it or PromQL | [Query fleet metrics](./query-fleet-metrics.mdx) | +| Deploy the GPU observability stack | [GPU observability templates](./gpu-observability-templates.mdx) | +| Detect and remediate GPU node faults | [Observe GPU nodes with NVSentinel](./nvsentinel-gpu-observability.mdx) | | Understand the platform label contract | [Labels reference](./labels-reference.mdx) | | Diagnose common failures | [Troubleshooting](./troubleshooting.mdx) | diff --git a/platform/maintenance/observability/query-fleet-metrics.mdx b/platform/maintenance/observability/query-fleet-metrics.mdx index 1cc858d6e2..28fd6e1568 100644 --- a/platform/maintenance/observability/query-fleet-metrics.mdx +++ b/platform/maintenance/observability/query-fleet-metrics.mdx @@ -3,21 +3,110 @@ title: Query fleet metrics sidebar_label: Query metrics sidebar_position: 6 description: Query fleet metrics through the Fleet Observability Query Proxy with scoped headers and platform-owned labels. -sidebar_class_name: pro --- +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import Flow, { Step } from "@site/src/components/Flow"; +import Button from "@site/src/components/Button"; +import Label from "@site/src/components/Label"; + Query fleet metrics through the bundled Grafana, the fastest path once it's deployed, or directly against the Query Proxy's Prometheus-compatible HTTP API for automation and your own query clients. In both cases the Query Proxy authorizes each request with Platform, injects the requested scope into PromQL, and forwards the rewritten query to your metrics backend. +## Deploy the bundled Grafana with Argo CD + +Grafana is optional. Deploy it when you want the pre-provisioned fleet dashboards and +Platform-authenticated access. If you use the +[Argo CD integration](../../integrations/argocd/overview.mdx), reference the +`fleet-observability-grafana` `ArgoCDApplicationTemplate`. + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="fleet-grafana.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-grafana + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet Grafana" + destination: + cluster: + name: loft-cluster # the reserved name for Platform's local cluster; don't change this + templateRef: + name: fleet-observability-grafana + parameters: + destinationNamespace: vcluster-platform + platformHost: platform.example.com + project: default +``` + + + + + + + Click in the left sidebar under the infrastructure section, then click the local cluster (named `loft-cluster` by default) to open its detail page. + + + Select the tab and click . + + + In the section, enter a such as `Fleet Grafana`. + + + From the dropdown, select the template. + + + Fill the . Set to your Platform host. + + + Click . + + + + + + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `vcluster-platform` | Namespace for Grafana. | +| `project` | `default` | Argo CD project that owns the application. | +| `platformHost` | Platform host | Platform host used to authenticate Grafana users. | +| `grafanaAdminGroup` | _(empty)_ | Group mapped to the Grafana admin role. | +| `grafanaCertSecret` | _(empty)_ | TLS Secret for Grafana serving. When empty, Platform self-signs a Secret named `grafana-server-tls`. | + +Configure the fleet observability connector so Platform can proxy and embed the bundled +Grafana dashboards. These values match the template defaults: + +```yaml +stringData: + grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local + grafanaCertSecretName: grafana-server-tls + fleetObservabilityDashboardUid: vcluster-fleet-observability +``` + +If you change `destinationNamespace`, update `grafanaUrl`. If you supply +`grafanaCertSecret`, use that Secret name instead of `grafana-server-tls`. + ## Query with the bundled Grafana -If you deploy the Fleet Observability: Grafana template, Platform runs a bundled Grafana -pre-equipped with fleet dashboards and scoped datasources. It is the fastest way to explore -fleet metrics without writing PromQL by hand. See -[Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx#deploy-grafana) to deploy it. +Once deployed, Platform runs a bundled Grafana pre-equipped with fleet dashboards and +scoped datasources. It is the fastest way to explore fleet metrics without writing PromQL +by hand. ### Sign in @@ -74,9 +163,10 @@ its scoped Cluster Metrics dashboard. Platform auto-provisions Prometheus datasources that point at the Query Proxy. You don't create them by hand: -- A **fleet-wide** datasource with no scope headers, for fleet and platform-admin views. +- A **fleet-wide** datasource with no scope headers, for fleet and platform-admin views. Its + UID is derived from the observability connector Secret's `metadata.uid`. - A **per-tenant-cluster** datasource for each tenant cluster, scoped to its project and - instance. + instance. Its UID is derived from the tenant cluster's `status.serviceUID`. - A **per-control-plane-cluster** datasource for each control plane cluster. Tenant cluster datasources inject scope through request headers: diff --git a/platform/maintenance/observability/quickstart.mdx b/platform/maintenance/observability/quickstart.mdx new file mode 100644 index 0000000000..cf718d4ed9 --- /dev/null +++ b/platform/maintenance/observability/quickstart.mdx @@ -0,0 +1,223 @@ +--- +title: Fleet Observability quickstart +sidebar_label: Quickstart +sidebar_position: 2 +description: Deploy the default Fleet Observability pipeline end to end with Argo CD, from the metrics backend to your first query. +--- + +import InterpolatedCodeBlock from "@site/src/components/InterpolatedCodeBlock"; + +This walks through the default, out-of-the-box path. It deploys the bundled Prometheus +backend and edge collectors with Argo CD to one tenant cluster, using YAML and `kubectl` +throughout. Each step links to a reference page with the full parameter list and the +Platform UI walkthrough. See those pages for alternatives, such as bringing your own +backend or targeting a control plane cluster. + +## Prerequisites + +- The [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled, with a + connector registered on the target cluster. See + [Connect to Argo CD](../../integrations/argocd/connect-argocd.mdx). +- A vCluster Platform license that includes Fleet Observability. +- `kubectl` access to a project namespace. +- A tenant cluster to collect metrics from. + +## 1. Deploy the metrics backend + +Deploy the bundled Prometheus backend to the local cluster, the cluster where Platform +itself is installed: + +```yaml title="fleet-metrics-backend.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-metrics-backend + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet metrics backend" + destination: + cluster: + name: loft-cluster # the reserved name for Platform's local cluster; don't change this + templateRef: + name: fleet-observability-prometheus + parameters: + destinationNamespace: vcluster-platform + prometheusStorageSize: 20Gi + project: default +``` + +```bash +kubectl apply -f fleet-metrics-backend.yaml +``` + +For other backends, the Platform UI, and the full parameter list, see +[Deploy the metrics backend with Argo CD](./install-observability-gateway.mdx#deploy-the-metrics-backend-with-argo-cd). + +## 2. Create the connector and deploy the gateway + +Create the observability connector Secret. These endpoints match the backend's default +namespace from step 1: + +```yaml title="fleet-observability-connector.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: fleet-observability + namespace: vcluster-platform + labels: + loft.sh/connector-type: observability + platform.vcluster.com/fleet-observability-connector: "true" +type: Opaque +stringData: + metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp + metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local + metricsBackendCertSecretName: metrics-backend-mtls +``` + +```bash +kubectl apply -f fleet-observability-connector.yaml +``` + +Platform deploys the gateway from this Secret. Check that the gateway pod comes up: + +```bash +kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability +``` + +For connector keys, TLS options, and the Platform UI, see +[Install the observability gateway](./install-observability-gateway.mdx). + +## 3. Create a metrics-writer key + +Create a `metrics-writer` access key scoped to your tenant cluster. Set `PROJECT` and +`TENANT_CLUSTER` to your tenant cluster's project and instance name; the command prints +the generated token as `token: ` at the end. + + + +Keep the printed token for the next step. + +For control plane cluster keys, reader keys, and rotation, see +[Configure metrics access](./configure-metrics-access.mdx). + +## 4. Deploy edge collectors + +Store the token from step 3 in a Secret in the collector namespace: + +```yaml title="otel-otlp-auth.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: otel-otlp-auth + namespace: observability +type: Opaque +stringData: + authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY" +``` + +```bash +kubectl apply -f otel-otlp-auth.yaml +``` + +Then deploy the collector to your tenant cluster: + +```yaml title="fleet-edge-collectors.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-edge-collectors + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet edge collectors" + destination: + virtualCluster: + name: my-vcluster # replace with your tenant cluster name + target: vCluster # deploy into the tenant cluster + templateRef: + name: cluster-collector + parameters: + destinationNamespace: observability + otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 + project: default +``` + +```bash +kubectl apply -f fleet-edge-collectors.yaml +``` + +For control plane cluster targets, manual collector configuration, and the full +parameter list, see [Configure edge collectors](./configure-edge-collectors.mdx). + +## 5. Verify + +Check that metrics are reaching the gateway: + +```bash +GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}') +kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 +curl http://127.0.0.1:8889/metrics +``` + +If writes aren't showing up, see [Troubleshooting](./troubleshooting.mdx). + +## 6. Deploy the bundled Grafana (optional) + +Deploy Grafana for pre-provisioned dashboards: + +```yaml title="fleet-grafana.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-grafana + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet Grafana" + destination: + cluster: + name: loft-cluster # the reserved name for Platform's local cluster; don't change this + templateRef: + name: fleet-observability-grafana + parameters: + destinationNamespace: vcluster-platform + platformHost: platform.example.com + project: default +``` + +```bash +kubectl apply -f fleet-grafana.yaml +``` + +For the connector settings Grafana needs and how to sign in, see +[Deploy the bundled Grafana with Argo CD](./query-fleet-metrics.mdx#deploy-the-bundled-grafana-with-argo-cd). + +## Next steps + +- [Query fleet metrics](./query-fleet-metrics.mdx) for dashboards and PromQL access. +- [GPU observability templates](./gpu-observability-templates.mdx) to add the GPU stack and NVSentinel. +- [Troubleshooting](./troubleshooting.mdx) if something isn't working. diff --git a/platform/maintenance/observability/troubleshooting.mdx b/platform/maintenance/observability/troubleshooting.mdx index 9727308df9..9be6ee8a71 100644 --- a/platform/maintenance/observability/troubleshooting.mdx +++ b/platform/maintenance/observability/troubleshooting.mdx @@ -1,9 +1,8 @@ --- title: Troubleshooting Fleet Observability sidebar_label: Troubleshooting -sidebar_position: 8 +sidebar_position: 10 description: Diagnose common Fleet Observability gateway, access-key, collector, and query-proxy problems. -sidebar_class_name: pro --- Use this page to diagnose the Fleet Observability path from edge collectors through the