diff --git a/.github/styles/config/vocabularies/Loft/accept.txt b/.github/styles/config/vocabularies/Loft/accept.txt index 4d7d73a9d3..27ba1dba63 100644 --- a/.github/styles/config/vocabularies/Loft/accept.txt +++ b/.github/styles/config/vocabularies/Loft/accept.txt @@ -81,6 +81,9 @@ SSH SSHKey[s]? StatefulSet TSNet +TLS +mTLS +PromQL URL vCluster vcluster.yaml @@ -89,4 +92,4 @@ rpm.vcluster.com [vVirtual] [cC]luster Velero VolumeAttachment -(?i)yaml +(?i)yaml \ No newline at end of file diff --git a/.vale.ini b/.vale.ini index 187713fb4a..a4a8fa3ddf 100644 --- a/.vale.ini +++ b/.vale.ini @@ -76,6 +76,17 @@ Vale.Terms = NO BasedOnStyles = Loft, Google, Vale Vale.Terms = NO +# `metrics-backend-mtls` is a literal Kubernetes Secret name (must stay +# lowercase per RFC 1123), and "Deploy Grafana" is correct sentence case with +# a proper noun that Google.Headings doesn't recognize as one. Neither needs +# to change, so scope both rules off for this page instead of the vocab (which +# would create new false positives on the "grafana" anchors and reverse-proxy +# path used throughout the fleet monitoring pages). +[**/observability/deploy-with-argocd.mdx] +BasedOnStyles = Loft, Google, Vale +Vale.Terms = NO +Google.Headings = NO + # Auto-generated by scripts/generate-platform-cli-index.js from # vcluster/cli/vcluster_platform*.md. The regenerate-instructions comment # references that file path in plain text, and command descriptions are diff --git a/CLAUDE.md b/CLAUDE.md index e09f1a8723..eccfb9e9cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -178,7 +178,10 @@ When writing docs for new features: ### Examples format - vcluster.yaml: description + code block -- Platform: description + step-by-step (no UI screenshots - they go stale) +- Platform: description + step-by-step. Minimize UI screenshots since the UI + changes often and screenshots go stale, but they're okay where warranted + (for example, illustrating a dashboard or layout that's hard to describe in + prose). ### Checklist diff --git a/platform/maintenance/monitoring/central-hostpath-mapper.mdx b/platform/maintenance/monitoring/central-hostpath-mapper.mdx index 34ad0cf78f..321390bc48 100644 --- a/platform/maintenance/monitoring/central-hostpath-mapper.mdx +++ b/platform/maintenance/monitoring/central-hostpath-mapper.mdx @@ -1,7 +1,7 @@ --- title: Central HostPath Mapper sidebar_label: Central HostPath Mapper -sidebar_position: 6 +sidebar_position: 7 sidebar_class_name: free --- @@ -17,9 +17,9 @@ import FeatureTable from '@site/src/components/FeatureTable'; You can install a common logging solution such as Fluentd or the ELK stack on the Control Plane Cluster and use it within the tenant cluster. However, in some scenarios, installing these tools directly on the tenant cluster is preferred. -These logging solutions access pod and container logs by mounting node host paths and using specific filename formats to associate logs with the correct pods. This approach doesn’t work in a vCluster because the vCluster rewrites pod names to a vCluster-specific format when syncing them to the Control Plane Cluster. +These logging solutions access pod and container logs by mounting node host paths and using specific filename formats to associate logs with the correct pods. This approach doesn't work in a tenant cluster because vCluster rewrites pod names to a vCluster-specific format when syncing them to the Control Plane Cluster. -The HostPath Mapper collects logs from tenant clusters. It works by matching virtual pod and container names to their physical names on the host and does this through symbolic links. +The HostPath Mapper collects logs from tenant clusters. It works by matching virtual pod and container names to their physical names on the host through symbolic links. To enable HostPath Mapper for a tenant cluster, install a DaemonSet in the same namespace. This setup works well for small environments with few nodes and tenant clusters. @@ -32,7 +32,7 @@ To use this feature, install the Central HostPath Mapper on the Control Plane Cl ::: :::warning Templates -This feature is only configurable from the UI for tenant clusters created without a template. When using a custom template that has the feature enabled, the UI does not display the configuration in **Advanced Options** as described below. In this case, configure the feature using the CLI or by modifying the manifests. +This feature is only configurable from the UI for tenant clusters created without a template. When using a custom template that has the feature enabled, the UI doesn't display the configuration in **Advanced Options** as described below. In this case, configure the feature using the CLI or by modifying the manifests. ::: - From the project drop-down menu, select the project you'd like to create - the tenant cluster in. + From the project drop-down menu, click the project where you want to + create the tenant cluster. - Click the button above the list of tenant clusters + Click the button above the list of tenant clusters. - [Optional] select the cluster in which to create the tenant + [Optional] Click the cluster where you want to create the tenant cluster. @@ -60,13 +60,13 @@ This feature is only configurable from the UI for tenant clusters created withou Click the configuration tab and expand - the 'Host Path Mapper' section. + the **Host Path Mapper** section. - Toggle the slider . This will add the - 'loft.sh/hpm-enabled' annotation to the template metadata which will be - automatically synced to the tenant cluster on creation. Secondly, it - will add `enabled: true` & `central: true` under the '[controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper)' field in the vcluster.yaml. + Toggle the slider . This adds the + `loft.sh/hpm-enabled` annotation to the template metadata, which syncs + automatically to the tenant cluster on creation. It also sets + `enabled: true` and `central: true` under the [controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper) field in the vcluster.yaml. Finish configuring anything else on your tenant cluster, @@ -78,10 +78,11 @@ This feature is only configurable from the UI for tenant clusters created withou - Select the project that contains the tenant cluster you want to enable Central HostPath Mapper on from the project drop-down menu. + From the project drop-down menu, click the project that contains the + tenant cluster where you want to enable Central HostPath Mapper. - Hover over the tenant cluster you would like to enable Central HostPath Mapper on to see an button displayed. + Hover over the tenant cluster where you want to enable Central HostPath Mapper to reveal the button. Click . @@ -89,10 +90,10 @@ This feature is only configurable from the UI for tenant clusters created withou the **Host Path Mapper** section. - Toggle the slider . This will add the - 'loft.sh/hpm-enabled' annotation to the template metadata which will be - automatically synced to the tenant cluster on creation. Secondly, it - will add `enabled: true` & `central: true` under the '[controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper)' field in the vcluster.yaml. + Toggle the slider . This adds the + `loft.sh/hpm-enabled` annotation to the template metadata, which syncs + automatically to the tenant cluster on creation. It also sets + `enabled: true` and `central: true` under the [controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper) field in the vcluster.yaml. Finish configuring anything else on your tenant cluster, diff --git a/platform/maintenance/monitoring/fleet-monitoring-otel.mdx b/platform/maintenance/monitoring/fleet-monitoring-otel.mdx index 07ae77d6eb..af400826c9 100644 --- a/platform/maintenance/monitoring/fleet-monitoring-otel.mdx +++ b/platform/maintenance/monitoring/fleet-monitoring-otel.mdx @@ -1,7 +1,7 @@ --- title: "Fleet monitoring with OpenTelemetry" sidebar_label: "Fleet monitoring with OpenTelemetry" -sidebar_position: 3 +sidebar_position: 4 description: "Monitor tenant clusters across shared and private nodes using OpenTelemetry Collectors with automatic label enrichment and Prometheus remote write." --- @@ -160,7 +160,7 @@ node. The `groupbyattrs` processor splits the batch into per-pod resource scopes (by `namespace`, `pod`, `node`) so each pod is matched correctly. The `k8sattributes` processor resolves vCluster identity from Platform-managed -namespace labels (`loft.sh/project`, `loft.sh/vcluster-instance-name`, and others) and +namespace labels (`loft.sh/project`, `loft.sh/vcluster-instance-name`, and so on) and pod labels/annotations set by the vCluster syncer (`vcluster.loft.sh/namespace`, `vcluster.loft.sh/name`). @@ -170,7 +170,7 @@ which also prevents duplicate series with any existing Prometheus scrapes. ### Private nodes -The private-nodes collector runs inside each vCluster as a DaemonSet with one +The private-nodes collector runs inside each tenant cluster as a DaemonSet with one pod per node. Each pod scrapes only its local node's kubelet `/metrics`, cAdvisor `/metrics/cadvisor`, and API server `/metrics` endpoints. @@ -188,14 +188,14 @@ prometheus receiver (kubelet, cAdvisor, API server) pod → vcluster_virtual_pod) ``` -Since the collector runs inside the vCluster, it can't access host-cluster -namespace labels. Instead, the Platform injects `{{ .Values.loft.* }}` template +Since the collector runs inside the tenant cluster, it can't access Control +Plane Cluster namespace labels. Instead, the Platform injects `{{ .Values.loft.* }}` template variables at deploy time, which are set as `external_labels` on the -`prometheusremotewrite` exporter. These are static per-vCluster values applied +`prometheusremotewrite` exporter. These are static per-tenant-cluster values applied to all exported metrics. The `metric_relabel_configs` copy `namespace` to `vcluster_virtual_namespace` -and `pod` to `vcluster_virtual_pod`. Inside a private-nodes vCluster, the +and `pod` to `vcluster_virtual_pod`. Inside a private-nodes tenant cluster, the `namespace` and `pod` labels already represent virtual names, so this copy ensures dashboard compatibility with the shared-nodes collector. @@ -219,7 +219,7 @@ All metrics from both apps carry a consistent set of identity labels: `vcluster_virtual_namespace` and `vcluster_virtual_pod` are missing on some metrics. These are vCluster system pods (syncer, CoreDNS) that don't have the syncer labels and annotations because they aren't user workloads synced from -inside the vCluster. +inside the tenant cluster. ::: ## Prerequisites @@ -248,7 +248,7 @@ server: - kubelet scraping disabled in any existing kube-prometheus-stack to avoid duplicate cAdvisor series (`kubelet.enabled: false`). -- Platform namespace labels present (added automatically by the vCluster Platform). +- Platform namespace labels present (added automatically by vCluster Platform). ### Private nodes prerequisites @@ -593,7 +593,7 @@ spec: - **`serviceMonitorSelector: app: vcluster`**: Without filtering, the Target Allocator discovers all ServiceMonitors in the cluster, overwhelming collectors with memory pressure. -- **`operator.targetallocator.mtls: true`**: Each vCluster exposes its API +- **`operator.targetallocator.mtls: true`**: Each tenant cluster exposes its API server metrics over mTLS. Without this feature gate, the Target Allocator redacts TLS private keys when passing scrape configs to collectors. - **`otel/opentelemetry-collector-contrib` image**: The default image doesn't @@ -622,7 +622,7 @@ kubectl apply -f otel-collector-shared-nodes-app.yaml Navigate to the Apps tab. - Click and select the + Click and click the **OTEL Collector - Shared Nodes** app. @@ -645,7 +645,7 @@ Repeat these steps for each Control Plane Cluster. ## Deploy the private nodes collector -Deploy one private-nodes collector into each private-nodes vCluster. All +Deploy one private-nodes collector into each private-nodes tenant cluster. All vCluster identity labels are injected automatically by the Platform using `{{ .Values.loft.* }}`. @@ -889,7 +889,7 @@ spec: ensures one collector per node, scraping only the local kubelet and cAdvisor using `${env:K8S_NODE_NAME}` filtering. No Target Allocator is needed. - **`external_labels` instead of `k8sattributes` for identity**: The collector - runs inside the vCluster, so it can't access Control Plane Cluster namespace labels. + runs inside the tenant cluster, so it can't access Control Plane Cluster namespace labels. The Platform injects `{{ .Values.loft.* }}` template variables at deploy time. - **`resource_to_telemetry_conversion: false`**: Setting this to `true` causes duplicate labels that break Grafana dashboards. @@ -916,7 +916,7 @@ kubectl apply -f otel-collector-private-nodes-app.yaml Go to the Projects section using the menu on the left. - Select the project containing your private-nodes tenant cluster. + Click the project containing your private-nodes tenant cluster. Click on the tenant cluster, then navigate to the Config @@ -1215,7 +1215,8 @@ with the event rate. :::note This section assumes Grafana is already deployed. For setup instructions, see -[Aggregating Metrics — Deploy Grafana](./aggregating-metrics.mdx#deploy-grafana). +the [Deploy Grafana](./aggregating-metrics.mdx#deploy-grafana) section in +Aggregating Metrics. ::: Two Grafana dashboards are provided for visualizing metrics collected by the @@ -1234,14 +1235,12 @@ time, so all panels use straightforward PromQL queries without joins. ### vCluster projects dashboard -A platform admin overview of vCluster projects across shared and private node -tenant clusters. Use this dashboard to monitor resource consumption and API health at -the project level. +A platform admin overview of vCluster projects across Shared Nodes and Private +Nodes tenant clusters. Use this dashboard to monitor resource consumption and +API health at the project level. **Dashboard JSON file:** [Download dashboard JSON](/media/dashboards/vcluster-projects.json) - - **Template variables:** | Variable | Description | @@ -1255,8 +1254,8 @@ the project level. | Section | Panels | |---------|--------| -| Project summary | Total Projects, Total vClusters, Total Pods, Total CPU (cores), Total Memory, Max Error Rate (stat panels) | -| Resource usage by project | CPU by Project (cores), Memory by Project, Pods by Project, vClusters by Project (bar charts) | +| Project summary | Total Projects, Total tenant clusters, Total Pods, Total CPU (cores), Total Memory, Max Error Rate (stat panels) | +| Resource usage by project | CPU by Project (cores), Memory by Project, Pods by Project, Tenant clusters by Project (bar charts) | | API health by project | API Request Rate, API Error Rate (5xx), API P95 Latency (time series) | | Resource trends | CPU Usage Trend (stacked), Memory Usage Trend (stacked) (time series) | diff --git a/platform/maintenance/monitoring/metrics.mdx b/platform/maintenance/monitoring/metrics.mdx index 2a0f38a0a9..fec9a8efa2 100644 --- a/platform/maintenance/monitoring/metrics.mdx +++ b/platform/maintenance/monitoring/metrics.mdx @@ -1,7 +1,7 @@ --- title: Metrics sidebar_label: Metrics -sidebar_position: 5 +sidebar_position: 6 description: Prometheus-conformant metrics exposed by vCluster Platform, including how to configure scraping with a ServiceMonitor. --- @@ -17,7 +17,7 @@ vCluster Platform exposes several [Prometheus-conformant](https://prometheus.io/ ### Create the Prometheus servicemonitor -The metrics can be scraped with the included `ServiceMonitor` in the vCluster Platform chart, which can be deployed with Helm. In order for this to work, make sure you have installed a [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) into your cluster. You'll also need to upgrade your vCluster Platform release with the following command to create the `ServiceMonitor`: +The metrics can be scraped with the included `ServiceMonitor` in the vCluster Platform chart, which can be deployed with Helm. For this to work, make sure you've installed a [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) in your cluster. You'll also need to upgrade your vCluster Platform release with the following command to create the `ServiceMonitor`: ```bash # Make sure you change serviceMonitor.namespace if your Prometheus installation requires this. @@ -45,12 +45,12 @@ The platform pods can be identified using the following Kubernetes labels: - `component=loft` - Component label for the platform - `release=loft` - Helm release name (may vary based on your installation) -By default, vCluster Platform requires an `Authorization` HTTP header field with a Kubernetes bearer token that has access to the non-resource +By default, vCluster Platform requires an `Authorization` HTTP header field with a Kubernetes bearer token. The token must have access to the non-resource URL `/metrics` in the Kubernetes cluster where vCluster Platform is installed. -If the header is not provided, vCluster Platform denies the request. +If the header isn't provided, vCluster Platform denies the request. :::info Turn off metrics authentication To scrape metrics without authentication, turn it off using the environment variable `INSECURE_METRICS=true` in the vCluster Platform Helm chart. @@ -78,11 +78,11 @@ apiserver_audit_requests_rejected_total 0 vCluster Platform collects metrics from its internal components and merges them together into a single metrics endpoint at `/metrics`. The following component metrics are merged together: -- **API gateway metrics**: vCluster Platform exposes metrics for its internal api gateway such as the total amount of requests (`apigateway_ENDPOINT_request_total`), request latency (`apigateway_ENDPOINT_request_duration_seconds`) and request sizes (`apigateway_ENDPOINT_response_sizes`) for the different endpoints. The metrics have different labels depending on the endpoint. The following endpoints and metrics are available: +- **API gateway metrics**: vCluster Platform exposes metrics for its internal API gateway, such as the total amount of requests (`apigateway_ENDPOINT_request_total`), request latency (`apigateway_ENDPOINT_request_duration_seconds`), and request sizes (`apigateway_ENDPOINT_response_sizes`) for the different endpoints. The metrics have different labels depending on the endpoint. The following endpoints and metrics are available: - **ui**: all requests that target vCluster Platform UI assets (metrics are `apigateway_ui_request_total`, `apigateway_ui_request_duration_seconds`, `apigateway_ui_response_sizes`) - **auth**: all requests that target vCluster Platform authentication endpoints (such as login and token refresh) - - **kubernetes**: all forwarded requests that target a kubernetes cluster. The targeted backend is provided by a label and can be either `management` (the integrated vCluster Platform kubernetes api server), `cluster` (a request forwarded to a connected cluster) or `vcluster` (a virtual kubernetes cluster deployed by vCluster Platform). Further labels are provided as well such as kubernetes resource, api version, api group, and subresource. + - **kubernetes**: all forwarded requests that target a Kubernetes cluster. The targeted backend is provided by a label with one of three values. `management` targets the integrated vCluster Platform Kubernetes API server. `cluster` targets a request forwarded to a connected cluster. `vcluster` targets a tenant cluster deployed by vCluster Platform. Further labels include Kubernetes resource, API version, API group, and subresource. - **oidc**: all requests that target the internal OIDC server -- **Integrated Kubernetes API server metrics**: all internal kubernetes api server metrics are exposed. An incomplete list can be found in the [kubernetes repository](https://github.com/kubernetes/apiserver/blob/master/pkg/endpoints/metrics/metrics.go) -- **Integrated Controller manager metrics**: all vCluster Platform management cluster controller metrics are exposed. -- **Go runtime metrics**: some general metrics about the golang runtime and resource usage are exposed. +- **Integrated Kubernetes API server metrics**: all internal Kubernetes API server metrics are exposed. An incomplete list can be found in the [Kubernetes repository](https://github.com/kubernetes/apiserver/blob/master/pkg/endpoints/metrics/metrics.go). +- **Integrated controller manager metrics**: all vCluster Platform management cluster controller metrics are exposed. +- **Go runtime metrics**: some general metrics about the Go runtime and resource usage are exposed. diff --git a/platform/maintenance/monitoring/overview.mdx b/platform/maintenance/monitoring/overview.mdx index 0fc4d3a9e0..c3d82f25b7 100644 --- a/platform/maintenance/monitoring/overview.mdx +++ b/platform/maintenance/monitoring/overview.mdx @@ -5,7 +5,7 @@ sidebar_position: 1 description: Monitor vCluster Platform health, workload metrics across tenant clusters, and logs with Prometheus, OpenTelemetry, and Grafana. --- -vCluster Platform provides full-stack observability across your entire multi-tenant fleet. For a single cluster, install a complete Prometheus and Grafana stack in one click. For fleet-wide visibility, workload metrics from every connected cluster can flow into a central Prometheus instance, automatically labeled by project, space, and tenant cluster. +vCluster Platform provides full operational visibility across your entire fleet of tenant clusters. For a single cluster, install a complete Prometheus and Grafana stack in one click. For fleet-wide visibility, choose between Platform-governed Fleet Observability and direct OpenTelemetry aggregation. Operators can also monitor the Platform's own health using built-in Prometheus metrics. For centralized log collection across all tenant clusters, the Central HostPath Mapper removes the need for per-cluster agents. @@ -16,8 +16,8 @@ vCluster Platform provides two approaches for monitoring tenant cluster workload | If you want to | Use | |---|---| | Monitor a single cluster with a quick, one-click setup | [Basic cluster monitoring](#basic-cluster-monitoring) | -| Aggregate metrics across multiple tenant clusters | [Fleet monitoring](#fleet-monitoring-multi-cluster) | -| Filter or label metrics by project, space, or tenant cluster | [Fleet monitoring](#fleet-monitoring-multi-cluster) | +| Aggregate metrics across multiple tenant clusters | [Fleet metrics approaches](#fleet-metrics-approaches) | +| Filter or label metrics by project, space, or tenant cluster | [Fleet metrics approaches](#fleet-metrics-approaches) | ### Basic cluster monitoring @@ -27,18 +27,19 @@ For a single connected cluster, install the `kube-prometheus-stack` app from the 2. Navigate to the **Apps** tab. 3. Click the **kube-prometheus-stack** recommended app and click **Install**. -After installation, you have a complete monitoring setup with Prometheus scraping cluster metrics and Grafana for visualization. If your tenant clusters run in Shared Nodes mode, see [Prometheus node metrics on shared nodes](./prometheus-shared-nodes) to configure kubelet scraping. +After installation, you have a complete monitoring setup with Prometheus scraping cluster metrics and Grafana for visualization. If your tenant clusters run in Shared Nodes mode, see [Prometheus node metrics on shared nodes](./prometheus-shared-nodes.mdx) to configure kubelet scraping. :::warning -Do not use `kube-prometheus-stack` if you want to aggregate metrics across multiple tenant clusters. Use the OpenTelemetry fleet monitoring approach below instead. +Don't use `kube-prometheus-stack` if you want to aggregate metrics across multiple tenant clusters. Choose one of the fleet metrics approaches below instead. ::: -### Fleet monitoring (multi-cluster) +### Fleet metrics approaches -For aggregating workload metrics across multiple tenant clusters, use the OpenTelemetry Collector with Prometheus. This approach deploys shared OpenTelemetry DaemonSets on each connected cluster. They push metrics to a central Prometheus instance, labeled by project, tenant cluster, and space. +To aggregate workload metrics across multiple tenant clusters, choose an approach based on the Platform integration you need. Fleet Observability provides Platform-authorized ingestion and querying with trusted labels. The OpenTelemetry guides focus on direct collector-to-backend pipelines. -- [Aggregating metrics with OpenTelemetry](./aggregating-metrics) — step-by-step setup with Prometheus, OpenTelemetry Collector, and Grafana -- [Fleet monitoring with OpenTelemetry](./fleet-monitoring-otel) — advanced configuration including remote_write for both Shared Nodes and Private Nodes tenancy models +- [Fleet Observability](../observability/overview.mdx) — Platform-governed path with the observability gateway, scoped metrics access, trusted labels, querying, and troubleshooting +- [Aggregating metrics with OpenTelemetry](./aggregating-metrics.mdx) — simpler direct pipeline with Prometheus, OpenTelemetry Collector, and Grafana +- [Fleet monitoring with OpenTelemetry](./fleet-monitoring-otel.mdx) — advanced direct pipeline with collector examples for Shared Nodes and Private Nodes tenancy models ## Platform health metrics @@ -48,8 +49,8 @@ Use a [Prometheus `ServiceMonitor`](./metrics.mdx#create-the-prometheus-servicem ## Log collection -To collect logs from workloads running inside tenant clusters, use the Central HostPath Mapper. It installs a single DaemonSet on the Control Plane Cluster. The DaemonSet handles log path remapping for all tenant clusters and removes the need for per-cluster logging agents. +To collect logs from workloads running inside tenant clusters, use the Central HostPath Mapper. It installs a single DaemonSet on the Control Plane Cluster. The DaemonSet handles log path remapping for all tenant clusters and removes the need for per-cluster logging agents. To understand the log types vCluster produces, see [vCluster Logging](/docs/vcluster/manage/logging). -- [Central HostPath Mapper](./central-hostpath-mapper) +- [Central HostPath Mapper](./central-hostpath-mapper.mdx) For platform audit logging and platform process log configuration, see [Logging](../logging/overview.mdx). diff --git a/platform/maintenance/monitoring/prometheus-shared-nodes.mdx b/platform/maintenance/monitoring/prometheus-shared-nodes.mdx index d2f3fa8626..5718184e87 100644 --- a/platform/maintenance/monitoring/prometheus-shared-nodes.mdx +++ b/platform/maintenance/monitoring/prometheus-shared-nodes.mdx @@ -1,24 +1,24 @@ --- title: "Prometheus node metrics on shared nodes" sidebar_label: "Prometheus on shared nodes" -sidebar_position: 4 +sidebar_position: 5 description: "Configure Prometheus to scrape kubelet and cAdvisor node metrics from tenant clusters running in Shared Nodes mode." --- In the Shared Nodes tenancy model, vCluster virtualizes node IP addresses by default as part of its kubelet proxying feature. Prometheus requires specific configuration to scrape kubelet and cAdvisor metrics in this environment. -This does not affect Private Nodes tenant clusters, which have dedicated real nodes with directly reachable IPs. +This doesn't affect Private Nodes tenant clusters, which have dedicated real nodes with directly reachable IPs. ## How it works -vCluster enables kubelet proxying by default (`networking.advanced.proxyKubelets.byIP: true` and `byHostname: true` in the Helm values). This feature lets vCluster intercept kubelet traffic and rewrite pod names and namespaces from host-cluster values to their virtual equivalents. +vCluster enables kubelet proxying by default (`networking.advanced.proxyKubelets.byIP: true` and `byHostname: true` in the Helm values). This feature lets vCluster intercept kubelet traffic and rewrite pod names and namespaces from control plane cluster values to their virtual equivalents. As part of this, vCluster assigns each synced node a virtual identity: - With `byIP: true`, vCluster replaces the node's IP address with the ClusterIP of a per-node Kubernetes Service in the vCluster namespace on the control plane cluster. - With `byHostname: true`, vCluster replaces the node hostname with a synthetic name like `.nodes.vcluster.com`. -Prometheus discovers scrape targets by reading node objects from the tenant cluster's Kubernetes API. It then attempts to connect to those virtual addresses. If the per-node service ClusterIP is not routable from where Prometheus runs, the scrape times out. This most commonly happens when Prometheus is outside the tenant cluster or on a separate connected cluster, +Prometheus discovers scrape targets by reading node objects from the tenant cluster's Kubernetes API. It then attempts to connect to those virtual addresses. If the per-node service ClusterIP isn't routable from where Prometheus runs, the scrape times out. This most commonly happens when Prometheus is outside the tenant cluster or on a separate connected cluster. ## Approaches @@ -30,7 +30,7 @@ The right approach depends on where Prometheus runs: ### Option 1: OTel fleet monitoring (recommended) -Deploy an OpenTelemetry Collector on the Control Plane Cluster. The collector scrapes cAdvisor and kubelet metrics using the control plane cluster's real node IPs, then pushes enriched metrics to a central Prometheus using `remote_write`. Because the collector operates at the control plane cluster level, it never reads tenant cluster node objects or depends on virtual node IPs. +Deploy an OpenTelemetry Collector on the control plane cluster. The collector scrapes cAdvisor and kubelet metrics using the control plane cluster's real node IPs, then pushes enriched metrics to a central Prometheus using `remote_write`. Because the collector operates at the control plane cluster level, it never reads tenant cluster node objects or depends on virtual node IPs. This approach also adds per-tenant identity labels such as `vcluster_name` and `vcluster_project` to every metric, making it suitable for multi-tenant observability. @@ -40,7 +40,7 @@ See [Fleet monitoring with OpenTelemetry](./fleet-monitoring-otel.mdx) for a com ### Option 2: API server proxy scraping (Prometheus inside the tenant cluster) -Configure Prometheus to route kubelet and cAdvisor scrapes through the Kubernetes API server proxy instead of connecting to node IPs directly. vCluster intercepts requests that go through the API server, forwards them to the real kubelet on the control plane cluster, and returns them with pod names and namespaces already rewritten to their virtual equivalents, so no additional relabeling is needed. +Configure Prometheus to route kubelet and cAdvisor scrapes through the Kubernetes API server proxy instead of connecting to node IPs directly. vCluster intercepts requests that go through the API server and forwards them to the real kubelet on the control plane cluster. It returns them with pod names and namespaces already rewritten to their virtual equivalents, so no additional relabeling is needed. Configure the cAdvisor and kubelet jobs to target the API server and set the metrics path to the node proxy endpoint: @@ -98,6 +98,6 @@ networking: :::info With kubelet proxying disabled: -- **vCluster does not rewrite metrics.** Pod and namespace labels reflect the control plane cluster's internal naming, not the tenant cluster's virtual names. On a shared node, metrics from different tenant clusters will have overlapping pod/namespace label values. +- **vCluster doesn't rewrite metrics.** Pod and namespace labels reflect the control plane cluster's internal naming, not the tenant cluster's virtual names. On a shared node, metrics from different tenant clusters have overlapping pod/namespace label values. - **Real node IPs are visible** in the tenant cluster's node objects, which reveals control plane network topology to tenant cluster users. ::: diff --git a/platform/maintenance/observability/_category_.json b/platform/maintenance/observability/_category_.json new file mode 100644 index 0000000000..82c186f295 --- /dev/null +++ b/platform/maintenance/observability/_category_.json @@ -0,0 +1,6 @@ +{ + "label": "Fleet Observability", + "position": 3.1, + "collapsible": true, + "collapsed": true +} diff --git a/platform/maintenance/observability/configure-edge-collectors.mdx b/platform/maintenance/observability/configure-edge-collectors.mdx new file mode 100644 index 0000000000..e3a82e5c24 --- /dev/null +++ b/platform/maintenance/observability/configure-edge-collectors.mdx @@ -0,0 +1,169 @@ +--- +title: Configure edge collectors +sidebar_label: Configure collectors +sidebar_position: 5 +description: Configure OpenTelemetry Collectors to send fleet metrics to the vCluster observability Write Gateway. +--- + +Edge collectors are OpenTelemetry Collectors that scrape or receive metrics near the +workloads, then send OTLP to the Write Gateway. Each collector request must include a bearer +token and scope headers that match a `metrics-writer` access key. + +:::tip Deploy collectors with Argo CD +If you use the Argo CD integration, you can deploy the collectors from a bundled template +instead of configuring them by hand. See +[Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). This manual path is the +only option for vCluster Standalone tenant clusters, which can't use the Argo CD +integration. +::: + +## Gateway endpoints + +Inside the Platform cluster, the fleet gateway Service is: + +```text +gateway-fleet-observability.vcluster-platform.svc.cluster.local +``` + +This endpoint assumes the example connector ID `fleet-observability`. For another ID, +replace the Service name with `gateway-`. + +Use this OTLP endpoint: + +| Protocol | Endpoint | +|---|---| +| OTLP HTTP | `https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318` | + +The examples on this page use OTLP HTTP on port `4318`. The gateway also accepts OTLP gRPC +on port `4317`, but the bundled OpenTelemetry Collector sends over HTTP, so use `4318`. + +The gateway always uses TLS. The OpenTelemetry Collector `otlp_http` exporter accepts the +base URL shown above. Direct OTLP HTTP clients send metrics to the `/v1/metrics` path on +that host. Mount the CA that issued the gateway serving certificate into the collector and +set the exporter `tls.ca_file`. The default Platform-managed certificate stores its trust +anchor in `ca.crt` in the `gateway--tls` Secret. + +Collectors outside the Platform cluster can't resolve the in-cluster Service DNS name. +Expose the gateway with a LoadBalancer, then use the LoadBalancer address in the exporter +endpoint instead of the Service name, for example +`https://:4318`. See +[Expose the gateway with a LoadBalancer](./install-observability-gateway.mdx#expose-the-gateway-with-a-loadbalancer) +to enable it and retrieve the address. The exporter `endpoint` and TLS settings in the +examples below are otherwise unchanged; only the host differs. + +## Store the writer key + +Create a Secret for the collector, using a `metrics-writer` access key from +[Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster) +or [Create a writer key for a control plane cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-control-plane-cluster): + +```yaml title="metrics-writer-token.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: metrics-writer-token + namespace: observability +type: Opaque +stringData: + token: REPLACE_WITH_METRICS_WRITER_KEY +``` + +Mount or project this Secret into your collector pod and expose it as an environment +variable or file. + +## Tenant cluster collector + +Use tenant scope headers when a collector writes metrics for a specific tenant cluster. + +```yaml title="otel-collector-tenant-exporter.yaml" +exporters: + otlp_http/fleet: + endpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 + tls: + ca_file: /etc/otel-gateway-tls/ca.crt + headers: + Authorization: Bearer ${env:METRICS_WRITER_TOKEN} + X-Vcluster-Platform-Project: my-project + X-Vcluster-Platform-Instance: my-vcluster + +service: + pipelines: + metrics: + receivers: [prometheus] + processors: [batch] + exporters: [otlp_http/fleet] +``` + +The `metrics-writer` key must include a matching `spec.scope.virtualClusters` entry. + +## Control plane cluster collector + +Use cluster scope headers when a collector writes metrics for a control plane cluster. + +```yaml title="otel-collector-cluster-exporter.yaml" +exporters: + otlp_http/fleet: + endpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 + tls: + ca_file: /etc/otel-gateway-tls/ca.crt + headers: + Authorization: Bearer ${env:METRICS_WRITER_TOKEN} + X-Vcluster-Platform-Cluster: local-cluster + +service: + pipelines: + metrics: + receivers: [prometheus] + processors: [batch] + exporters: [otlp_http/fleet] +``` + +The `metrics-writer` key must include a matching `spec.scope.clusters` entry. + +## Scrape custom targets + +Use the collector's `prometheus` receiver to scrape exporters such as kubelet, +cAdvisor, DCGM exporter, or application endpoints. The gateway doesn't require a +specific scrape receiver. It only requires OTLP metrics with valid authentication and +scope. + +Example DCGM scrape job: + +```yaml title="dcgm-prometheus-receiver.yaml" +receivers: + prometheus: + config: + scrape_configs: + - job_name: dcgm-exporter + kubernetes_sd_configs: + - role: pod + relabel_configs: + - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] + action: keep + regex: dcgm-exporter + - source_labels: [__meta_kubernetes_pod_ip] + target_label: __address__ + replacement: ${1}:9400 +``` + +## Protect platform labels + +Don't set labels with the `vcluster_platform_*` prefix in edge collectors. The Write +Gateway strips this namespace from incoming resource, scope, and datapoint attributes +before stamping authenticated labels. + +Use your own label prefix for application or infrastructure dimensions. + +## Verify writes + +Check the Write Gateway metrics endpoint: + +```bash +CONNECTOR_ID=fleet-observability +GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}') +kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 +curl http://127.0.0.1:8889/metrics +``` + +Then query through the Query Proxy as described in +[Query fleet metrics](./query-fleet-metrics.mdx). diff --git a/platform/maintenance/observability/configure-metrics-access.mdx b/platform/maintenance/observability/configure-metrics-access.mdx new file mode 100644 index 0000000000..4e07bdcf29 --- /dev/null +++ b/platform/maintenance/observability/configure-metrics-access.mdx @@ -0,0 +1,219 @@ +--- +title: Configure metrics access +sidebar_label: Configure access +sidebar_position: 4 +description: Create metrics-writer and metrics-reader access keys for the Fleet Observability gateway. +--- + +import InterpolatedCodeBlock from "@site/src/components/InterpolatedCodeBlock"; + +Fleet Observability uses scoped access keys to separate write access from read access. +The gateway never trusts the metrics payload to define scope. Each request is +authenticated with Platform and authorized against the requested cluster or tenant +cluster. + +## Access roles + +| Role | Used by | Allows | +|---|---|---| +| `metrics-writer` | Edge collectors | Write metrics for the access key's scoped cluster or tenant cluster. | +| `metrics-reader` | Grafana, PromQL clients, automation | Read metrics for the access key's scoped cluster or tenant cluster. | + +The roles are intentionally separate. A `metrics-writer` key can't read metrics, and +a `metrics-reader` key can't write metrics. A key can carry both roles when an +integration needs both paths. + +## Scope choices + +Scope metrics keys as tightly as possible: + +- Use `spec.scope.clusters` for control plane cluster metrics. +- Use `spec.scope.virtualClusters` for tenant cluster metrics. +- Use `*` only for trusted fleet-wide automation. + +The Platform authorization check still applies the permissions of the user or subject +that owns the key. Scope narrows permission. It doesn't expand it. + +:::note Examples are blueprints, not prescriptions +The snippets below are one example of how to create access keys. In particular, the line +that generates the `KEY` value shows just one way to produce a random secret. Use whatever +key generation, secret storage, and provisioning workflow fits your environment. Treat these +commands as general blueprints rather than the required procedure. +::: + +## Create a writer key for a tenant cluster + +Create one `metrics-writer` key for each tenant cluster collector, or for each +automation unit that writes on behalf of one tenant cluster. + +Set `PROJECT` and `TENANT_CLUSTER` to the target tenant cluster's project and instance name. The +`KEY` value is a freshly generated secret, and the command prints it as `token: ` +at the end. + + + +Use the printed token as the collector bearer token. Store it in a Kubernetes Secret in +the cluster where the collector runs. + +## Create a writer key for a control plane cluster + +Use cluster scope when a collector writes metrics for a control plane cluster. + +Set `CLUSTER` to the control plane cluster name. + + + +## Create a reader key + +Use a `metrics-reader` key for automation that queries metrics directly through the +Query Proxy. + +Set `PROJECT` and `TENANT_CLUSTER` to the target tenant cluster's project and instance name. + + + +Grafana datasources generated by Platform can also use the user's Platform session +cookie. In that case the Query Proxy authorizes reads as the user instead of a static +reader key. For details, see +[Authorization with your access key](./query-fleet-metrics.mdx#authorization-with-your-access-key) +in Query fleet metrics. + +## Request scope headers + +Writers and readers declare scope with headers. + +For a control plane cluster: + +```text +X-Vcluster-Platform-Cluster: local-cluster +``` + +For a tenant cluster: + +```text +X-Vcluster-Platform-Project: my-project +X-Vcluster-Platform-Instance: my-vcluster +``` + +The Write Gateway can derive tenant scope from a writer key only when the key contains +exactly one concrete tenant cluster scope. Explicit headers are easier to audit and are +recommended for collector configurations. + +:::note Argo CD collectors set these headers for you +When you deploy collectors with the `cluster-collector` template as described in +[Deploy collectors to each cluster](./deploy-with-argocd.mdx#deploy-collectors-to-each-cluster), the template sets +these scope headers automatically from the deployment destination. You only set them by hand +when you configure a collector manually. +::: + +## Rotation + +:::important Restart collectors after a key or config change +Edge collectors don't roll their pods automatically when their token or config changes. +After you rotate a `metrics-writer` key or change collector config, you must restart the +collector so it loads the new value. Until you restart it, the collector keeps using the +old token, and its writes fail once you disable the old key. +::: + +To rotate a metrics key: + +1. Create a new key with the same role and scope. +2. Update the collector or query client Secret with the new key value. +3. Restart the collector so it picks up the new token. Reload query clients that hold the + reader token. +4. Delete or disable the old key. + +Restart the collector workloads in the collector namespace. For collectors deployed with +the `cluster-collector` template, restart the OpenTelemetry Collector Deployment and +DaemonSet: + +```bash +kubectl rollout restart daemonset,deployment -n observability \ + -l app.kubernetes.io/name=opentelemetry-collector +``` + +For a manually deployed collector, restart its own Deployment or DaemonSet the same way. + +Authorization decisions are cached briefly. After you delete, disable, or narrow a key, +allow up to the cache TTL for all gateway and query-proxy decisions to expire. + +## Next step + +Configure collectors to send metrics with these keys in +[Configure edge collectors](./configure-edge-collectors.mdx). diff --git a/platform/maintenance/observability/deploy-with-argocd.mdx b/platform/maintenance/observability/deploy-with-argocd.mdx new file mode 100644 index 0000000000..fc54a6580e --- /dev/null +++ b/platform/maintenance/observability/deploy-with-argocd.mdx @@ -0,0 +1,358 @@ +--- +title: Deploy Fleet Observability with Argo CD +sidebar_label: Argo CD templates +sidebar_position: 3 +description: Use the default Argo CD Application Templates that vCluster Platform ships to deploy the fleet metrics backend, Grafana, and edge collectors through GitOps. +--- + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import Flow, { Step } from "@site/src/components/Flow"; +import Button from "@site/src/components/Button"; +import Label from "@site/src/components/Label"; + +When the [Argo CD integration](../../integrations/argocd/overview.mdx) is enabled, +vCluster Platform ships a set of default `ArgoCDApplicationTemplate` resources for Fleet +Observability. You reference these templates from `ArgoCDApplication` resources to deploy the +metrics backend, Grafana, and edge collectors through GitOps, instead of installing and +wiring them up by hand. + +The templates are an alternative to the manual setup in +[Install the observability gateway](./install-observability-gateway.mdx) and +[Configure edge collectors](./configure-edge-collectors.mdx). The observability gateway +itself is still deployed from an observability connector Secret, as described on the +install page. + +:::info Prerequisites +The bundled templates are installed only when the Argo CD integration is enabled and your +license includes Fleet Observability. You also need an Argo CD connector registered on the +target cluster. See [Connect to Argo CD](../../integrations/argocd/connect-argocd.mdx). +::: + +## Bundled templates + +Platform installs the following cluster-scoped `ArgoCDApplicationTemplate` resources. The +Platform UI lists them by display name. In an `ArgoCDApplication`, you reference them by +their resource name through `spec.templateRef.name`. + +| Template (Platform UI) | `templateRef.name` | Deploys | Default namespace | +|---|---|---|---| +| Fleet Observability: Prometheus | `fleet-observability-prometheus` | Prometheus configured as an OTLP metrics backend | `vcluster-platform` | +| Fleet Observability: Grafana | `fleet-observability-grafana` | Grafana with the fleet, cluster, and GPU dashboards pre-provisioned | `vcluster-platform` | +| Fleet Observability: Cluster Collector | `cluster-collector` | OpenTelemetry Collectors | `observability` | + +The integration also ships templates for cert-manager and GPU monitoring, which are +outside the scope of this guide. + +Because the templates are cluster-scoped, you don't copy them into each project. You +reference them by name and pass per-deployment values through `spec.parameters`. For the +general template and application model, see +[Deploy applications](../../integrations/argocd/deploy-applications.mdx). + +## How the pieces fit together + +The full setup assembles four pieces in order: + +1. `fleet-observability-prometheus` deploys the metrics backend on the control plane + cluster. +2. The observability connector Secret deploys the Write Gateway and Query Proxy and points + them at that backend. +3. `cluster-collector` deploys collectors that push OTLP metrics to the Write Gateway. +4. `fleet-observability-grafana` deploys Grafana with datasources that read through the + Query Proxy. + +## Deploy to the control plane cluster + +The metrics backend and Grafana run on the control plane cluster, alongside the gateway. +Deploy them once, before creating the connector. + +### Deploy the metrics backend + +Deploy an `ArgoCDApplication` that references `fleet-observability-prometheus`. This +deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform +namespace on your control plane cluster. + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="fleet-metrics-backend.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-metrics-backend + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet metrics backend" + destination: + cluster: + name: loft-cluster # replace with your control plane cluster name + templateRef: + name: fleet-observability-prometheus + parameters: + destinationNamespace: vcluster-platform + prometheusStorageSize: 20Gi + project: default +``` + + + + + + + Click in the left sidebar under the infrastructure section, then click the name of the control plane cluster to open its detail page. + + + Select the tab and click . + + + In the section, enter a such as `Fleet metrics backend`. + + + From the dropdown, select the template. + + + Fill the . Keep as `vcluster-platform`, and adjust the storage size or chart version if needed. + + + Click . + + + + + + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `vcluster-platform` | Namespace for the Prometheus backend. | +| `project` | `default` | Argo CD project that owns the application. | +| `repoURL` | Prometheus community Helm repo | Helm repository for the Prometheus chart. | +| `version` | `29.14.0` | Prometheus Helm chart version. | +| `prometheusStorageSize` | `20Gi` | Persistent volume size for Prometheus storage. | +| `metricsBackendTLSSecret` | _(empty)_ | Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named `metrics-backend-mtls`. | + +The observability connector you create on the +[install page](./install-observability-gateway.mdx) must use these endpoints when the +template is deployed in its default namespace: + +```yaml +stringData: + metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp + metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local + metricsBackendCertSecretName: metrics-backend-mtls +``` + +The `/api/v1/otlp` path is required for OTLP ingestion. If you change +`destinationNamespace`, replace `vcluster-platform` in both endpoints. If you supply +`metricsBackendTLSSecret`, use that Secret name instead of `metrics-backend-mtls`. + +### Deploy Grafana + +Grafana is optional. Deploy it when you want the pre-provisioned fleet dashboards and +Platform-authenticated access. Reference `fleet-observability-grafana`. + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="fleet-grafana.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-grafana + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet Grafana" + destination: + cluster: + name: loft-cluster # replace with your control plane cluster name + templateRef: + name: fleet-observability-grafana + parameters: + destinationNamespace: vcluster-platform + platformHost: platform.example.com + project: default +``` + + + + + + + Click in the left sidebar under the infrastructure section, then click the name of the control plane cluster to open its detail page. + + + Select the tab and click . + + + In the section, enter a such as `Fleet Grafana`. + + + From the dropdown, select the template. + + + Fill the . Set to your Platform host. + + + Click . + + + + + + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `vcluster-platform` | Namespace for Grafana. | +| `project` | `default` | Argo CD project that owns the application. | +| `platformHost` | Platform host | Platform host used to authenticate Grafana users. | +| `grafanaAdminGroup` | _(empty)_ | Group mapped to the Grafana admin role. | +| `grafanaCertSecret` | _(empty)_ | TLS Secret for Grafana serving. When empty, Platform self-signs a Secret named `grafana-server-tls`. | + +Configure the fleet observability connector so Platform can proxy and embed the bundled +Grafana dashboards. These values match the template defaults: + +```yaml +stringData: + grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local + grafanaCertSecretName: grafana-server-tls + fleetObservabilityDashboardUid: vcluster-fleet-observability +``` + +If you change `destinationNamespace`, update `grafanaUrl`. If you supply +`grafanaCertSecret`, use that Secret name instead of `grafana-server-tls`. + +## Deploy collectors to each cluster + +Collectors run on the clusters to observe. Deploy one to each cluster once the gateway is +running and a `metrics-writer` key is in place. + +The `cluster-collector` template deploys OpenTelemetry Collectors that push OTLP metrics +to the Write Gateway. Deploy it to each tenant cluster you want to collect from. + +:::note vCluster Standalone tenant clusters +The out-of-the-box collector path relies on the Argo CD integration, which vCluster +Standalone tenant clusters can't use. There is no bundled collector for them. Deploy a +collector manually instead, as described in +[Configure edge collectors](./configure-edge-collectors.mdx). +::: + +Each collector authenticates to the gateway with a `metrics-writer` bearer token. Create a +Secret named `otel-otlp-auth` in the collector namespace before you deploy, or pass the +token through the `bearerToken` parameter. + +```yaml title="otel-otlp-auth.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: otel-otlp-auth + namespace: observability +type: Opaque +stringData: + authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY" +``` + +Create the writer key first, as described in +[Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster) +or [Create a writer key for a control plane cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-control-plane-cluster), +then deploy the collectors: + + + + +Create the `ArgoCDApplication` object in your project namespace: + +```yaml title="fleet-edge-collectors.yaml" +apiVersion: management.loft.sh/v1 +kind: ArgoCDApplication +metadata: + name: fleet-edge-collectors + namespace: p-my-project # replace with your project namespace +spec: + displayName: "Fleet edge collectors" + destination: + virtualCluster: + name: my-vcluster # replace with your tenant cluster name + target: vCluster # deploy into the tenant cluster + templateRef: + name: cluster-collector + parameters: + destinationNamespace: observability + otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 + project: default +``` + + + + + + + Select your project from the projects dropdown at the top of the left navigation bar. + + + Navigate to and open the tenant cluster you want to collect from. + + + Select the tab and click . + + + In the section, enter a such as `Fleet edge collectors`. + + + Select to deploy into the tenant cluster itself. + + + From the dropdown, select the template. + + + Fill the . Set to `observability` and to the Write Gateway endpoint. Use the in-cluster Service endpoint for collectors in the Platform cluster, or the gateway LoadBalancer address for collectors in a tenant cluster elsewhere. + + + Click to add the Argo CD Application to the tenant cluster. + + + + + + +| Parameter | Default | Description | +|---|---|---| +| `destinationNamespace` | `observability` | Namespace for the collectors. | +| `project` | `default` | Argo CD project that owns the application. | +| `otlpEndpoint` | Fleet Write Gateway Service | OTLP HTTP endpoint the collectors push to. Use the in-cluster Service endpoint (`:4318`) for collectors in the Platform cluster, or the gateway LoadBalancer address (`https://:4318`) for collectors elsewhere. See [Expose the gateway with a LoadBalancer](./install-observability-gateway.mdx#expose-the-gateway-with-a-loadbalancer). | +| `bearerToken` | _(empty)_ | `metrics-writer` bearer token. When empty, the collectors read the `otel-otlp-auth` Secret. | +| `otlpCaData` | _(empty)_ | PEM CA data used to verify the gateway serving certificate. | +| `otlpInsecureSkipVerify` | `false` | Skip gateway TLS verification. Use only for test gateways. | + +The collectors stamp the request scope from the destination, so target a tenant cluster +for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set +`destination.cluster` instead of `destination.virtualCluster` to target a control plane +cluster. Don't set `vcluster_platform_*` labels in collector configuration; the Write +Gateway owns that namespace. See [Configure edge collectors](./configure-edge-collectors.mdx) for the +scope headers and label rules. + +## Verify + +Once the applications sync, verify the backend and collectors with the checks in +[Install the observability gateway](./install-observability-gateway.mdx#verify-the-gateway) +and [Configure edge collectors](./configure-edge-collectors.mdx#verify-writes). diff --git a/platform/maintenance/observability/install-observability-gateway.mdx b/platform/maintenance/observability/install-observability-gateway.mdx new file mode 100644 index 0000000000..7254a86aef --- /dev/null +++ b/platform/maintenance/observability/install-observability-gateway.mdx @@ -0,0 +1,284 @@ +--- +title: Install the observability gateway +sidebar_label: Install gateway +sidebar_position: 2 +description: Create the Fleet Observability connector that deploys the Write Gateway and Query Proxy. +--- + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import Flow, { Step } from "@site/src/components/Flow"; +import Button from "@site/src/components/Button"; +import Label from "@site/src/components/Label"; + +The observability gateway runs in the vCluster Platform namespace. It contains two +containers: + +- `otel-gateway`, the Write Gateway that receives OTLP. +- `query-proxy`, the read proxy that serves a Prometheus-compatible query API for PromQL reads. + +Platform deploys the gateway from an observability connector Secret. The connector +stores the metrics backend endpoints and optional TLS settings. + +## Prerequisites + +Before you create the connector, prepare: + +- A metrics backend that ingests OTLP metrics and serves the Prometheus HTTP API for + queries. Deploy the bundled Prometheus out of the box with the Argo CD integration, or + bring your own. +- Network access from the Platform namespace to both backend endpoints. +- A vCluster Platform license that includes Fleet Observability. +- Optional custom gateway serving certificate or backend mTLS Secrets. + +:::note +The observability gateway doesn't install or size the metrics backend for you. Bring your +own backend, or deploy the bundled Prometheus backend out of the box with the Argo CD +integration. See [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). +::: + +## Create the connector Secret + +Create the Fleet Observability connector in the Platform namespace. The connector ID can be +any valid Secret name. Platform derives the gateway resource name as `gateway-`. +The examples use `fleet-observability`, which produces `gateway-fleet-observability`. + +The `loft.sh/connector-type: observability` label is required. Platform only recognizes and +reconciles a Secret as an observability connector when this label is present. + +Mark the connector as the fleet default with the +`platform.vcluster.com/fleet-observability-connector` label set to `"true"`. Only one +connector can be the fleet default. + + + + +```yaml title="fleet-observability-connector.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: fleet-observability + namespace: vcluster-platform + labels: + loft.sh/connector-type: observability + platform.vcluster.com/fleet-observability-connector: "true" +type: Opaque +stringData: + metricsBackendOtlpEndpoint: https://metrics.example.com/otlp + metricsBackendPromQLEndpoint: https://metrics.example.com +``` + +Apply the Secret: + +```bash +kubectl apply -f fleet-observability-connector.yaml +``` + + + + + + + Click and select the tab. + + + Click . + + + In the field, enter a human-readable name for the connector. + The is auto-generated from the display name. + The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default. + + + In the field, enter the OTLP endpoint that receives metrics from the Write Gateway. + + + In the field, enter the Prometheus HTTP API endpoint queried by the Query Proxy. + + + Optionally configure TLS for the metrics backend and the gateway. See [Connector keys](#connector-keys) for the available settings. + + + Click . + + + + + + +Platform reconciles the following resources: + +- `Deployment/gateway-fleet-observability` +- `Service/gateway-fleet-observability` +- Grafana datasource ConfigMaps for fleet, control plane cluster, and tenant cluster + views when Grafana datasource reconciliation is enabled. + +## Connector keys + +| Key | Required | Description | +|---|---:|---| +| `metricsBackendOtlpEndpoint` | Yes | OTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends `/v1/metrics` (for example, `https://prom-prometheus-server.vcluster-platform/api/v1/otlp` receives writes at `/api/v1/otlp/v1/metrics`). | +| `metricsBackendPromQLEndpoint` | Yes | Prometheus HTTP API endpoint queried by the Query Proxy. | +| `metricsBackendInsecureSkipVerify` | No | Set to `"true"` only for test backends with unverifiable TLS. | +| `metricsBackendServerName` | No | TLS server name override for the metrics backend. | +| `observabilityGatewayCertSecretName` | No | Custom Secret with `tls.crt` and `tls.key` for gateway serving TLS. Include `ca.crt` when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted. | +| `metricsBackendCertSecretName` | No | Secret with `ca.crt`, `tls.crt`, and `tls.key` for mTLS to the backend. The gateway trusts the backend with `ca.crt` and presents the certificate and key as its client identity. | +| `gatewayResources` | No | YAML `resources` block for the Write Gateway container. | +| `queryProxyResources` | No | YAML `resources` block for the Query Proxy container. | +| `serviceType` | No | Gateway Service type. Valid values are `ClusterIP`, the default, and `LoadBalancer`. | +| `serviceAnnotations` | No | YAML map of annotations applied to a `LoadBalancer` gateway Service. | +| `grafanaUrl` | No | Grafana upstream URL used by the Platform `/grafana/` reverse proxy. | +| `grafanaInsecureSkipVerify` | No | Set to `"true"` only to test a Grafana endpoint with an unverifiable certificate. | +| `grafanaCertSecretName` | No | Grafana serving certificate Secret. Platform uses its `ca.crt` to trust the Grafana upstream. | +| `fleetObservabilityDashboardUid` | No | UID of the fleet dashboard embedded by the Fleet Observability page. | + +## Customize the gateway serving certificate + +The gateway always serves OTLP and query traffic over TLS. When you don't specify a +certificate Secret, Platform creates and manages a self-signed Secret named +`gateway--tls`. + +To use your own certificate, create a Secret in the Platform namespace: + +```yaml title="gateway-tls.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: fleet-gateway-tls + namespace: vcluster-platform +type: kubernetes.io/tls +stringData: + tls.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + tls.key: | + -----BEGIN PRIVATE KEY----- + ... + -----END PRIVATE KEY----- + ca.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +``` + +Reference it from the connector: + +```yaml +stringData: + observabilityGatewayCertSecretName: fleet-gateway-tls +``` + +Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom +certificate cases. Distribute `ca.crt` to collectors that don't already trust the issuing +CA. Grafana datasources generated by Platform trust the `ca.crt` from the gateway serving +certificate Secret. If a custom certificate is signed by a public CA, `ca.crt` can be +omitted and clients use their system trust stores. + +## Expose the gateway with a LoadBalancer + +The gateway Service defaults to `ClusterIP`. To make the gateway reachable from collectors +and query clients outside the Platform cluster, set `serviceType` to `LoadBalancer`. You can +optionally apply provider-specific Service annotations: + +```yaml +stringData: + serviceType: LoadBalancer + serviceAnnotations: | + service.beta.kubernetes.io/aws-load-balancer-scheme: internal +``` + +For a Platform-managed certificate, the controller waits for the LoadBalancer address and +adds it to the certificate as a subject alternative name before deploying the gateway. This +means clients that verify TLS trust the address without extra configuration. For a +self-signed certificate, distribute `ca.crt` as described in +[Customize the gateway serving certificate](#customize-the-gateway-serving-certificate). A +certificate signed by a public CA needs no extra trust configuration. + +### Retrieve the load balancer address + +The cloud provider assigns the address asynchronously, so it stays empty until provisioning +finishes. Read it from the gateway Service: + +```bash +CONNECTOR_ID=fleet-observability +kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \ + -o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}' +``` + +The provider reports either a hostname (for example, AWS) or an IP address (for example, +GCP and Azure). Build the reachable endpoints from that address: + +| Endpoint | Port | Purpose | +|---|---:|---| +| `https://:4318` | `4318` | OTLP HTTP writes. Use this for collectors. | +| `https://:4317` | `4317` | OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use `4318` for collectors. | +| `https://:8081` | `8081` | Prometheus HTTP API through the Query Proxy. Use this for external query clients. | + +Use this address wherever a collector or query client connects from outside the Platform +cluster. + +## Use backend mTLS + +To authenticate the gateway to your backend with mTLS, create a Secret: + +```yaml title="backend-client-tls.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: metrics-backend-client-tls + namespace: vcluster-platform +type: kubernetes.io/tls +stringData: + ca.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + tls.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + tls.key: | + -----BEGIN PRIVATE KEY----- + ... + -----END PRIVATE KEY----- +``` + +Reference it from the connector: + +```yaml +stringData: + metricsBackendCertSecretName: metrics-backend-client-tls +``` + +## Verify the gateway + +Check that the gateway pod is running: + +```bash +CONNECTOR_ID=fleet-observability +kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" +``` + +Check the service ports: + +```bash +CONNECTOR_ID=fleet-observability +kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" +``` + +Expected service ports: + +| Port | Name | Purpose | +|---:|---|---| +| `4317` | `otlp` | OTLP gRPC writes. | +| `4318` | `otlp-http` | OTLP HTTP writes. | +| `8081` | `query` | Prometheus HTTP API through the Query Proxy. | + +Next, create metrics access keys in [Configure metrics access](./configure-metrics-access.mdx). diff --git a/platform/maintenance/observability/labels-reference.mdx b/platform/maintenance/observability/labels-reference.mdx new file mode 100644 index 0000000000..36fdd55c30 --- /dev/null +++ b/platform/maintenance/observability/labels-reference.mdx @@ -0,0 +1,94 @@ +--- +title: Fleet Observability labels reference +sidebar_label: Labels reference +sidebar_position: 7 +description: Reference for vcluster_platform labels stamped by the Fleet Observability Write Gateway. +--- + +The Write Gateway owns the full `vcluster_platform_*` label namespace. These labels are +stamped from authenticated Platform context and are safe to use for dashboards, +recording rules, and backend queries. + +:::warning +Don't set labels with the `vcluster_platform_*` prefix in collectors or applications. +The Write Gateway strips caller-supplied platform labels before adding authenticated +values. +::: + +## Scope-controlling labels + +The Query Proxy uses these labels for read authorization. + +| Label | Meaning | Header equivalent | +|---|---|---| +| `vcluster_platform_cluster` | Control plane cluster identity. | `X-Vcluster-Platform-Cluster` | +| `vcluster_platform_project` | Tenant cluster project. | `X-Vcluster-Platform-Project` | +| `vcluster_platform_instance` | VirtualClusterInstance name. | `X-Vcluster-Platform-Instance` | + +Use either cluster scope or tenant scope, not both: + +- Cluster scope: `vcluster_platform_cluster` +- Tenant scope: `vcluster_platform_project` and `vcluster_platform_instance` + +## Descriptive labels + +These labels are stamped at write time for filtering and grouping, but they don't +authorize reads. + +| Label | Meaning | +|---|---| +| `vcluster_platform_hosted_on` | Registered control plane cluster that hosts the tenant cluster. | +| `vcluster_platform_placement` | Placement category, such as `hosted`, `standalone`, or `external`. | +| `vcluster_platform_shape` | Node arrangement, such as `control_plane`, `shared_nodes`, or `private_nodes`. | +| `vcluster_platform_node_management` | Private-node management mode, such as `manual` or `auto`. Empty for non-private-node shapes. | + +## Matcher rules + +The Query Proxy restricts matchers on scope-controlling labels: + +| Matcher | Scope labels | Example | +|---|---|---| +| Equality | Allowed | `vcluster_platform_instance="api"` | +| Literal alternation regular expression | Allowed | `vcluster_platform_instance=~"api\|worker"` | +| Negative equality | Rejected | `vcluster_platform_instance!="api"` | +| Negative regular expression | Rejected | `vcluster_platform_instance!~"api"` | +| Wildcard regular expression | Rejected | `vcluster_platform_instance=~".*"` | + +Descriptive labels can use normal PromQL matcher forms because they don't determine +authorization scope. + +## Label examples + +Tenant cluster metric: + +```text +vcluster_platform_project="team-a" +vcluster_platform_instance="api" +vcluster_platform_hosted_on="local-cluster" +vcluster_platform_placement="hosted" +vcluster_platform_shape="shared_nodes" +``` + +Control plane cluster metric: + +```text +vcluster_platform_cluster="local-cluster" +``` + +Private-nodes tenant cluster metric: + +```text +vcluster_platform_project="gpu-team" +vcluster_platform_instance="inference" +vcluster_platform_hosted_on="local-cluster" +vcluster_platform_placement="hosted" +vcluster_platform_shape="private_nodes" +vcluster_platform_node_management="auto" +``` + +## Migration from older labels + +Older OpenTelemetry examples used labels such as `cluster`, `vcluster_name`, +`vcluster_project`, or `loft_virtualcluster_name`. For the Fleet Observability gateway, +prefer the `vcluster_platform_*` labels. They are stamped after authentication and are +the label contract used by the Query Proxy. diff --git a/platform/maintenance/observability/overview.mdx b/platform/maintenance/observability/overview.mdx new file mode 100644 index 0000000000..6f21344096 --- /dev/null +++ b/platform/maintenance/observability/overview.mdx @@ -0,0 +1,143 @@ +--- +title: Fleet Observability +sidebar_label: Overview +sidebar_position: 1 +description: Monitor tenant clusters across your vCluster Platform fleet with an observability gateway, scoped access keys, and platform-owned metric labels. +--- + +import FleetObservabilityArchitecture from '@site/static/media/diagrams/fleet-observability-architecture.svg'; + +Fleet Observability gives platform operators a central telemetry path for tenant clusters and +control plane clusters. Telemetry flows over OTLP through a vCluster Platform observability +gateway before it reaches your backend. The gateway authenticates writes, stamps +platform-owned labels, and exposes a scoped Prometheus query API for Grafana and other +read clients. + +Use this guide when you need fleet-wide visibility across projects, tenant clusters, +shared nodes, private nodes, or connected clusters. + +## Get started + +The default setup is out of the box and driven from the Platform UI. Three steps: + +1. **Deploy the metrics backend and Grafana.** Use the bundled Argo CD templates to deploy + Prometheus and Grafana. See [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). +2. **Create the observability connector.** In the Platform UI, open **Connectors** and + connect your backend on the **Observability** tab. Platform then deploys the gateway. See + [Install the observability gateway](./install-observability-gateway.mdx). +3. **Deploy the edge collectors.** Create a `metrics-writer` access key, then deploy the + bundled collector template to each cluster to collect from. See + [Configure metrics access](./configure-metrics-access.mdx) and + [Deploy collectors to each cluster](./deploy-with-argocd.mdx#deploy-collectors-to-each-cluster). + +:::info Scope +Fleet Observability is currently intended for platform administrators. It provides the +gateway, query proxy, metrics access-key roles, label contract, and both an out-of-the-box +and a manual configuration path for collectors and backends. The bundled Grafana template +provides pre-provisioned dashboards today. Platform automatically manages fleet, tenant +cluster, and control plane cluster datasource ConfigMaps. Per-tenant connector selection, +access-key generation, and automatic deployment and reconciliation of the observability +applications are planned as follow-up work. + +The current release focuses on metrics. Edge collectors and the gateway communicate over +OTLP. +::: + +:::warning Not available on embedded Kubernetes +Fleet Observability runs the metrics gateway and Grafana datasources on the platform +cluster, so it requires a platform cluster that can run those applications. It is +currently not available when the platform runs on embedded Kubernetes, which backs the +external-database and multi-region platform features. In those deployments Fleet +Observability is turned off automatically. +::: + +## Architecture + +
+ +
Fleet Observability write and query paths through the vCluster Platform observability gateway
+
+ +The architecture has three main parts: + +- **Edge collectors** are OpenTelemetry Collectors that collect metrics from control plane + clusters, tenant clusters, and optional exporters. They send OTLP to the Write Gateway. +- **Write Gateway** is the vCluster-built ingestion and enforcement layer. It receives OTLP, + authenticates each write with a `metrics-writer` access key, authorizes the requested + scope with the Platform API, strips caller-supplied platform labels, and stamps the + authenticated `vcluster_platform_*` labels. +- **Query Proxy** exposes a Prometheus-compatible read API. It authenticates each read, + authorizes the requested scope with a `metrics-reader` key or user session, rewrites + PromQL so reads stay scope-bound, and forwards the query to your metrics backend. + +## What platform manages + +An observability connector Secret tells Platform how to reach your metrics backend. +When the connector is present, Platform deploys and reconciles: + +- The observability gateway: the Write Gateway and Query Proxy that front your backend. +- The Grafana datasources that power the fleet, tenant cluster, and control plane cluster + dashboards and metrics explorer. + +You must provide a backend that can import OTLP metrics and serve the Prometheus HTTP +API. You must also configure the edge collectors that send metrics to the gateway. + +If you enable the [Argo CD integration](../../integrations/argocd/overview.mdx), +Platform ships default `ArgoCDApplicationTemplate` resources for the metrics backend +and edge collectors. You can leverage these templates to create the `ArgoCDApplication` +resources that deploy the backend and collectors, instead of wiring them up by hand. +See [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). + +## Security model + +Fleet Observability doesn't rely on backend tenant isolation. The gateway and query proxy +are the enforcement layer. + +- Metrics writers must authenticate with an access key that has the `metrics-writer` + role and a matching cluster or tenant cluster scope. +- Metrics readers must authenticate with a `metrics-reader` key, a user access key, or + a Grafana session passed through to Platform. +- Platform authorizes both read and write requests with SelfSubjectAccessReview. +- The Write Gateway owns the full `vcluster_platform_*` label namespace. Caller-supplied + labels with that prefix, or dotted variants that would sanitize to that prefix, are + removed before authenticated labels are added. +- Positive authorization results are cached for a short TTL. On Platform outage, cached + decisions expire and the system fails closed. + +## Pluggable observability + +The bundled components are the default. Replace any of them as your environment requires: + +- **Metrics backend.** Swap the bundled Prometheus for any backend that ingests OTLP and + serves the Prometheus HTTP API. Integration is limited to an OTLP-enabled endpoint with + mTLS for authentication, so backends that require other auth methods aren't supported yet. +- **Edge collectors.** Run your own OpenTelemetry Collectors instead of the bundled + template. See [Configure edge collectors](./configure-edge-collectors.mdx). +- **Grafana.** The bundled Grafana is tightly coupled to the provided dashboards and + datasources, and to Platform SSO with access-key cookie forwarding for scoped queries. + Replacing it is possible but involved. + +## Page guide + +| Task | Page | +|---|---| +| Deploy the metrics backend and Grafana with Argo CD | [Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx) | +| Create the connector and deploy the gateway | [Install the observability gateway](./install-observability-gateway.mdx) | +| Create scoped metrics access keys | [Configure metrics access](./configure-metrics-access.mdx) | +| Send metrics from collectors to the gateway | [Configure edge collectors](./configure-edge-collectors.mdx) | +| Query metrics through Grafana or PromQL | [Query fleet metrics](./query-fleet-metrics.mdx) | +| Understand the platform label contract | [Labels reference](./labels-reference.mdx) | +| Diagnose common failures | [Troubleshooting](./troubleshooting.mdx) | + +## Related monitoring guides + +For a simpler OpenTelemetry setup without the observability gateway, see +[Aggregating metrics](../monitoring/aggregating-metrics.mdx). For Shared Nodes kubelet scraping +background, see [Prometheus node metrics on shared nodes](../monitoring/prometheus-shared-nodes.mdx). +For centralized workload log collection across your fleet, see +[Central HostPath Mapper](../monitoring/central-hostpath-mapper.mdx). To understand the log +types vCluster produces, see [vCluster Logging](/docs/vcluster/manage/logging). diff --git a/platform/maintenance/observability/query-fleet-metrics.mdx b/platform/maintenance/observability/query-fleet-metrics.mdx new file mode 100644 index 0000000000..0124754626 --- /dev/null +++ b/platform/maintenance/observability/query-fleet-metrics.mdx @@ -0,0 +1,245 @@ +--- +title: Query fleet metrics +sidebar_label: Query metrics +sidebar_position: 6 +description: Query fleet metrics through the Fleet Observability Query Proxy with scoped headers and platform-owned labels. +--- + +Query fleet metrics through the bundled Grafana, the fastest path once it's deployed, or +directly against the Query Proxy's Prometheus-compatible HTTP API for automation and your +own query clients. In both cases the Query Proxy authorizes each request with Platform, +injects the requested scope into PromQL, and forwards the rewritten query to your metrics +backend. + +## Query with the bundled Grafana + +If you deploy the Fleet Observability: Grafana template, Platform runs a bundled Grafana +pre-equipped with fleet dashboards and scoped datasources. It is the fastest way to explore +fleet metrics without writing PromQL by hand. See +[Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx#deploy-grafana) to deploy it. + +### Sign in + +The bundled Grafana is served at `https:///grafana/` and reverse-proxied by +Platform. Sign in with Platform SSO; the local Grafana login form is disabled. Your Grafana +role follows your Platform group membership: members of the template's `grafanaAdminGroup` +get the Grafana Admin role, everyone else gets Viewer. The Grafana role governs Grafana +features, not which metrics you can read. + +### Dashboards + +Three dashboards ship pre-provisioned: + +| Dashboard | View | +|---|---| +| **Fleet Observability** | Fleet-wide summary: tenant cluster counts, GPU inventory and utilization, hardware-error hotspots, and per-tenant scrape health. | +| **Cluster Metrics** | A single tenant cluster or control plane cluster. Scope comes from the selected datasource, so there are no dashboard filters. | +| **GPU Overview** | GPU fleet view: DCGM utilization, thermals, hardware-error signals, and NVSentinel fault detection. | + +Each dashboard has a **Datasource** dropdown. Pick the datasource for the scope you want to +view. The Platform UI also surfaces these dashboards directly: the **Fleet Observability** +page shows the fleet and GPU views, and each tenant cluster's **Observability** tab shows +its scoped Cluster Metrics dashboard. + +
+ Fleet Observability dashboard showing tenant cluster counts, ready nodes, GPU inventory, per-project tenant cluster breakdowns, resource utilization, and top clusters by memory and CPU across the fleet +
The Fleet Observability dashboard: a fleet-wide summary across projects and tenant clusters.
+
+ +
+ GPU Overview dashboard showing GPU count, GPUs in use, utilization, temperature, total power, ECC error counts, and per-GPU utilization, filterable by GPU model and node +
The GPU Overview dashboard: GPU utilization, thermals, and hardware-error signals, filterable by GPU model and node.
+
+ +
+ Cluster Metrics dashboard for a single tenant cluster showing ready nodes, GPUs, resource utilization, pod count, and attached GPUs by type +
The Cluster Metrics dashboard for a single tenant cluster, opened from its Observability tab.
+
+ +### Datasources and scope + +Platform auto-provisions Prometheus datasources that point at the Query Proxy. You don't +create them by hand: + +- A **fleet-wide** datasource with no scope headers, for fleet and platform-admin views. +- A **per-tenant-cluster** datasource for each tenant cluster, scoped to its project and + instance. +- A **per-control-plane-cluster** datasource for each control plane cluster. + +Tenant cluster datasources inject scope through request headers: + +```yaml +jsonData: + httpHeaderName1: X-Vcluster-Platform-Project + httpHeaderName2: X-Vcluster-Platform-Instance + keepCookies: + - loft_access_key +secureJsonData: + httpHeaderValue1: my-project + httpHeaderValue2: my-vcluster +``` + +Control plane cluster datasources set `X-Vcluster-Platform-Cluster` instead, and the +fleet-wide datasource sets no scope headers. + +### Authorization with your access key + +Every datasource forwards your signed-in Platform access key as the `loft_access_key` +cookie to the Query Proxy through Grafana's `keepCookies` setting. This is the session-cookie +path noted in [Create a reader key](./configure-metrics-access.mdx#create-a-reader-key), so +Grafana users don't need a static `metrics-reader` key. + +The Query Proxy authorizes every query as your user, so the datasource you pick can't widen +your access. If you select the fleet-wide datasource, you still only get the metrics your +access key is allowed to read. Only Platform admins receive fleet-wide results from it. + +## Endpoint + +Query the Query Proxy's HTTP API directly when you automate reads or bring your own query +client. Inside the Platform cluster: + +```text +https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:8081/api/v1/query +``` + +This endpoint assumes the example connector ID `fleet-observability`. For another ID, +replace the Service name with `gateway-`. + +To connect your own datasource or query client from outside the Platform cluster, expose +the gateway with a LoadBalancer and use its address on port `8081` instead of the in-cluster +Service name: + +```text +https://:8081/api/v1/query +``` + +See [Expose the gateway with a LoadBalancer](./install-observability-gateway.mdx#expose-the-gateway-with-a-loadbalancer) +to enable it and retrieve the address. Scope headers and `metrics-reader` authorization are +unchanged; only the host differs. + +The gateway always uses TLS. For the default Platform-managed certificate, read `ca.crt` +from the `gateway--tls` Secret and make it available to your query client. The +examples use its local path through the `GATEWAY_CA_FILE` environment variable: + +```bash +CONNECTOR_ID=fleet-observability +kubectl get secret -n vcluster-platform "gateway-$CONNECTOR_ID-tls" -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/gateway-ca.crt +export GATEWAY_CA_FILE=/tmp/gateway-ca.crt +``` + +## Query a tenant cluster + +Send a `metrics-reader` token from +[Create a reader key](./configure-metrics-access.mdx#create-a-reader-key) and tenant scope +headers: + +```bash +curl -G \ + --cacert "$GATEWAY_CA_FILE" \ + -H "Authorization: Bearer $METRICS_READER_TOKEN" \ + -H "X-Vcluster-Platform-Project: my-project" \ + -H "X-Vcluster-Platform-Instance: my-vcluster" \ + --data-urlencode 'query=sum(rate(container_cpu_usage_seconds_total[5m]))' \ + https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:8081/api/v1/query +``` + +The Query Proxy injects tenant scope into vector selectors before forwarding the query. +The backend receives an equivalent query constrained by: + +```promql +vcluster_platform_project="my-project", +vcluster_platform_instance="my-vcluster" +``` + +## Query a control plane cluster + +```bash +curl -G \ + --cacert "$GATEWAY_CA_FILE" \ + -H "Authorization: Bearer $METRICS_READER_TOKEN" \ + -H "X-Vcluster-Platform-Cluster: local-cluster" \ + --data-urlencode 'query=sum(rate(apiserver_request_total[5m]))' \ + https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:8081/api/v1/query +``` + +The backend query is constrained by: + +```promql +vcluster_platform_cluster="local-cluster" +``` + +## Use labels in PromQL + +You can also include scope labels directly in PromQL. Equality matchers and literal +alternation regular expression matchers are allowed on scope-controlling labels. + +```promql +sum by (vcluster_platform_project, vcluster_platform_instance) ( + rate(container_cpu_usage_seconds_total{ + vcluster_platform_project="my-project", + vcluster_platform_instance=~"api|worker" + }[5m]) +) +``` + +The Query Proxy rejects unsafe scope matchers such as `!=`, `!~`, `.*`, or regexes +that aren't top-level literal alternations. Descriptive labels can use normal PromQL +matchers because they don't drive authorization. + +## Example queries + +CPU by tenant cluster: + +```promql +sum by (vcluster_platform_project, vcluster_platform_instance) ( + rate(container_cpu_usage_seconds_total[5m]) +) +``` + +Memory by tenant cluster: + +```promql +sum by (vcluster_platform_project, vcluster_platform_instance) ( + container_memory_working_set_bytes +) +``` + +API request rate by cluster: + +```promql +sum by (vcluster_platform_cluster, verb) ( + rate(apiserver_request_total[5m]) +) +``` + +GPU utilization by tenant cluster, when DCGM metrics are collected: + +```promql +avg by (vcluster_platform_project, vcluster_platform_instance, gpu) ( + DCGM_FI_DEV_GPU_UTIL +) +``` + +## Query failures + +If a query is rejected, check: + +- The bearer token has `metrics-reader` scope or the signed-in user can read metrics + for the requested resource. +- The scope headers match the key scope. +- Scope-controlling label matchers use `=` or a literal `=~` alternation. +- The Platform API is reachable from the Query Proxy. + +For more details, see [Troubleshooting](./troubleshooting.mdx). diff --git a/platform/maintenance/observability/troubleshooting.mdx b/platform/maintenance/observability/troubleshooting.mdx new file mode 100644 index 0000000000..6d7e29b6d6 --- /dev/null +++ b/platform/maintenance/observability/troubleshooting.mdx @@ -0,0 +1,165 @@ +--- +title: Troubleshooting Fleet Observability +sidebar_label: Troubleshooting +sidebar_position: 8 +description: Diagnose common Fleet Observability gateway, access-key, collector, and query-proxy problems. +--- + +Use this page to diagnose the Fleet Observability path from edge collectors through the +observability gateway and into your metrics backend. + +## Gateway resources aren't created + +Check the connector Secret: + +```bash +kubectl get secret -n vcluster-platform fleet-observability -o yaml +``` + +Verify: + +- The Secret is in the Platform namespace. +- `metadata.labels["loft.sh/connector-type"]` is `observability`. +- The fleet connector has the `platform.vcluster.com/fleet-observability-connector` label set to `"true"`. +- `metricsBackendOtlpEndpoint` and `metricsBackendPromQLEndpoint` are set. + +The gateway isn't reconciled until both backend endpoints are present. + +## Multiple fleet connectors + +Only one connector can carry the fleet label: + +```yaml +platform.vcluster.com/fleet-observability-connector: "true" +``` + +If multiple connector Secrets have this label, Platform still creates per-connector +gateway resources, but it doesn't publish the fleet-wide platform admin Grafana +datasource. Remove the fleet label from all but one connector. + +## Collector writes are rejected + +Check the collector exporter configuration: + +- The `Authorization` header uses a valid bearer token. +- The token is a `metrics-writer` access key. +- The request includes either `X-Vcluster-Platform-Cluster` or the + `X-Vcluster-Platform-Project` and `X-Vcluster-Platform-Instance` pair. +- The header values match the key scope. + +Check gateway logs: + +```bash +kubectl logs -n vcluster-platform deploy/gateway-fleet-observability -c otel-gateway +``` + +Common causes: + +| Symptom | Likely cause | +|---|---| +| Missing bearer token | Collector didn't set the `Authorization` header. | +| Scope denied | Key scope doesn't include the requested cluster or tenant cluster. | +| Ambiguous scope | Request included both cluster and tenant scope, or multiple values for one scope header. | +| Headerless write rejected | The key has zero or multiple concrete tenant scopes, so scope can't be derived. | + +## Queries are rejected + +Check query-proxy logs: + +```bash +kubectl logs -n vcluster-platform deploy/gateway-fleet-observability -c query-proxy +``` + +Common causes: + +| Symptom | Likely cause | +|---|---| +| HTTP `400` | Missing scope for a non-admin user, invalid scope matcher, or malformed PromQL. | +| HTTP `403` | The user or key is authenticated but not authorized for the requested metrics scope. | +| Query succeeds for admins only | The datasource is using the unscoped fleet-admin path. Add tenant or cluster scope headers for non-admin users. | +| Regex query rejected | Scope-controlling labels only allow equality or literal alternation regular expression matchers. | + +## Platform labels are missing + +If metrics arrive without `vcluster_platform_*` labels: + +- Confirm the metrics went through the Write Gateway. Direct writes to the backend + don't get platform labels. +- Confirm the collector used a valid metrics writer key and scope headers. +- Confirm the backend stores labels from OTLP datapoint attributes. +- Check whether the metric is expected to have tenant labels or only cluster labels. + +The Write Gateway strips caller-supplied platform labels and then stamps authenticated +values. Empty authenticated values are dropped. + +## Backend TLS failures + +If gateway pods start but can't write or query the backend: + +- Verify `metricsBackendOtlpEndpoint` and `metricsBackendPromQLEndpoint`. +- Set `metricsBackendServerName` when the backend certificate's DNS name differs from + the endpoint host. +- Use `metricsBackendCertSecretName` when the backend requires mTLS. This Secret is also the + source of trust for a private backend CA. It must contain `ca.crt`, `tls.crt`, and + `tls.key`. +- Use `metricsBackendInsecureSkipVerify: "true"` only as a temporary test setting. + +## Gateway TLS failures + +The gateway always serves OTLP and query traffic over TLS. If collectors or query clients +can't connect: + +- For the Platform-managed certificate, read `ca.crt` from the + `gateway--tls` Secret and configure the client to trust it. +- For a custom private certificate, include `ca.crt` in the Secret referenced by + `observabilityGatewayCertSecretName`. +- Verify that the client uses `https` for OTLP HTTP and query traffic or enables TLS for + OTLP gRPC. +- If the gateway uses a `LoadBalancer` Service and a Platform-managed certificate, wait for + the LoadBalancer address before expecting the gateway Deployment to become available. + +## Platform API outage + +The gateway and query proxy fail closed when Platform authorization is unavailable. +Previously allowed decisions can continue only until their short positive cache TTL +expires. + +Expected behavior: + +- Collectors may queue and retry writes locally, depending on their exporter settings. +- Queries fail visibly after cached authorization decisions expire. +- Writes and reads recover when the Platform API is reachable again. + +## Useful checks + +Gateway pods: + +```bash +CONNECTOR_ID=fleet-observability +kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" +``` + +Gateway service: + +```bash +CONNECTOR_ID=fleet-observability +kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" +``` + +Write Gateway metrics: + +```bash +CONNECTOR_ID=fleet-observability +GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}') +kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 +curl http://127.0.0.1:8889/metrics +``` + +Query Proxy metrics: + +```bash +CONNECTOR_ID=fleet-observability +GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}') +kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8080:8080 +curl http://127.0.0.1:8080/metrics +``` diff --git a/static/img/fleet-observability/fleet-observability-gpu-detail-dashboard.png b/static/img/fleet-observability/fleet-observability-gpu-detail-dashboard.png new file mode 100644 index 0000000000..f7de81bd9d Binary files /dev/null and b/static/img/fleet-observability/fleet-observability-gpu-detail-dashboard.png differ diff --git a/static/img/fleet-observability/fleet-observability-overview-dashboard.png b/static/img/fleet-observability/fleet-observability-overview-dashboard.png new file mode 100644 index 0000000000..20697ab11e Binary files /dev/null and b/static/img/fleet-observability/fleet-observability-overview-dashboard.png differ diff --git a/static/img/fleet-observability/tenant-cluster-observability-dashboard.png b/static/img/fleet-observability/tenant-cluster-observability-dashboard.png new file mode 100644 index 0000000000..3a88e64b14 Binary files /dev/null and b/static/img/fleet-observability/tenant-cluster-observability-dashboard.png differ diff --git a/static/media/diagrams/fleet-observability-architecture.svg b/static/media/diagrams/fleet-observability-architecture.svg new file mode 100644 index 0000000000..816496b100 --- /dev/null +++ b/static/media/diagrams/fleet-observability-architecture.svg @@ -0,0 +1,116 @@ + + + + + + + + + + + + + Edge collectors + + + + + Control Plane Cluster + + + OTel collector + + + + + Tenant cluster + + + OTel collector + + + + + GPU and custom exporters + + + scraped by collectors + + + + + vCluster Platform namespace + + + + + Write Gateway + + + metrics-writer auth + + + platform label stamping + + + + + Query Proxy + + + metrics-reader auth + + + PromQL scope rewrite + + + + + Platform API + + + SSAR authorization + + + + + Metrics backend + + + OTLP ingest + + + Prometheus HTTP API + + + + + Grafana or PromQL + + + scoped read clients + + + + OTLP + + + + + + + + + OTLP + + + + + + + + +