Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
edf941e
docs: add fleet monitoring phase 1 guide
djwfyi Jul 9, 2026
7db8753
docs: add fleet monitoring architecture diagram
djwfyi Jul 9, 2026
bbafaf3
docs(monitoring): fix vale warnings on fleet monitoring phase 1 docs
djwfyi Jul 9, 2026
4ecedd8
docs(monitoring): style pass on fleet monitoring phase 1 docs
djwfyi Jul 9, 2026
ca989db
docs: add fleet monitoring dashboard screenshots
djwfyi Jul 9, 2026
6efa5eb
docs(monitoring): correct platform-managed resources in fleet overview
johannesfrey Jul 13, 2026
66b7f4e
docs(monitoring): add argo cd out-of-the-box deployment guide for fle…
johannesfrey Jul 16, 2026
e8a9f43
docs(monitoring): reword phase 1 scope note in fleet overview
johannesfrey Jul 16, 2026
ab09f81
docs(monitoring): switch fleet access key examples to generate-and-ap…
johannesfrey Jul 16, 2026
12c3830
docs(monitoring): document querying fleet metrics via the bundled gra…
johannesfrey Jul 16, 2026
330b2e5
docs(monitoring): link access key creation sections from bearer token…
johannesfrey Jul 16, 2026
3435edb
docs(monitoring): add bundled grafana dashboard screenshots
johannesfrey Jul 16, 2026
cb6755b
docs(monitoring): link reader key note to the bundled grafana authori…
johannesfrey Jul 16, 2026
5640add
docs(monitoring): document manual collector restart on key or config …
johannesfrey Jul 16, 2026
5b42255
docs(monitoring): note fleet monitoring is for platform admins only
johannesfrey Jul 16, 2026
b882ef4
docs(monitoring): target a tenant cluster in the edge collector deplo…
johannesfrey Jul 16, 2026
951cc77
docs(monitoring): add yaml and platform ui tabs to connector secret s…
johannesfrey Jul 16, 2026
03c9cbc
docs(monitoring): escape pipe in query proxy matcher table
johannesfrey Jul 16, 2026
59d7a61
docs: allow platform screenshots when warranted
djwfyi Jul 16, 2026
e67920f
docs(observability): rename fleet monitoring to fleet observability
djwfyi Jul 16, 2026
79daf41
docs(observability): distinguish gateway path from direct otel pipelines
djwfyi Jul 16, 2026
fdc1b09
docs(observability): fix connector label, gateway TLS, and key inaccu…
djwfyi Jul 16, 2026
e7abbaf
docs(observability): show how to populate GATEWAY_CA_FILE
djwfyi Jul 16, 2026
76b5922
docs(observability): drop node-exporter and kube-state-metrics from c…
johannesfrey Jul 17, 2026
338a0e4
docs(observability): note fleet observability is unavailable on embed…
johannesfrey Jul 17, 2026
8db846d
docs(observability): note connector-type label is required for the co…
johannesfrey Jul 17, 2026
05967a9
docs(observability): clarify access key snippets are examples, not pr…
johannesfrey Jul 17, 2026
7857c4a
docs(observability): note argo cd collectors set scope headers automa…
johannesfrey Jul 17, 2026
9493aa8
docs(observability): soften embedded k8s limitation to currently not …
johannesfrey Jul 17, 2026
ee565e4
docs(observability): load balancer exposing the otel gateway, rewording
seanschneeweiss Jul 20, 2026
ad7202f
docs(observability): getting started with ootb, ui first, optimize order
seanschneeweiss Jul 20, 2026
00f8079
merge: resolve main conflicts in vale config, CLAUDE.md, and legacy o…
djwfyi Jul 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/styles/config/vocabularies/Loft/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ SSH
SSHKey[s]?
StatefulSet
TSNet
TLS
mTLS
PromQL
URL
vCluster
vcluster.yaml
Expand All @@ -89,4 +92,4 @@ rpm.vcluster.com
[vVirtual] [cC]luster
Velero
VolumeAttachment
(?i)yaml
(?i)yaml
11 changes: 11 additions & 0 deletions .vale.ini
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,17 @@ Vale.Terms = NO
BasedOnStyles = Loft, Google, Vale
Vale.Terms = NO

# `metrics-backend-mtls` is a literal Kubernetes Secret name (must stay
# lowercase per RFC 1123), and "Deploy Grafana" is correct sentence case with
# a proper noun that Google.Headings doesn't recognize as one. Neither needs
# to change, so scope both rules off for this page instead of the vocab (which
# would create new false positives on the "grafana" anchors and reverse-proxy
# path used throughout the fleet monitoring pages).
[**/observability/deploy-with-argocd.mdx]
BasedOnStyles = Loft, Google, Vale
Vale.Terms = NO
Google.Headings = NO

# Auto-generated by scripts/generate-platform-cli-index.js from
# vcluster/cli/vcluster_platform*.md. The regenerate-instructions comment
# references that file path in plain text, and command descriptions are
Expand Down
5 changes: 4 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,10 @@ When writing docs for new features:
### Examples format

- vcluster.yaml: description + code block
- Platform: description + step-by-step (no UI screenshots - they go stale)
- Platform: description + step-by-step. Minimize UI screenshots since the UI
changes often and screenshots go stale, but they're okay where warranted
(for example, illustrating a dashboard or layout that's hard to describe in
prose).

### Checklist

Expand Down
39 changes: 20 additions & 19 deletions platform/maintenance/monitoring/central-hostpath-mapper.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Central HostPath Mapper
sidebar_label: Central HostPath Mapper
sidebar_position: 6
sidebar_position: 7
sidebar_class_name: free
---

Expand All @@ -17,9 +17,9 @@ import FeatureTable from '@site/src/components/FeatureTable';

You can install a common logging solution such as Fluentd or the ELK stack on the Control Plane Cluster and use it within the tenant cluster. However, in some scenarios, installing these tools directly on the tenant cluster is preferred.

These logging solutions access pod and container logs by mounting node host paths and using specific filename formats to associate logs with the correct pods. This approach doesnt work in a vCluster because the vCluster rewrites pod names to a vCluster-specific format when syncing them to the Control Plane Cluster.
These logging solutions access pod and container logs by mounting node host paths and using specific filename formats to associate logs with the correct pods. This approach doesn't work in a tenant cluster because vCluster rewrites pod names to a vCluster-specific format when syncing them to the Control Plane Cluster.

The HostPath Mapper collects logs from tenant clusters. It works by matching virtual pod and container names to their physical names on the host and does this through symbolic links.
The HostPath Mapper collects logs from tenant clusters. It works by matching virtual pod and container names to their physical names on the host through symbolic links.

To enable HostPath Mapper for a tenant cluster, install a DaemonSet in the same namespace. This setup works well for small environments with few nodes and tenant clusters.

Expand All @@ -32,7 +32,7 @@ To use this feature, install the Central HostPath Mapper on the Control Plane Cl
:::

:::warning Templates
This feature is only configurable from the UI for tenant clusters created without a template. When using a custom template that has the feature enabled, the UI does not display the configuration in **Advanced Options** as described below. In this case, configure the feature using the CLI or by modifying the manifests.
This feature is only configurable from the UI for tenant clusters created without a template. When using a custom template that has the feature enabled, the UI doesn't display the configuration in **Advanced Options** as described below. In this case, configure the feature using the CLI or by modifying the manifests.
:::

<Tabs
Expand All @@ -45,28 +45,28 @@ This feature is only configurable from the UI for tenant clusters created withou
<TabItem value="creation">
<Flow id="enable-central-hpm-virtualcluster-creation">
<Step>
From the project drop-down menu, select the project you'd like to create
the tenant cluster in.
From the project drop-down menu, click the project where you want to
create the tenant cluster.
</Step>
<Step>
Click the <Button>New Tenant Cluster</Button> button above the list of tenant clusters
Click the <Button>New Tenant Cluster</Button> button above the list of tenant clusters.
</Step>
<Step>
<b>[Optional]</b> select the cluster in which to create the tenant
<b>[Optional]</b> Click the cluster where you want to create the tenant
cluster.
</Step>
<Step>
Leave the <Label>Select a template</Label> section blank and click the <Button>Configure</Button> to continue.
</Step>
<Step>
Click the <Label>Advanced Options</Label> configuration tab and expand
the 'Host Path Mapper' section.
the **Host Path Mapper** section.
</Step>
<Step>
Toggle the slider <Label>Enable</Label>. This will add the
'loft.sh/hpm-enabled' annotation to the template metadata which will be
automatically synced to the tenant cluster on creation. Secondly, it
will add `enabled: true` & `central: true` under the '[controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper)' field in the vcluster.yaml.
Toggle the slider <Label>Enable</Label>. This adds the
`loft.sh/hpm-enabled` annotation to the template metadata, which syncs
automatically to the tenant cluster on creation. It also sets
`enabled: true` and `central: true` under the [controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper) field in the vcluster.yaml.
</Step>
<Step>
Finish configuring anything else on your tenant cluster,
Expand All @@ -78,21 +78,22 @@ This feature is only configurable from the UI for tenant clusters created withou
<TabItem value="existing">
<Flow id="enable-central-hpm-virtualcluster-existing">
<Step>
Select the project that contains the tenant cluster you want to enable Central HostPath Mapper on from the project drop-down menu.
From the project drop-down menu, click the project that contains the
tenant cluster where you want to enable Central HostPath Mapper.
</Step>
<Step>
Hover over the tenant cluster you would like to enable Central HostPath Mapper on to see an <Label>Edit</Label> button displayed.
Hover over the tenant cluster where you want to enable Central HostPath Mapper to reveal the <Label>Edit</Label> button.
Click <Label>Edit</Label>.
</Step>
<Step>
Click the <Label>Advanced Options</Label> configuration tab and expand
the **Host Path Mapper** section.
</Step>
<Step>
Toggle the slider <Label>Enable</Label>. This will add the
'loft.sh/hpm-enabled' annotation to the template metadata which will be
automatically synced to the tenant cluster on creation. Secondly, it
will add `enabled: true` & `central: true` under the '[controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper)' field in the vcluster.yaml.
Toggle the slider <Label>Enable</Label>. This adds the
`loft.sh/hpm-enabled` annotation to the template metadata, which syncs
automatically to the tenant cluster on creation. It also sets
`enabled: true` and `central: true` under the [controlPlane.hostPathMapper](/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper) field in the vcluster.yaml.
</Step>
<Step>
Finish configuring anything else on your tenant cluster,
Expand Down
43 changes: 21 additions & 22 deletions platform/maintenance/monitoring/fleet-monitoring-otel.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Fleet monitoring with OpenTelemetry"
sidebar_label: "Fleet monitoring with OpenTelemetry"
sidebar_position: 3
sidebar_position: 4
description: "Monitor tenant clusters across shared and private nodes using OpenTelemetry Collectors with automatic label enrichment and Prometheus remote write."
---

Expand Down Expand Up @@ -160,7 +160,7 @@ node. The `groupbyattrs` processor splits the batch into per-pod resource scopes
(by `namespace`, `pod`, `node`) so each pod is matched correctly.

The `k8sattributes` processor resolves vCluster identity from Platform-managed
namespace labels (`loft.sh/project`, `loft.sh/vcluster-instance-name`, and others) and
namespace labels (`loft.sh/project`, `loft.sh/vcluster-instance-name`, and so on) and
pod labels/annotations set by the vCluster syncer
(`vcluster.loft.sh/namespace`, `vcluster.loft.sh/name`).

Expand All @@ -170,7 +170,7 @@ which also prevents duplicate series with any existing Prometheus scrapes.

### Private nodes

The private-nodes collector runs inside each vCluster as a DaemonSet with one
The private-nodes collector runs inside each tenant cluster as a DaemonSet with one
pod per node. Each pod scrapes only its local node's kubelet `/metrics`,
cAdvisor `/metrics/cadvisor`, and API server `/metrics` endpoints.

Expand All @@ -188,14 +188,14 @@ prometheus receiver (kubelet, cAdvisor, API server)
pod → vcluster_virtual_pod)
```

Since the collector runs inside the vCluster, it can't access host-cluster
namespace labels. Instead, the Platform injects `{{ .Values.loft.* }}` template
Since the collector runs inside the tenant cluster, it can't access Control
Plane Cluster namespace labels. Instead, the Platform injects `{{ .Values.loft.* }}` template
variables at deploy time, which are set as `external_labels` on the
`prometheusremotewrite` exporter. These are static per-vCluster values applied
`prometheusremotewrite` exporter. These are static per-tenant-cluster values applied
to all exported metrics.

The `metric_relabel_configs` copy `namespace` to `vcluster_virtual_namespace`
and `pod` to `vcluster_virtual_pod`. Inside a private-nodes vCluster, the
and `pod` to `vcluster_virtual_pod`. Inside a private-nodes tenant cluster, the
`namespace` and `pod` labels already represent virtual names, so this copy
ensures dashboard compatibility with the shared-nodes collector.

Expand All @@ -219,7 +219,7 @@ All metrics from both apps carry a consistent set of identity labels:
`vcluster_virtual_namespace` and `vcluster_virtual_pod` are missing on some
metrics. These are vCluster system pods (syncer, CoreDNS) that don't have the
syncer labels and annotations because they aren't user workloads synced from
inside the vCluster.
inside the tenant cluster.
:::

## Prerequisites
Expand Down Expand Up @@ -248,7 +248,7 @@ server:

- kubelet scraping disabled in any existing kube-prometheus-stack to avoid
duplicate cAdvisor series (`kubelet.enabled: false`).
- Platform namespace labels present (added automatically by the vCluster Platform).
- Platform namespace labels present (added automatically by vCluster Platform).

### Private nodes prerequisites

Expand Down Expand Up @@ -593,7 +593,7 @@ spec:
- **`serviceMonitorSelector: app: vcluster`**: Without filtering, the Target
Allocator discovers all ServiceMonitors in the cluster, overwhelming
collectors with memory pressure.
- **`operator.targetallocator.mtls: true`**: Each vCluster exposes its API
- **`operator.targetallocator.mtls: true`**: Each tenant cluster exposes its API
server metrics over mTLS. Without this feature gate, the Target Allocator
redacts TLS private keys when passing scrape configs to collectors.
- **`otel/opentelemetry-collector-contrib` image**: The default image doesn't
Expand Down Expand Up @@ -622,7 +622,7 @@ kubectl apply -f otel-collector-shared-nodes-app.yaml
Navigate to the <NavStep>Apps</NavStep> tab.
</Step>
<Step>
Click <Button>Install App</Button> and select the
Click <Button>Install App</Button> and click the
**OTEL Collector - Shared Nodes** app.
</Step>
<Step>
Expand All @@ -645,7 +645,7 @@ Repeat these steps for each Control Plane Cluster.
## Deploy the private nodes collector
<!-- vale on -->

Deploy one private-nodes collector into each private-nodes vCluster. All
Deploy one private-nodes collector into each private-nodes tenant cluster. All
vCluster identity labels are injected automatically by the Platform using
`{{ .Values.loft.* }}`.

Expand Down Expand Up @@ -889,7 +889,7 @@ spec:
ensures one collector per node, scraping only the local kubelet and cAdvisor
using `${env:K8S_NODE_NAME}` filtering. No Target Allocator is needed.
- **`external_labels` instead of `k8sattributes` for identity**: The collector
runs inside the vCluster, so it can't access Control Plane Cluster namespace labels.
runs inside the tenant cluster, so it can't access Control Plane Cluster namespace labels.
The Platform injects `{{ .Values.loft.* }}` template variables at deploy time.
- **`resource_to_telemetry_conversion: false`**: Setting this to `true` causes
duplicate labels that break Grafana dashboards.
Expand All @@ -916,7 +916,7 @@ kubectl apply -f otel-collector-private-nodes-app.yaml
Go to the <NavStep>Projects</NavStep> section using the menu on the left.
</Step>
<Step>
Select the project containing your private-nodes tenant cluster.
Click the project containing your private-nodes tenant cluster.
</Step>
<Step>
Click on the tenant cluster, then navigate to the <NavStep>Config</NavStep>
Expand Down Expand Up @@ -1215,7 +1215,8 @@ with the event rate.

:::note
This section assumes Grafana is already deployed. For setup instructions, see
[Aggregating Metrics — Deploy Grafana](./aggregating-metrics.mdx#deploy-grafana).
the [Deploy Grafana](./aggregating-metrics.mdx#deploy-grafana) section in
Aggregating Metrics.
:::

Two Grafana dashboards are provided for visualizing metrics collected by the
Expand All @@ -1234,14 +1235,12 @@ time, so all panels use straightforward PromQL queries without joins.
### vCluster projects dashboard
<!-- vale on -->

A platform admin overview of vCluster projects across shared and private node
tenant clusters. Use this dashboard to monitor resource consumption and API health at
the project level.
A platform admin overview of vCluster projects across Shared Nodes and Private
Nodes tenant clusters. Use this dashboard to monitor resource consumption and
API health at the project level.

**Dashboard JSON file:** [Download dashboard JSON](/media/dashboards/vcluster-projects.json)



**Template variables:**

| Variable | Description |
Expand All @@ -1255,8 +1254,8 @@ the project level.
<!-- vale off -->
| Section | Panels |
|---------|--------|
| Project summary | Total Projects, Total vClusters, Total Pods, Total CPU (cores), Total Memory, Max Error Rate (stat panels) |
| Resource usage by project | CPU by Project (cores), Memory by Project, Pods by Project, vClusters by Project (bar charts) |
| Project summary | Total Projects, Total tenant clusters, Total Pods, Total CPU (cores), Total Memory, Max Error Rate (stat panels) |
| Resource usage by project | CPU by Project (cores), Memory by Project, Pods by Project, Tenant clusters by Project (bar charts) |
| API health by project | API Request Rate, API Error Rate (5xx), API P95 Latency (time series) |
| Resource trends | CPU Usage Trend (stacked), Memory Usage Trend (stacked) (time series) |
<!-- vale on -->
Expand Down
Loading
Loading