Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/styles/Google/Headings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,7 @@ exceptions:
- Helm
- FIPS
- SSO
- App Project
- App Project
- GPU
- DCGM
- NVSentinel
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,4 @@ hack/cli-drift/cli-drift
hack/cli-drift/bin
hack/config-drift/config-drift
hack/config-drift/bin
.serena/
12 changes: 12 additions & 0 deletions .vale.ini
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,18 @@ BasedOnStyles = Loft, Google, Vale
Vale.Terms = NO
Google.Headings = NO

# "Terminate" here always means the cloud/Kubernetes action of destroying a node or
# instance (matches the actual API names, e.g. ec2:TerminateInstances), not ending a
# session, so Google.WordList's stop/exit/cancel/end suggestion would be inaccurate.
# "GCP service account name" is a real GCP resource-name field, not a login username,
# so the account-name-to-username substitution is also a false positive. Scope the
# whole rule off for this page rather than editing the shared word list, since inline
# per-instance vale toggles proved unreliable across this page's wrapped list items
# and table cells.
[**/observability/nvsentinel-gpu-monitoring.mdx]
BasedOnStyles = Loft, Google, Vale
Google.WordList = NO

# Auto-generated by scripts/generate-platform-cli-index.js from
# vcluster/cli/vcluster_platform*.md. The regenerate-instructions comment
# references that file path in plain text, and command descriptions are
Expand Down
150 changes: 138 additions & 12 deletions platform/maintenance/observability/configure-edge-collectors.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,148 @@ title: Configure edge collectors
sidebar_label: Configure collectors
sidebar_position: 5
description: Configure OpenTelemetry Collectors to send fleet metrics to the vCluster observability Write Gateway.
sidebar_class_name: pro
---

import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
import Flow, { Step } from "@site/src/components/Flow";
import Button from "@site/src/components/Button";
import Label from "@site/src/components/Label";

Edge collectors are OpenTelemetry Collectors that scrape or receive metrics near the
workloads, then send OTLP to the Write Gateway. Each collector request must include a bearer
token and scope headers that match a `metrics-writer` access key.

:::tip Deploy collectors with Argo CD
If you use the Argo CD integration, you can deploy the collectors from a bundled template
instead of configuring them by hand. See
[Deploy Fleet Observability with Argo CD](./deploy-with-argocd.mdx). This manual path is the
only option for vCluster Standalone tenant clusters, which can't use the Argo CD
integration.
Deploy collectors once the gateway is running and a `metrics-writer` key is in place, as
described in [Install the observability gateway](./install-observability-gateway.mdx) and
[Configure metrics access](./configure-metrics-access.mdx).

## Deploy collectors with Argo CD

Collectors run on every tenant cluster and control plane cluster you want metrics from.
If you use the [Argo CD integration](../../integrations/argocd/overview.mdx), deploy one
`cluster-collector` `ArgoCDApplication` to each cluster instead of configuring the
OpenTelemetry Collector by hand.

:::note vCluster Standalone tenant clusters
The out-of-the-box collector path relies on the Argo CD integration, which vCluster
Standalone tenant clusters can't use. There is no bundled collector for them. Configure a
collector manually instead, as described below.
:::

## Gateway endpoints
Each collector authenticates to the gateway with a `metrics-writer` bearer token. Create a
Secret named `otel-otlp-auth` in the collector namespace before you deploy, or pass the
token through the `bearerToken` parameter.

```yaml title="otel-otlp-auth.yaml"
apiVersion: v1
kind: Secret
metadata:
name: otel-otlp-auth
namespace: observability
type: Opaque
stringData:
authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY"
```

Create the writer key first, as described in
[Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster)
or [Create a writer key for a control plane cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-control-plane-cluster),
then deploy the collectors. The walkthrough below deploys to a tenant cluster. For a
control plane cluster, swap `destination.cluster` in for `destination.virtualCluster`, as
described at the end of this section.

<Tabs
defaultValue="ui"
values={[
{ label: "Platform UI", value: "ui" },
{ label: "YAML", value: "yaml" },
]}
>
<TabItem value="yaml">

Create the `ArgoCDApplication` object in your project namespace:

```yaml title="fleet-edge-collectors.yaml"
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-edge-collectors
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet edge collectors"
destination:
virtualCluster:
name: my-vcluster # replace with your tenant cluster name
target: vCluster # deploy into the tenant cluster
templateRef:
name: cluster-collector
parameters:
destinationNamespace: observability
otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318
project: default
```

</TabItem>
<TabItem value="ui">

<Flow>
<Step>
Select your project from the projects dropdown at the top of the left navigation bar.
</Step>
<Step>
Navigate to <Label>Tenant Clusters</Label> and open the tenant cluster you want to collect from.
</Step>
<Step>
Select the <Label>ArgoCD Apps</Label> tab and click <Button>Add Argo CD Application</Button>.
</Step>
<Step>
In the <Label>Basics</Label> section, enter a <Label>Display Name</Label> such as `Fleet edge collectors`.
</Step>
<Step>
Select <Label>Deploy to vCluster</Label> to deploy into the tenant cluster itself.
</Step>
<Step>
From the <Label>Argo CD Application Template</Label> dropdown, select the <Label>Fleet Observability: Cluster Collector</Label> template.
</Step>
<Step>
Fill the <Label>Template Parameters</Label>. Set <Label>destinationNamespace</Label> to `observability` and <Label>otlpEndpoint</Label> to the Write Gateway endpoint. Use the in-cluster Service endpoint for collectors in the Platform cluster, or the gateway LoadBalancer address for collectors in a tenant cluster elsewhere.
</Step>
<Step>
Click <Button>Save</Button> to add the Argo CD Application to the tenant cluster.
</Step>
</Flow>

</TabItem>
</Tabs>

| Parameter | Default | Description |
|---|---|---|
| `destinationNamespace` | `observability` | Namespace for the collectors. |
| `project` | `default` | Argo CD project that owns the application. |
| `otlpEndpoint` | Fleet Write Gateway Service | OTLP HTTP endpoint the collectors push to. Use the in-cluster Service endpoint (`:4318`) for collectors in the Platform cluster, or the gateway LoadBalancer address (`https://<loadbalancer-address>:4318`) for collectors elsewhere. See [Expose the gateway with a LoadBalancer](./install-observability-gateway.mdx#expose-the-gateway-with-a-loadbalancer). |
| `bearerToken` | _(empty)_ | `metrics-writer` bearer token. When empty, the collectors read the `otel-otlp-auth` Secret. |
| `otlpCaData` | _(empty)_ | PEM CA data used to verify the gateway serving certificate. |
| `otlpInsecureSkipVerify` | `false` | Skip gateway TLS verification. Use only for test gateways. |

The collectors stamp the request scope from the destination, so target a tenant cluster
for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set
`destination.cluster` instead of `destination.virtualCluster` to target a control plane
cluster. Don't set `vcluster_platform_*` labels in collector configuration; the Write
Gateway owns that namespace. See [Protect platform labels](#protect-platform-labels) for
the label rules, and [Gateway endpoints](#gateway-endpoints) below for the scope headers
the template sets on your behalf.

On a cluster with the GPU stack deployed, this template also scrapes GPU metrics without
extra configuration. See
[The cluster-collector template scrapes GPU metrics automatically](./gpu-observability-templates.mdx#the-cluster-collector-template-scrapes-gpu-metrics-automatically).

## Configure collectors manually

To configure a collector yourself instead of using the bundled template, or to customize
behavior the template doesn't expose as a parameter, wire it up directly.

### Gateway endpoints

Inside the Platform cluster, the fleet gateway Service is:

Expand Down Expand Up @@ -52,7 +178,7 @@ endpoint instead of the Service name, for example
to enable it and retrieve the address. The exporter `endpoint` and TLS settings in the
examples below are otherwise unchanged; only the host differs.

## Store the writer key
### Store the writer key

Create a Secret for the collector, using a `metrics-writer` access key from
[Create a writer key for a tenant cluster](./configure-metrics-access.mdx#create-a-writer-key-for-a-tenant-cluster)
Expand All @@ -72,7 +198,7 @@ stringData:
Mount or project this Secret into your collector pod and expose it as an environment
variable or file.

## Tenant cluster collector
### Tenant cluster collector

Use tenant scope headers when a collector writes metrics for a specific tenant cluster.

Expand All @@ -97,7 +223,7 @@ service:

The `metrics-writer` key must include a matching `spec.scope.virtualClusters` entry.

## Control plane cluster collector
### Control plane cluster collector

Use cluster scope headers when a collector writes metrics for a control plane cluster.

Expand All @@ -121,7 +247,7 @@ service:

The `metrics-writer` key must include a matching `spec.scope.clusters` entry.

## Scrape custom targets
### Scrape custom targets

Use the collector's `prometheus` receiver to scrape exporters such as kubelet,
cAdvisor, DCGM exporter, or application endpoints. The gateway doesn't require a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ title: Configure metrics access
sidebar_label: Configure access
sidebar_position: 4
description: Create metrics-writer and metrics-reader access keys for the Fleet Observability gateway.
sidebar_class_name: pro
---

import InterpolatedCodeBlock from "@site/src/components/InterpolatedCodeBlock";
Expand Down Expand Up @@ -178,7 +177,7 @@ recommended for collector configurations.

:::note Argo CD collectors set these headers for you
When you deploy collectors with the `cluster-collector` template as described in
[Deploy collectors to each cluster](./deploy-with-argocd.mdx#deploy-collectors-to-each-cluster), the template sets
[Deploy collectors with Argo CD](./configure-edge-collectors.mdx#deploy-collectors-with-argo-cd), the template sets
these scope headers automatically from the deployment destination. You only set them by hand
when you configure a collector manually.
:::
Expand Down
Loading
Loading