Skip to content

Commit 7fa5d82

Browse files
loft-botloft-bot
authored
[v4.9.0] docs(doc-1415): add maintenance/logging section (#2133) (#2189)
* Backport: Copy platform/configure/platform-configs/audit.mdx to platform_versioned_docs/version-4.9.0/configure/platform-configs/audit.mdx * Backport: Copy platform/maintenance/logging/_category_.json to platform_versioned_docs/version-4.9.0/maintenance/logging/_category_.json * Backport: Copy platform/maintenance/logging/overview.mdx to platform_versioned_docs/version-4.9.0/maintenance/logging/overview.mdx * Backport: Copy platform/maintenance/logging/platform-logging.mdx to platform_versioned_docs/version-4.9.0/maintenance/logging/platform-logging.mdx * Backport: Copy platform/maintenance/monitoring/metrics.mdx to platform_versioned_docs/version-4.9.0/maintenance/monitoring/metrics.mdx * Backport: Copy platform/maintenance/monitoring/overview.mdx to platform_versioned_docs/version-4.9.0/maintenance/monitoring/overview.mdx
1 parent d36a0ac commit 7fa5d82

6 files changed

Lines changed: 236 additions & 80 deletions

File tree

‎platform_versioned_docs/version-4.9.0/configure/platform-configs/audit.mdx‎

Lines changed: 26 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,16 @@ Auditing in the platform is similar to [auditing Kubernetes clusters](https://ku
2727

2828
## Enable auditing
2929

30-
:::warning Two audit fields
31-
There is a field named as `audit` in `values.yaml` of `vcluster-platform` chart and a same field under `config` section, ie, `config.audit`. The former
32-
is to configure the audit log settings for the platform pod container. Enable this to record logs. The `config.audit` enables Audit Log UI feature in vCluster Platform UI.
33-
Thus, make sure the audit logs is being recorded by setting `audit: true` in `values.yaml` first.
30+
:::info Two separate audit fields
31+
`values.yaml` has two distinct `audit` fields with different purposes:
32+
33+
- **`audit`** (top-level Helm value) — configures audit log recording and persistence for the platform pod. This is a Helm-only setting. Changes require a Helm upgrade and a platform restart.
34+
- **`config.audit`** (under the `config` section) — controls audit policy and enables the Audit Log UI. This can be set in **Admin > Config** in the UI or under `config` in `values.yaml`.
35+
36+
`audit.enabled: true` must be set at the top level first. Without it, no audit events are recorded regardless of `config.audit` settings.
3437
:::
3538

36-
Configure auditing through the platform config in the platform UI (`Admin -> Config`). The content added in the UI will automatically be under the `config` section of the `values.yaml`.
39+
Configure auditing through the platform config in the platform UI (**Admin > Config**). Content added in the UI goes automatically under the `config` section of `values.yaml`.
3740

3841
An example configuration could look like:
3942

@@ -43,7 +46,9 @@ audit:
4346
level: 1
4447
```
4548
46-
Changing the platform auditing configuration requires a restart to take effect. Restart the platform either through the platform UI or via kubectl: `kubectl rollout restart deploy/loft -n loft`.
49+
:::warning
50+
Changing `config.audit` settings requires a platform restart to take effect. Restart through **Admin > Config** in the UI or with `kubectl rollout restart deploy/loft -n loft`.
51+
:::
4752

4853
Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.
4954

@@ -136,18 +141,18 @@ audit:
136141
# size: 30Gi
137142
```
138143

139-
:::warning
140-
The `audit` here is the `audit` in `values.yaml`, not the `audit` under `config` section of `values.yaml`. It controls whether the
141-
audit log is recorded and saved.
144+
:::note
145+
This is the top-level `audit` field in `values.yaml`, not `config.audit`. See [Enable auditing](#enable-auditing) for the distinction.
142146
:::
143147

144148
Apply the values using Helm:
145149

146150
```bash title="Apply Helm values"
147-
Helm upgrade vcluster-platform vcluster-platform -n vcluster-platform --version 4.0.0-alpha.12 \
148-
--repo https://charts.loft.sh \
149-
--reuse-values \
150-
-f values.yaml
151+
helm upgrade vcluster-platform vcluster-platform \
152+
--namespace vcluster-platform \
153+
--repo https://charts.loft.sh \
154+
--reuse-values \
155+
-f values.yaml
151156
```
152157

153158
### Use a persistent database as platform audit backend
@@ -191,21 +196,18 @@ audit:
191196
enableSideCar: true
192197
```
193198

194-
:::warning
195-
The `audit` here is the `audit` in `values.yaml`, not the `audit` under `config` section of `values.yaml`. It controls whether the
196-
audit log is recorded and saved.
197-
198-
Thus, you cannot configure this under `Admin > Config`, since this requires a change in the platform deployment itself, which is why this is a Helm option only
199+
:::note
200+
This is the top-level `audit` field in `values.yaml`, not `config.audit`. It cannot be set in **Admin > Config**. See [Enable auditing](#enable-auditing) for the distinction.
199201
:::
200202

201203
Update the Helm release:
202204

203-
```bash title="Update helm release"
204-
helm upgrade vcluster-platform vcluster-platform --namespace vcluster-platform \
205-
--repo https://charts.loft.sh \
206-
--version 4.0.0-alpha.12 \
207-
--reuse-values \
208-
--values values.yaml
205+
```bash title="Update Helm release"
206+
helm upgrade vcluster-platform vcluster-platform \
207+
--namespace vcluster-platform \
208+
--repo https://charts.loft.sh \
209+
--reuse-values \
210+
--values values.yaml
209211
```
210212

211213
Wait until the platform has restarted, then view the audit logs:

‎platform_versioned_docs/version-4.9.0/maintenance/logging/_category_.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"label": "Logging",
3+
"position": 3.5,
4+
"collapsible": true,
5+
"collapsed": true
6+
}

‎platform_versioned_docs/version-4.9.0/maintenance/logging/overview.mdx‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
title: Logging
3+
sidebar_label: Overview
4+
sidebar_position: 1
5+
description: Understand platform audit logs, platform process logs, and workload log collection in vCluster Platform.
6+
---
7+
8+
import Icon from '@site/src/components/Icon';
9+
10+
vCluster Platform surfaces three distinct types of logs, each serving a different purpose and requiring different configuration.
11+
12+
| Type | What it captures | Configure |
13+
|------|-----------------|-----------|
14+
| [Platform audit logs](#platform-audit-logs) | API requests that pass through the platform gateway, including who acted and when | [Audit Logging](../../configure/platform-configs/audit.mdx) |
15+
| [Platform process logs](#platform-process-logs) | Operational output from the platform pod, including startup, controller activity, and errors | [Platform Process Logging](./platform-logging.mdx) |
16+
| [Workload logs](#workload-logs) | Output from applications running inside tenant clusters | [Central HostPath Mapper](../monitoring/central-hostpath-mapper.mdx) |
17+
18+
These three types are independent. Enabling audit logging does not affect process log verbosity. A separate infrastructure component handles workload log collection.
19+
20+
## Platform audit logs
21+
22+
The platform's API gateway records every request that passes through it. This covers requests from the UI, the CLI, and any API call routed through the platform proxy. Each log entry captures:
23+
24+
- The requesting user (or service account), their groups, and source IP
25+
- The resource being accessed (type, name, namespace, cluster)
26+
- The HTTP verb (`get`, `list`, `create`, `update`, `delete`, `patch`)
27+
- Whether impersonation was in use
28+
- Timestamps and a unique audit ID
29+
30+
### What gets logged
31+
32+
Four audit levels control the volume and detail of what the platform captures.
33+
34+
| Level | Writes (create/update/delete/patch) | Reads (get/list/watch) | Request body | Response body |
35+
|-------|-------------------------------------|------------------------|:------------:|:-------------:|
36+
| 1 | Metadata only | Not logged | <Icon type="cross" /> | <Icon type="cross" /> |
37+
| 2 | Metadata only | Metadata only | <Icon type="cross" /> | <Icon type="cross" /> |
38+
| 3 | Logged with request body | Logged with request body | <Icon type="check" /> | <Icon type="cross" /> |
39+
| 4 | Logged with request and response | Logged with request and response | <Icon type="check" /> | <Icon type="check" /> |
40+
41+
At all levels, the following are always excluded from audit logs:
42+
43+
- Internal access-review requests (`selfsubjectaccessreviews`, `subjectaccessreviews`)
44+
- Agent coordination resources (`agentauditevents`, `directclusterendpointtokens`, `ingressauthtokens`)
45+
- Platform token resources (`licensetokens`)
46+
47+
The platform records long-running streaming operations (`pods/log`, `pods/exec`, `pods/portforward`) at metadata level only, regardless of the configured level. This prevents unbounded log growth.
48+
49+
The platform limits sensitive resources (`sharedsecrets`, `ownedaccesskeys`, `resetaccesskeys`, `users/profile`) to metadata level. This prevents credential material from appearing in logs.
50+
51+
### Configure audit logging
52+
53+
See [Audit Logging](../../configure/platform-configs/audit.mdx) for how to enable audit logging, choose a level, define a custom policy, and persist logs to a PVC or external database.
54+
55+
## Platform process logs
56+
57+
The platform pod writes operational logs to stdout. These cover initialization, controller reconciliation, API request handling, and runtime errors.
58+
59+
The platform supports three log levels.
60+
61+
| Level | What it captures | When to use |
62+
|-------|-----------------|-------------|
63+
| `info` | Startup sequences, controller events, handled requests | Default for all environments |
64+
| `debug` | Full operation context for every request and reconciliation loop | Temporarily, when diagnosing an issue |
65+
| `error` | Failures and warnings only | Stable production environments with low noise requirements |
66+
67+
By default, logs use a human-readable console format. Switch to JSON format when feeding logs into an aggregation pipeline such as Elasticsearch or Grafana Loki.
68+
69+
Access platform logs with:
70+
71+
```bash
72+
kubectl logs -n vcluster-platform -l app=loft -f
73+
```
74+
75+
See [Platform Process Logging](./platform-logging.mdx) for how to change log level and output format.
76+
77+
## Workload logs
78+
79+
Standard log collectors such as Fluentd, the ELK stack, and Grafana Loki read container logs from the host filesystem at paths like:
80+
81+
```
82+
/var/log/pods/<namespace>_<pod-name>_<uid>/<container>/
83+
```
84+
85+
Tenant clusters introduce a naming problem. When vCluster syncs pods to the control plane cluster, it rewrites their names. A pod named `web-6d4f9b` inside a tenant cluster appears as `web-6d4f9b-x-default-x-my-vcluster` on the control plane cluster. Log collectors see only the physical name and cannot correlate entries back to the originating tenant cluster workload.
86+
87+
The HostPath Mapper solves this by creating symlinks on each node that map physical pod paths to their virtual names. Log collectors follow the symlinks and see the names they expect.
88+
89+
vCluster Platform offers two deployment options:
90+
91+
- **Central HostPath Mapper**: A single DaemonSet on the control plane cluster handles path remapping for all tenant clusters. Recommended for fleet deployments.
92+
- **Per-cluster HostPath Mapper**: A DaemonSet installed in each tenant cluster's namespace. Better suited to small environments or where per-cluster isolation is needed.
93+
94+
See [Central HostPath Mapper](../monitoring/central-hostpath-mapper.mdx) for setup instructions for both options.

‎platform_versioned_docs/version-4.9.0/maintenance/logging/platform-logging.mdx‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
title: Platform Process Logging
3+
sidebar_label: Platform Process Logging
4+
sidebar_position: 2
5+
description: Configure the log level and output format for the vCluster Platform pod.
6+
---
7+
8+
import Tabs from "@theme/Tabs";
9+
import TabItem from "@theme/TabItem";
10+
11+
vCluster Platform writes operational logs to `stdout` from the platform pod. These logs cover initialization, controller reconciliation, API request handling, and runtime errors. Check these logs first when diagnosing unexpected platform behavior.
12+
13+
By default, logs use a human-readable console format at `info` level.
14+
15+
## Log levels
16+
17+
Platform logging supports three log levels:
18+
- `info`
19+
- `debug`
20+
- `error`
21+
22+
For a description of each level, see [Platform process logs](./overview.mdx#platform-process-logs).
23+
24+
:::warning
25+
Debug logging significantly increases log volume. Enable it only while reproducing an issue, then return to `info`.
26+
:::
27+
28+
To change the log level, upgrade the Helm release with the `logging.level` value:
29+
30+
```bash
31+
helm upgrade vcluster-platform vcluster-platform \
32+
--repo https://charts.loft.sh \
33+
--namespace vcluster-platform \
34+
--reuse-values \
35+
--set logging.level=debug
36+
```
37+
38+
## Log format
39+
40+
vCluster Platform supports two output formats.
41+
42+
<Tabs
43+
defaultValue="console"
44+
values={[
45+
{ label: "Console", value: "console" },
46+
{ label: "JSON", value: "json" },
47+
]}
48+
>
49+
<TabItem value="console">
50+
51+
The default console format produces human-readable output suited to manual log review:
52+
53+
```text
54+
2023-07-11 09:20:56 INFO controller-runtime.metrics metrics/listener.go:44 Metrics server is starting to listen {"component": "loft", "addr": "127.0.0.1:12000"}
55+
2023-07-11 09:20:56 INFO initialize/context.go:68 Initialize... {"component": "loft"}
56+
2023-07-11 09:20:56 INFO initialize/context.go:72 Ensure certificates... {"component": "loft"}
57+
```
58+
59+
</TabItem>
60+
<TabItem value="json">
61+
62+
The JSON format produces structured output suited to log aggregation systems such as Elasticsearch, Splunk, or Grafana Loki:
63+
64+
```json
65+
{"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"}
66+
{"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"}
67+
{"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"}
68+
```
69+
70+
Each entry contains at minimum:
71+
72+
| Field | Type | Description |
73+
|-------|------|-------------|
74+
| `level` | string | Log level (`info`, `warn`, `error`, `debug`) |
75+
| `ts` | number | Unix timestamp (seconds since epoch) |
76+
| `caller` | string | Source file and line number |
77+
| `msg` | string | Log message |
78+
| `component` | string | Platform component that emitted the log |
79+
80+
Additional fields vary by subsystem and operation type. For example, controller events may include `namespace` and `name` while API handler logs may include `method` and `status`.
81+
82+
To switch to JSON format:
83+
84+
```bash
85+
helm upgrade vcluster-platform vcluster-platform \
86+
--repo https://charts.loft.sh \
87+
--namespace vcluster-platform \
88+
--reuse-values \
89+
--set logging.encoding=json
90+
```
91+
92+
</TabItem>
93+
</Tabs>
94+
95+
## View logs
96+
97+
```bash
98+
kubectl logs -n vcluster-platform -l app=loft -f
99+
```
100+
101+
In high-availability deployments, multiple platform pods run concurrently. Pipe through a log aggregator, or target a specific pod, to avoid interleaved output from multiple replicas.

‎platform_versioned_docs/version-4.9.0/maintenance/monitoring/metrics.mdx‎

Lines changed: 6 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,62 +1,13 @@
11
---
2-
title: Metrics And Monitoring
3-
sidebar_label: Metrics And Monitoring
2+
title: Metrics
3+
sidebar_label: Metrics
44
sidebar_position: 4
5+
description: Prometheus-conformant metrics exposed by vCluster Platform, including how to configure scraping with a ServiceMonitor.
56
---
67

7-
## Configure logging
8-
9-
By default, vCluster Platform prints out structured logs to the console with a log level of `info`.
10-
11-
These logs include constant log messages, a human-readable timestamp, log levels, the file and line number of the source caller, and with variables in the body:
12-
13-
```text
14-
2023-07-11 09:20:56 INFO controller-runtime.metrics metrics/listener.go:44 Metrics server is starting to listen {"component": "loft", "addr": "127.0.0.1:12000"}
15-
2023-07-11 09:20:56 INFO initialize/context.go:68 Initialize... {"component": "loft"}
16-
2023-07-11 09:20:56 INFO initialize/context.go:72 Ensure certificates... {"component": "loft"}
17-
2023-07-11 09:21:02 INFO initialize/context.go:80 Ensure crds... {"component": "loft"}
18-
```
19-
20-
You can change the log level by setting the Helm value `logging.level` to one of the following values: `debug`, `info`, `error`, and upgrading your vCluster Platform release with the following command:
21-
22-
```bash
23-
helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \
24-
--namespace vcluster-platform \
25-
--reuse-values \
26-
--set logging.level=error # or debug or info
27-
```
28-
29-
### Use JSON output encoding
30-
31-
In addition to the console output, vCluster Platform also supports printing logs encoded in JSON format.
32-
33-
This output format is helpful for structured logging, which allows you to search and filter logs easily.
34-
35-
You can change the log output by setting the Helm value `logging.encoding` to one of the following values: `json`, and upgrading your vCluster Platform release with the following command:
36-
37-
```bash
38-
helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \
39-
--namespace vcluster-platform \
40-
--reuse-values \
41-
--set logging.encoding=json
42-
```
43-
44-
These JSON-encoded logs include constant log messages with different variables in the body, like the default console output:
45-
46-
```json
47-
{"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"}
48-
{"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"}
49-
{"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"}
50-
{"level":"info","ts":1689067389.678155,"caller":"initialize/context.go:80","msg":"Ensure crds...","component":"loft"}
51-
```
52-
53-
The JSON encoded logs contain at least the following fields:
54-
55-
- `level`: The log level of the message
56-
- `ts`: The timestamp of the message (floating-point number of seconds since the Unix epoch)
57-
- `caller`: The file and line number of the message
58-
- `msg`: The message itself
59-
- `component`: The vCluster Platform component that emitted the message
8+
:::note Platform process logging
9+
To configure log level or output format for the platform pod, see [Platform Process Logging](../logging/platform-logging.mdx).
10+
:::
6011

6112
## Metrics
6213

‎platform_versioned_docs/version-4.9.0/maintenance/monitoring/overview.mdx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,10 +44,12 @@ For aggregating workload metrics across multiple tenant clusters, use the OpenTe
4444

4545
vCluster Platform exposes [Prometheus-conformant metrics](./metrics.mdx) from its internal components, including the API gateway, integrated Kubernetes API server, controller manager, and Go runtime. These metrics cover request counts, latency, and error rates for all platform operations.
4646

47-
Use a [Prometheus `ServiceMonitor`](./metrics.mdx#create-the-prometheus-servicemonitor) to scrape these metrics automatically, or [access the `/metrics` endpoint directly](./metrics.mdx#access-the-metrics-endpoint-directly-without-a-servicemonitor). You can also [configure Platform log levels and JSON encoding](./metrics.mdx#configure-logging) from the same page.
47+
Use a [Prometheus `ServiceMonitor`](./metrics.mdx#create-the-prometheus-servicemonitor) to scrape these metrics automatically, or [access the `/metrics` endpoint directly](./metrics.mdx#access-the-metrics-endpoint-directly-without-a-servicemonitor). For platform pod log level and output format, see [Platform Process Logging](../logging/platform-logging.mdx).
4848

4949
## Log collection
5050

5151
To collect logs from workloads running inside tenant clusters, use the Central HostPath Mapper. It installs a single DaemonSet on the Control Plane Cluster. The DaemonSet handles log path remapping for all tenant clusters and removes the need for per-cluster logging agents.
5252

5353
- [Central HostPath Mapper](./central-hostpath-mapper)
54+
55+
For platform audit logging and platform process log configuration, see [Logging](../logging/overview.mdx).

0 commit comments

Comments
 (0)