From fc38ca8c00150f328877001a60b4142fa0a73472 Mon Sep 17 00:00:00 2001 From: Daryl White Date: Fri, 15 May 2026 17:00:30 -0400 Subject: [PATCH 1/4] docs(doc-1415): add maintenance/logging section with audit and process log overview Create platform/maintenance/logging/ as a new section alongside monitoring. Add overview.mdx explaining all three platform logging types (audit logs, process logs, workload logs) with per-level audit breakdown table and HostPath Mapper deployment options. Add platform-logging.mdx covering log level and format configuration, extracted from metrics.mdx. Trim metrics.mdx to metrics only and add a callout linking to the new logging section. Update monitoring/overview.mdx to point to logging section for platform process log configuration. Co-Authored-By: Claude Sonnet 4.6 --- platform/maintenance/logging/_category_.json | 6 ++ platform/maintenance/logging/overview.mdx | 94 ++++++++++++++++ .../maintenance/logging/platform-logging.mdx | 101 ++++++++++++++++++ platform/maintenance/monitoring/metrics.mdx | 61 ++--------- platform/maintenance/monitoring/overview.mdx | 2 +- 5 files changed, 208 insertions(+), 56 deletions(-) create mode 100644 platform/maintenance/logging/_category_.json create mode 100644 platform/maintenance/logging/overview.mdx create mode 100644 platform/maintenance/logging/platform-logging.mdx diff --git a/platform/maintenance/logging/_category_.json b/platform/maintenance/logging/_category_.json new file mode 100644 index 000000000..b5bc52e59 --- /dev/null +++ b/platform/maintenance/logging/_category_.json @@ -0,0 +1,6 @@ +{ + "label": "Logging", + "position": 3.5, + "collapsible": true, + "collapsed": true +} diff --git a/platform/maintenance/logging/overview.mdx b/platform/maintenance/logging/overview.mdx new file mode 100644 index 000000000..0953d2c5e --- /dev/null +++ b/platform/maintenance/logging/overview.mdx @@ -0,0 +1,94 @@ +--- +title: Logging +sidebar_label: Overview +sidebar_position: 1 +description: Understand platform audit logs, platform process logs, and workload log collection in vCluster Platform. +--- + +import Icon from '@site/src/components/Icon'; + +vCluster Platform surfaces three distinct types of logs, each serving a different purpose and requiring different configuration. + +| Type | What it captures | Configure | +|------|-----------------|-----------| +| [Platform audit logs](#platform-audit-logs) | API requests that pass through the platform gateway, including who acted and when | [Audit Logging](../../configure/platform-configs/audit.mdx) | +| [Platform process logs](#platform-process-logs) | Operational output from the platform pod, including startup, controller activity, and errors | [Platform Process Logging](./platform-logging.mdx) | +| [Workload logs](#workload-logs) | Output from applications running inside tenant clusters | [Central HostPath Mapper](../monitoring/central-hostpath-mapper.mdx) | + +These three types are independent. Enabling audit logging does not affect process log verbosity. A separate infrastructure component handles workload log collection. + +## Platform audit logs + +The platform's API gateway processes every API request and records it in the audit log. This includes requests from the UI, the CLI, and direct API access. Each log entry captures: + +- The requesting user (or service account), their groups, and source IP +- The resource being accessed (type, name, namespace, cluster) +- The HTTP verb (`get`, `list`, `create`, `update`, `delete`, `patch`) +- Whether impersonation was in use +- Timestamps and a unique audit ID + +### What gets logged + +Four audit levels control the volume and detail of what the platform captures. + +| Level | Writes (create/update/delete/patch) | Reads (get/list/watch) | Request body | Response body | +|-------|-------------------------------------|------------------------|:------------:|:-------------:| +| 1 | Metadata only | Not logged | | | +| 2 | Metadata only | Metadata only | | | +| 3 | Logged with request body | Logged with request body | | | +| 4 | Logged with request and response | Logged with request and response | | | + +At all levels, the following are always excluded from audit logs: + +- Internal access-review requests (`selfsubjectaccessreviews`, `subjectaccessreviews`) +- Agent coordination resources (`agentauditevents`, `directclusterendpointtokens`, `ingressauthtokens`) +- Platform token resources (`licensetokens`) + +The platform records long-running streaming operations (`pods/log`, `pods/exec`, `pods/portforward`) at metadata level only, regardless of the configured level. This prevents unbounded log growth. + +The platform limits sensitive resources (`sharedsecrets`, `ownedaccesskeys`, `resetaccesskeys`, `users/profile`) to metadata level. This prevents credential material from appearing in logs. + +### Configure audit logging + +See [Audit Logging](../../configure/platform-configs/audit.mdx) for how to enable audit logging, choose a level, define a custom policy, and persist logs to a PVC or external database. + +## Platform process logs + +The platform pod writes operational logs to stdout. These cover initialization, controller reconciliation, API request handling, and runtime errors. + +The platform supports three log levels. + +| Level | What it captures | When to use | +|-------|-----------------|-------------| +| `info` | Startup sequences, controller events, handled requests | Default for all environments | +| `debug` | Full operation context for every request and reconciliation loop | Temporarily, when diagnosing an issue | +| `error` | Failures and warnings only | Stable production environments with low noise requirements | + +By default, logs use a human-readable console format. Switch to JSON format when feeding logs into an aggregation pipeline such as Elasticsearch or Grafana Loki. + +Access platform logs with: + +```bash +kubectl logs -n vcluster-platform -l app=loft -f +``` + +See [Platform Process Logging](./platform-logging.mdx) for how to change log level and output format. + +## Workload logs + +Standard log collectors such as Fluentd, the ELK stack, and Grafana Loki read container logs from the host filesystem at paths like: + +``` +/var/log/pods/__// +``` + +Tenant clusters introduce a naming problem. When vCluster syncs pods to the control plane cluster, it rewrites their names. A pod named `web-6d4f9b` inside a tenant cluster appears as `web-6d4f9b-x-default-x-my-vcluster` on the control plane cluster. Log collectors see only the physical name and cannot correlate entries back to the originating tenant cluster workload. + +The HostPath Mapper solves this by creating symlinks on each node that map physical pod paths to their virtual names. Log collectors follow the symlinks and see the names they expect. + +vCluster Platform offers two deployment options: + +- **Central HostPath Mapper**: A single DaemonSet on the control plane cluster handles path remapping for all tenant clusters. Recommended for fleet deployments. +- **Per-cluster HostPath Mapper**: A DaemonSet installed in each tenant cluster's namespace. Better suited to small environments or where per-cluster isolation is needed. + +See [Central HostPath Mapper](../monitoring/central-hostpath-mapper.mdx) for setup instructions for both options. diff --git a/platform/maintenance/logging/platform-logging.mdx b/platform/maintenance/logging/platform-logging.mdx new file mode 100644 index 000000000..65d6050fd --- /dev/null +++ b/platform/maintenance/logging/platform-logging.mdx @@ -0,0 +1,101 @@ +--- +title: Platform Process Logging +sidebar_label: Platform Process Logging +sidebar_position: 2 +description: Configure the log level and output format for the vCluster Platform pod. +--- + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; + +vCluster Platform writes operational logs to `stdout` from the platform pod. These logs cover initialization, controller reconciliation, API request handling, and runtime errors. Check these logs first when diagnosing unexpected platform behavior. + +By default, logs use a human-readable console format at `info` level. + +## Log levels + +Platform logging supports three log levels: +- `info` +- `debug` +- `error` + +For a description of each level, see [Platform process logs](./overview.mdx#platform-process-logs). + +:::warning +Debug logging significantly increases log volume. Enable it only while reproducing an issue, then return to `info`. +::: + +To change the log level, upgrade the Helm release with the `logging.level` value: + +```bash +helm upgrade vcluster-platform vcluster-platform \ + --repo https://charts.loft.sh \ + --namespace vcluster-platform \ + --reuse-values \ + --set logging.level=debug +``` + +## Log format + +vCluster Platform supports two output formats. + + + + +The default console format produces human-readable output suited to manual log review: + +```text +2023-07-11 09:20:56 INFO controller-runtime.metrics metrics/listener.go:44 Metrics server is starting to listen {"component": "loft", "addr": "127.0.0.1:12000"} +2023-07-11 09:20:56 INFO initialize/context.go:68 Initialize... {"component": "loft"} +2023-07-11 09:20:56 INFO initialize/context.go:72 Ensure certificates... {"component": "loft"} +``` + + + + +The JSON format produces structured output suited to log aggregation systems such as Elasticsearch, Splunk, or Grafana Loki: + +```json +{"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"} +{"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"} +{"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"} +``` + +Each entry contains at minimum: + +| Field | Type | Description | +|-------|------|-------------| +| `level` | string | Log level (`info`, `warn`, `error`, `debug`) | +| `ts` | number | Unix timestamp (seconds since epoch) | +| `caller` | string | Source file and line number | +| `msg` | string | Log message | +| `component` | string | Platform component that emitted the log | + +Additional fields vary by subsystem and operation type. For example, controller events may include `namespace` and `name` while API handler logs may include `method` and `status`. + +To switch to JSON format: + +```bash +helm upgrade vcluster-platform vcluster-platform \ + --repo https://charts.loft.sh \ + --namespace vcluster-platform \ + --reuse-values \ + --set logging.encoding=json +``` + + + + +## View logs + +```bash +kubectl logs -n vcluster-platform -l app=loft -f +``` + +In high-availability deployments, multiple platform pods run concurrently. Pipe through a log aggregator, or target a specific pod, to avoid interleaved output from multiple replicas. diff --git a/platform/maintenance/monitoring/metrics.mdx b/platform/maintenance/monitoring/metrics.mdx index e67711736..5970f3778 100644 --- a/platform/maintenance/monitoring/metrics.mdx +++ b/platform/maintenance/monitoring/metrics.mdx @@ -1,62 +1,13 @@ --- -title: Metrics And Monitoring -sidebar_label: Metrics And Monitoring +title: Metrics +sidebar_label: Metrics sidebar_position: 4 +description: Prometheus-conformant metrics exposed by vCluster Platform, including how to configure scraping with a ServiceMonitor. --- -## Configure logging - -By default, vCluster Platform prints out structured logs to the console with a log level of `info`. - -These logs include constant log messages, a human-readable timestamp, log levels, the file and line number of the source caller, and with variables in the body: - -```text -2023-07-11 09:20:56 INFO controller-runtime.metrics metrics/listener.go:44 Metrics server is starting to listen {"component": "loft", "addr": "127.0.0.1:12000"} -2023-07-11 09:20:56 INFO initialize/context.go:68 Initialize... {"component": "loft"} -2023-07-11 09:20:56 INFO initialize/context.go:72 Ensure certificates... {"component": "loft"} -2023-07-11 09:21:02 INFO initialize/context.go:80 Ensure crds... {"component": "loft"} -``` - -You can change the log level by setting the Helm value `logging.level` to one of the following values: `debug`, `info`, `error`, and upgrading your vCluster Platform release with the following command: - -```bash -helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \ - --namespace vcluster-platform \ - --reuse-values \ - --set logging.level=error # or debug or info -``` - -### Use JSON output encoding - -In addition to the console output, vCluster Platform also supports printing logs encoded in JSON format. - -This output format is helpful for structured logging, which allows you to search and filter logs easily. - -You can change the log output by setting the Helm value `logging.encoding` to one of the following values: `json`, and upgrading your vCluster Platform release with the following command: - -```bash -helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \ - --namespace vcluster-platform \ - --reuse-values \ - --set logging.encoding=json -``` - -These JSON-encoded logs include constant log messages with different variables in the body, like the default console output: - -```json -{"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"} -{"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"} -{"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"} -{"level":"info","ts":1689067389.678155,"caller":"initialize/context.go:80","msg":"Ensure crds...","component":"loft"} -``` - -The JSON encoded logs contain at least the following fields: - -- `level`: The log level of the message -- `ts`: The timestamp of the message (floating-point number of seconds since the Unix epoch) -- `caller`: The file and line number of the message -- `msg`: The message itself -- `component`: The vCluster Platform component that emitted the message +:::note Platform process logging +To configure log level or output format for the platform pod, see [Platform Process Logging](../logging/platform-logging.mdx). +::: ## Metrics diff --git a/platform/maintenance/monitoring/overview.mdx b/platform/maintenance/monitoring/overview.mdx index 122724032..ec23266f8 100644 --- a/platform/maintenance/monitoring/overview.mdx +++ b/platform/maintenance/monitoring/overview.mdx @@ -44,7 +44,7 @@ For aggregating workload metrics across multiple tenant clusters, use the OpenTe vCluster Platform exposes [Prometheus-conformant metrics](./metrics.mdx) from its internal components, including the API gateway, integrated Kubernetes API server, controller manager, and Go runtime. These metrics cover request counts, latency, and error rates for all platform operations. -Use a [Prometheus `ServiceMonitor`](./metrics.mdx#create-the-prometheus-servicemonitor) to scrape these metrics automatically, or [access the `/metrics` endpoint directly](./metrics.mdx#access-the-metrics-endpoint-directly-without-a-servicemonitor). You can also [configure Platform log levels and JSON encoding](./metrics.mdx#configure-logging) from the same page. +Use a [Prometheus `ServiceMonitor`](./metrics.mdx#create-the-prometheus-servicemonitor) to scrape these metrics automatically, or [access the `/metrics` endpoint directly](./metrics.mdx#access-the-metrics-endpoint-directly-without-a-servicemonitor). For platform pod log level and output format, see [Platform Process Logging](../logging/platform-logging.mdx). ## Log collection From ba7d547ff281f144d8f793386c0a138006b40b80 Mon Sep 17 00:00:00 2001 From: Daryl White Date: Fri, 15 May 2026 17:11:24 -0400 Subject: [PATCH 2/4] docs(doc-1415): improve audit.mdx clarity and add logging cross-links Consolidate three repeated audit/config.audit warnings into one clear Info callout at the top of Enable auditing. Replace prose restart notice with a Warning admonition. Remove pinned Helm version 4.0.0-alpha.12 from PVC and sidecar commands. Simplify the two secondary notes to brief references back to the main explanation. Add audit logging reference to monitoring/overview.mdx Log collection section so users find the Logging section from the monitoring page. Co-Authored-By: Claude Sonnet 4.6 --- platform/configure/platform-configs/audit.mdx | 50 ++++++++++--------- platform/maintenance/monitoring/overview.mdx | 2 + 2 files changed, 28 insertions(+), 24 deletions(-) diff --git a/platform/configure/platform-configs/audit.mdx b/platform/configure/platform-configs/audit.mdx index 4ee749488..b9c450ca7 100644 --- a/platform/configure/platform-configs/audit.mdx +++ b/platform/configure/platform-configs/audit.mdx @@ -27,13 +27,16 @@ Auditing in the platform is similar to [auditing Kubernetes clusters](https://ku ## Enable auditing -:::warning Two audit fields -There is a field named as `audit` in `values.yaml` of `vcluster-platform` chart and a same field under `config` section, ie, `config.audit`. The former -is to configure the audit log settings for the platform pod container. Enable this to record logs. The `config.audit` enables Audit Log UI feature in vCluster Platform UI. -Thus, make sure the audit logs is being recorded by setting `audit: true` in `values.yaml` first. +:::info Two separate audit fields +`values.yaml` has two distinct `audit` fields with different purposes: + +- **`audit`** (top-level Helm value) — configures audit log recording and persistence for the platform pod. This is a Helm-only setting. Changes require a Helm upgrade and a platform restart. +- **`config.audit`** (under the `config` section) — controls audit policy and enables the Audit Log UI. This can be set in **Admin > Config** in the UI or under `config` in `values.yaml`. + +`audit.enabled: true` must be set at the top level first. Without it, no audit events are recorded regardless of `config.audit` settings. ::: -Configure auditing through the platform config in the platform UI (`Admin -> Config`). The content added in the UI will automatically be under the `config` section of the `values.yaml`. +Configure auditing through the platform config in the platform UI (**Admin > Config**). Content added in the UI goes automatically under the `config` section of `values.yaml`. An example configuration could look like: @@ -43,7 +46,9 @@ audit: level: 1 ``` -Changing the platform auditing configuration requires a restart to take effect. Restart the platform either through the platform UI or via kubectl: `kubectl rollout restart deploy/loft -n loft`. +:::warning +Changing `config.audit` settings requires a platform restart to take effect. Restart through **Admin > Config** in the UI or with `kubectl rollout restart deploy/loft -n loft`. +::: Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records. @@ -136,18 +141,18 @@ audit: # size: 30Gi ``` -:::warning -The `audit` here is the `audit` in `values.yaml`, not the `audit` under `config` section of `values.yaml`. It controls whether the -audit log is recorded and saved. +:::note +This is the top-level `audit` field in `values.yaml`, not `config.audit`. See [Enable auditing](#enable-auditing) for the distinction. ::: Apply the values using Helm: ```bash title="Apply Helm values" -Helm upgrade vcluster-platform vcluster-platform -n vcluster-platform --version 4.0.0-alpha.12 \ ---repo https://charts.loft.sh \ ---reuse-values \ --f values.yaml +helm upgrade vcluster-platform vcluster-platform \ + --namespace vcluster-platform \ + --repo https://charts.loft.sh \ + --reuse-values \ + -f values.yaml ``` ### Use a persistent database as platform audit backend @@ -191,21 +196,18 @@ audit: enableSideCar: true ``` -:::warning -The `audit` here is the `audit` in `values.yaml`, not the `audit` under `config` section of `values.yaml`. It controls whether the -audit log is recorded and saved. - -Thus, you cannot configure this under `Admin > Config`, since this requires a change in the platform deployment itself, which is why this is a Helm option only +:::note +This is the top-level `audit` field in `values.yaml`, not `config.audit`. It cannot be set in **Admin > Config**. See [Enable auditing](#enable-auditing) for the distinction. ::: Update the Helm release: -```bash title="Update helm release" -helm upgrade vcluster-platform vcluster-platform --namespace vcluster-platform \ - --repo https://charts.loft.sh \ - --version 4.0.0-alpha.12 \ - --reuse-values \ - --values values.yaml +```bash title="Update Helm release" +helm upgrade vcluster-platform vcluster-platform \ + --namespace vcluster-platform \ + --repo https://charts.loft.sh \ + --reuse-values \ + --values values.yaml ``` Wait until the platform has restarted, then view the audit logs: diff --git a/platform/maintenance/monitoring/overview.mdx b/platform/maintenance/monitoring/overview.mdx index ec23266f8..021603d55 100644 --- a/platform/maintenance/monitoring/overview.mdx +++ b/platform/maintenance/monitoring/overview.mdx @@ -51,3 +51,5 @@ Use a [Prometheus `ServiceMonitor`](./metrics.mdx#create-the-prometheus-servicem To collect logs from workloads running inside tenant clusters, use the Central HostPath Mapper. It installs a single DaemonSet on the Control Plane Cluster. The DaemonSet handles log path remapping for all tenant clusters and removes the need for per-cluster logging agents. - [Central HostPath Mapper](./central-hostpath-mapper) + +For platform audit logging and platform process log configuration, see [Logging](../logging/overview.mdx). From 7d93a754870381d6a603335ccd26aa507216f33b Mon Sep 17 00:00:00 2001 From: Daryl White Date: Mon, 18 May 2026 11:14:31 -0400 Subject: [PATCH 3/4] docs(doc-1415): clarify platform audit log coverage limitation Platform audit logs only capture requests routed through the platform gateway. Add a Coverage limitation section to make clear that direct tenant cluster access (e.g. AccessPoint) bypasses the gateway and is not captured, with a cross-link to tenant cluster audit logging. Co-Authored-By: Claude Sonnet 4.6 --- platform/maintenance/logging/overview.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/platform/maintenance/logging/overview.mdx b/platform/maintenance/logging/overview.mdx index 0953d2c5e..255d3d7da 100644 --- a/platform/maintenance/logging/overview.mdx +++ b/platform/maintenance/logging/overview.mdx @@ -19,7 +19,7 @@ These three types are independent. Enabling audit logging does not affect proces ## Platform audit logs -The platform's API gateway processes every API request and records it in the audit log. This includes requests from the UI, the CLI, and direct API access. Each log entry captures: +The platform's API gateway records every request that passes through it. This covers requests from the UI, the CLI, and any API call routed through the platform proxy. Each log entry captures: - The requesting user (or service account), their groups, and source IP - The resource being accessed (type, name, namespace, cluster) @@ -48,6 +48,12 @@ The platform records long-running streaming operations (`pods/log`, `pods/exec`, The platform limits sensitive resources (`sharedsecrets`, `ownedaccesskeys`, `resetaccesskeys`, `users/profile`) to metadata level. This prevents credential material from appearing in logs. +### Coverage limitation + +Platform audit logging only captures requests routed through the platform gateway. Requests that reach a tenant cluster directly (for example, when using the [AccessPoint](/docs/vcluster/key-features/ingress-access) ingress feature) bypass the platform and are not recorded here. + +To audit direct tenant cluster access, configure audit logging at the tenant cluster level. See [Control plane configuration](/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane#32-logging) for setup steps. + ### Configure audit logging See [Audit Logging](../../configure/platform-configs/audit.mdx) for how to enable audit logging, choose a level, define a custom policy, and persist logs to a PVC or external database. From d36471a235b52a31ec562fe72b79e865fb5c4fa4 Mon Sep 17 00:00:00 2001 From: Daryl White Date: Mon, 18 May 2026 11:59:11 -0400 Subject: [PATCH 4/4] docs(doc-1415): revert coverage limitation section Co-Authored-By: Claude Sonnet 4.6 --- platform/maintenance/logging/overview.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/platform/maintenance/logging/overview.mdx b/platform/maintenance/logging/overview.mdx index 255d3d7da..0ec66e080 100644 --- a/platform/maintenance/logging/overview.mdx +++ b/platform/maintenance/logging/overview.mdx @@ -48,12 +48,6 @@ The platform records long-running streaming operations (`pods/log`, `pods/exec`, The platform limits sensitive resources (`sharedsecrets`, `ownedaccesskeys`, `resetaccesskeys`, `users/profile`) to metadata level. This prevents credential material from appearing in logs. -### Coverage limitation - -Platform audit logging only captures requests routed through the platform gateway. Requests that reach a tenant cluster directly (for example, when using the [AccessPoint](/docs/vcluster/key-features/ingress-access) ingress feature) bypass the platform and are not recorded here. - -To audit direct tenant cluster access, configure audit logging at the tenant cluster level. See [Control plane configuration](/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane#32-logging) for setup steps. - ### Configure audit logging See [Audit Logging](../../configure/platform-configs/audit.mdx) for how to enable audit logging, choose a level, define a custom policy, and persist logs to a PVC or external database.