Add OpenTelemetry observability monitoring guides (#135)

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>

Author: ArnobKumarSaha

docs/platform/guides/client-organization/create-client-organization.md

@@ -93,7 +93,7 @@ In the final step, configure the telemetry settings that will be applied to the
 
 ![Create Client Organization wizard — Step 4, Telemetry Configuration with monitoring type, log retention, and metrics retention periods](.././images/telemetry-configuration.png)
 
-- **Select Monitoring Type** — choose the monitoring stack to provision for the organization.
+- **Select Monitoring Cluster** — the observability cluster that will store this organization's telemetry. Its `tenant-operator` provisions a `Tenant` on the cluster's TelemetryStack, isolating the organization's data within the shared stack. See [OpenTelemetry Monitoring](../../cluster-management/otel-monitoring.md) for how the monitoring cluster is set up.
 - **Logs** — set the **Retention Period** for collected logs.
 - **Metrics** — set the retention periods for collected metrics:
   - **Yearly Retention** — how long aggregated yearly metrics are kept.

docs/platform/guides/cluster-management/add-cluster/import-observability-cluster.md

@@ -0,0 +1,59 @@
+---
+layout: docs
+menu:
+  docsplatform_{{.version}}:
+    identifier: cluster-management-add-observability
+    name: Import Observability Cluster
+    parent: cluster-management-add
+    weight: 20
+menu_name: docsplatform_{{.version}}
+section_menu_id: guides
+---
+
+# Importing an Observability Cluster
+
+The Platform Console can run a dedicated **observability cluster** that collects
+telemetry (metrics, logs, and traces) from your spoke clusters through
+OpenTelemetry. This guide covers the two import-time tasks:
+
+1. Importing the observability cluster itself.
+2. Enabling the OpenTelemetry feature on the spoke clusters you want to monitor.
+
+For the full end-to-end setup — MinIO, the `telemetry` bucket, and the
+TelemetryStack — see [OpenTelemetry Monitoring](../../otel-monitoring.md).
+
+## Import the Observability Cluster
+
+Import the cluster as a spoke as you normally would (see
+[Adding a Cluster](overview.md)). During the import flow, select the
+**Observability Cluster** cluster profile. This provisions the components
+required to receive and store telemetry from other clusters.
+
+## Enable the OpenTelemetry Feature on a Spoke Cluster
+
+Once the observability cluster is ready, enable the OpenTelemetry feature on
+each spoke cluster you want it to monitor. There are two ways to do this.
+
+### Option A: During Cluster Import
+
+When importing a new cluster, select the destination observability cluster from
+the **Monitoring Cluster** drop-down in the import flow. This automatically
+enables the OpenTelemetry feature on the new spoke and routes its telemetry to
+the chosen monitoring cluster.
+
+![Add Cluster import form with the Monitoring Cluster drop-down set to the observability cluster](../../images/otel-monitoring/import-select-monitoring-cluster.png)
+
+### Option B: On an Existing Spoke Cluster
+
+1. Import the cluster as a spoke (if it is not already imported).
+2. Install the Perses dashboards on the cluster:
+
+   ```bash
+   helm upgrade -i kubedb-perses-dashboards \
+     oci://ghcr.io/appscode-charts/kubedb-perses-dashboards \
+     -n monitoring --create-namespace \
+     --version=v2026.4.27
+   ```
+
+3. Navigate to the cluster's **Observability** feature set.
+4. Enable the **`appscode-otel-stack`** feature.

docs/platform/guides/cluster-management/images/otel-monitoring/import-select-monitoring-cluster.png

@@ -0,0 +1,195 @@
+---
+layout: docs
+menu:
+  docsplatform_{{.version}}:
+    identifier: cluster-management-otel-monitoring
+    name: OpenTelemetry Monitoring
+    parent: cluster-management
+    weight: 90
+menu_name: docsplatform_{{.version}}
+section_menu_id: guides
+---
+
+# OpenTelemetry Monitoring
+
+This guide walks through setting up an observability stack that collects
+telemetry from your spoke clusters using OpenTelemetry, stores it in an
+object-storage bucket, and exposes it through a TelemetryStack.
+
+## Architecture
+
+The setup involves three kinds of clusters:
+
+- **Hub cluster** — the central Platform Console (ACE). This is where you
+  operate clusters, manage organizations, and view dashboards. It is the
+  management plane; it does not store telemetry itself.
+- **Monitoring cluster** — a dedicated observability cluster that runs the
+  TelemetryStack and stores all telemetry: metrics (Thanos + object storage),
+  and logs and traces (ClickHouse). It serves multi-tenant queries back to the
+  hub. This is the cluster imported with the **Observability Cluster** profile.
+- **Spoke / imported clusters** — the clusters you actually want to observe.
+  Once the OpenTelemetry feature (`appscode-otel-stack`) is enabled on them,
+  their collectors ship metrics, logs, and traces to the monitoring cluster.
+
+In short: spoke clusters produce telemetry, the monitoring cluster stores and
+serves it, and the hub is where you view it.
+
+## The Monitoring Cluster's Special Features
+
+Beyond a normal spoke, the monitoring cluster runs three components (in the
+`monitoring` namespace) that make multi-tenant telemetry storage work:
+
+- **`thanos-operator`** — reconciles the Thanos custom resources
+  (`ThanosReceive`, `ThanosStore`, `ThanosQuery`, `ThanosCompact`,
+  `ThanosRuler`) that the TelemetryStack generates into running workloads. It
+  is the controller that turns your TelemetryStack spec into an actual Thanos
+  deployment.
+- **`tenant-operator`** — manages `Tenant` custom resources. Each client
+  organization / monitored cluster gets a `Tenant` that references the
+  TelemetryStack and defines its retention and ingest/query endpoints, giving
+  every tenant an isolated slice of the shared stack.
+- **`prom-label-proxy`** — the multi-tenant gateway on the ingest and query
+  paths. It enforces per-tenant label filtering (via the tenant ID / label)
+  so each tenant can only write to and read back its own data. The `Tenant`
+  endpoints point at this proxy.
+
+## Overview
+
+The end-to-end flow is:
+
+1. Create a self-hosted installer.
+2. Create the observability cluster.
+3. Install MinIO.
+4. Create the `telemetry` bucket.
+5. Create a TelemetryStack.
+6. Enable OpenTelemetry on the spoke clusters to be monitored.
+7. Create a client organization.
+
+## Step 1: Create the Self-Hosted Installer
+
+Set up a self-hosted installer for your environment.
+
+## Step 2: Create the Observability Cluster
+
+Create a new cluster and import it as a spoke, choosing the **Observability
+Cluster** cluster profile during import. See
+[Import Observability Cluster](add-cluster/import-observability-cluster.md) for
+details.
+
+## Step 3: Install MinIO
+
+The TelemetryStack can be backed by any S3-compatible object storage — MinIO is
+used here only as an example. Substitute your own storage solution (AWS S3, GCS,
+Ceph, etc.) as needed.
+
+Install MinIO on the observability cluster to provide the object storage that
+backs the TelemetryStack:
+
+```bash
+kubectl create ns minio
+
+kubectl create secret generic tls-ssl-minio -n minio \
+  --from-file=private.key \
+  --from-file=public.crt
+
+kubectl create secret generic tls-ssl-minio -n monitoring \
+  --from-file=private.key \
+  --from-file=public.crt
+
+helm repo add minio-comm https://charts.min.io/
+
+helm upgrade -i minio \
+  --namespace minio \
+  --create-namespace \
+  --set rootUser=rootuser,rootPassword=rootpass123 \
+  minio-comm/minio \
+  -f ./minio/minio-values.yaml
+```
+
+The `tls-ssl-minio` secret is created in both the `minio` and `monitoring`
+namespaces so that MinIO and the telemetry components can serve and consume TLS
+using the same certificate pair (`private.key` / `public.crt`).
+
+## Step 4: Create the `telemetry` Bucket
+
+Port-forward the MinIO console and create the bucket through the UI:
+
+```bash
+kubectl port-forward -n minio svc/minio-console 9001:9001
+```
+
+Open `https://localhost:9001` and log in:
+
+| Field    | Value          |
+|----------|----------------|
+| Username | `rootuser`     |
+| Password | `rootpass123`  |
+
+Create a bucket named **`telemetry`**.
+
+## Step 5: Create a TelemetryStack
+
+In the observability cluster, navigate to:
+
+**Settings → TelemetryStack → Create a new Telemetry stack**
+
+Fill in the configuration sections as shown below.
+
+Configure Metrics (Compact, Store, Query, Router, Ingester):
+
+![Create TelemetryStack — Metrics configuration](../images/otel-monitoring/telemetrystack-1.png)
+
+Ruler, Additional Config, and Configure Clickhouse:
+
+![Create TelemetryStack — Ruler, Additional Config, and Clickhouse](../images/otel-monitoring/telemetrystack-2.png)
+
+Configure Clickhouse (Standalone / Create Topology), Configure S3, and Configure ID:
+
+![Create TelemetryStack — Clickhouse, S3, and ID configuration](../images/otel-monitoring/telemetrystack-3.png)
+
+### Resources Created by the TelemetryStack
+
+Once the TelemetryStack is created, the operators reconcile it into a set of
+resources in the `monitoring` namespace. You can find everything the stack owns
+by looking for resources whose owner is the TelemetryStack:
+
+```bash
+# The Thanos and ClickHouse custom resources generated from the stack
+kubectl get thanosreceives,thanosstores,thanosqueries,thanoscompacts,thanosrulers.monitoring.thanos.io,clickhouse -n monitoring
+
+# Any resource owned by the telemetry-stack (children carry it in ownerReferences)
+kubectl get all -n monitoring -o json | \
+  jq -r '.items[] | select(.metadata.ownerReferences[]?.kind=="TelemetryStack") | "\(.kind)/\(.metadata.name)"'
+```
+
+For the **metrics** pipeline, `thanos-operator` produces:
+
+- **`ThanosReceive`** → a receive-router deployment plus a receive-ingester
+  StatefulSet — the remote-write ingestion endpoint for incoming metrics.
+- **`ThanosStore`** → a store StatefulSet that serves historical blocks from
+  object storage.
+- **`ThanosQuery`** → query and query-frontend deployments that fan out reads
+  across the receivers and store.
+- **`ThanosCompact`** → one StatefulSet per retention tier that downsamples and
+  compacts blocks in object storage.
+- **`ThanosRuler`** → a ruler StatefulSet that evaluates recording/alerting
+  rules and forwards alerts to Alertmanager.
+
+For the **logs / traces** pipeline, a KubeDB **`ClickHouse`** database
+(`telemetry-stack-clickhouse`) is provisioned, along with the supporting
+secrets (`telemetry-stack-object-storage`,
+`telemetry-stack-clickhouse-storage-config`).
+
+## Step 6: Enable OpenTelemetry on a Spoke Cluster
+
+Enable the OpenTelemetry feature on each spoke cluster you want to monitor —
+either during import or on an existing spoke. See
+[Import Observability Cluster](add-cluster/import-observability-cluster.md#enable-the-opentelemetry-feature-on-a-spoke-cluster).
+
+## Step 7: Create a Client Organization
+
+Create a client organization and point it at the monitoring cluster so its
+telemetry is stored in the shared TelemetryStack. See
+[Create a Client Organization](../../client-organization/create-client-organization.md)
+— the **Telemetry Configuration** step is where you select the monitoring
+cluster and set retention.