docs(platform): add telemetry data verification guide

Signed-off-by: Arnob kumar saha <arnob@appscode.com>

Author: ArnobKumarSaha

docs/platform/guides/cluster-management/telemetry-data-check.md

@@ -0,0 +1,187 @@
+---
+layout: docs
+menu:
+  docsplatform_{{.version}}:
+    identifier: cluster-management-telemetry-data-check
+    name: Verifying Telemetry Data
+    parent: cluster-management
+    weight: 100
+menu_name: docsplatform_{{.version}}
+section_menu_id: guides
+---
+
+# Verifying Telemetry Data (Logs & Metrics)
+
+This guide shows how to confirm your logs and metrics are actually being stored — not just
+collected — in the monitoring stack: **ClickHouse** for logs, **Thanos + S3** for metrics.
+See [OpenTelemetry Monitoring](otel-monitoring.md) for how the stack is set up.
+
+## Basics
+
+- Logs and metrics are stored in the **same S3 bucket**, under different prefixes:
+  - Logs → `<bucket>/logs/`
+  - Metrics → `<bucket>/metrics/`
+- Bucket, endpoint, and prefixes are defined in your `TelemetryStack` resource:
+  ```bash
+  kubectl get telemetrystack -n monitoring -o yaml
+  ```
+  Look under `spec.logs.clickhouse.s3` and `spec.metrics.thanos.s3`.
+- ClickHouse database name is your tenant ID, not a fixed name like `otel` or `default`.
+  Find it with `SHOW DATABASES` (see below) — the table inside is `otel_logs`.
+- All checks below run from inside the cluster via `kubectl exec`, so no port-forwarding
+  or extra credentials are required except for the optional S3 listing step.
+
+---
+
+## 1. Check logs are landing in ClickHouse
+
+Find the ClickHouse pod:
+
+```bash
+kubectl get pods -n monitoring -l app.kubernetes.io/name=clickhouse
+```
+
+List databases and confirm the `otel_logs` table exists:
+
+```bash
+kubectl exec -n monitoring <clickhouse-pod> -- clickhouse-client --query "SHOW DATABASES"
+kubectl exec -n monitoring <clickhouse-pod> -- clickhouse-client --query "SHOW TABLES FROM <your-tenant-db>"
+```
+
+Check row count and freshness:
+
+```bash
+kubectl exec -n monitoring <clickhouse-pod> -- clickhouse-client --query \
+  "SELECT count(), min(Timestamp), max(Timestamp), now() FROM <your-tenant-db>.otel_logs"
+
+kubectl exec -n monitoring <clickhouse-pod> -- clickhouse-client --query \
+  "SELECT count() FROM <your-tenant-db>.otel_logs WHERE Timestamp > now() - INTERVAL 5 MINUTE"
+```
+
+**Healthy result:** `max(Timestamp)` is close to `now()`, and the 5-minute count is greater
+than zero. If both queries return recent data, logs are flowing end-to-end.
+
+---
+
+## 2. Check metrics are landing in Thanos / S3
+
+Find the Thanos query pod:
+
+```bash
+kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos-query
+```
+
+Ask it which StoreAPIs it's talking to and what time range each one covers:
+
+```bash
+kubectl exec -n monitoring <thanos-query-pod> -- \
+  wget -qO- --header="THANOS-TENANT: <your-tenant>" "http://localhost:9090/api/v1/stores"
+```
+
+Look at the `minTime`/`maxTime` fields (Unix epoch **milliseconds**) for each store:
+
+```bash
+date -u -d @<maxTime-in-seconds>   # drop the last 3 digits first to convert ms -> s
+```
+
+**Healthy result:** `maxTime` should be within the last few minutes for the live/receive
+store, and within your compaction interval for the long-term S3-backed store.
+
+For a second confirmation, check the ingester's own TSDB head timestamp:
+
+```bash
+kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos-receive-ingester
+kubectl exec -n monitoring <thanos-receive-ingester-pod> -- wget -qO- http://localhost:10902/metrics \
+  | grep prometheus_tsdb_head_max_time
+```
+
+If this timestamp is stale (not advancing over successive checks), metrics are not being
+ingested. Check that Prometheus (or your agent) has `remoteWrite` configured to point at
+the Thanos receive router:
+
+```bash
+kubectl get prometheus -n monitoring <prometheus-name> -o jsonpath='{.spec.remoteWrite}'
+```
+
+An empty result means nothing is shipping samples to Thanos, regardless of whether
+Prometheus itself is scraping normally.
+
+---
+
+## 3. Check the S3 bucket directly (optional)
+
+Use the MinIO/S3 client (`mc` or `aws s3`) with your access credentials from the
+`TelemetryStack` spec:
+
+```bash
+kubectl run mc-check -n monitoring --rm -it --restart=Never --image=minio/mc:latest -- sh
+mc alias set myminio <endpoint> <accessKey> <secretKey> --insecure
+mc ls myminio/<bucket>/logs/    --insecure --recursive | tail
+mc ls myminio/<bucket>/metrics/ --insecure --recursive | tail
+```
+
+**Healthy result:** recent objects under both prefixes, with timestamps matching your
+retention/compaction schedule.
+
+---
+
+## Summary
+
+| Pipeline | Where to look | Healthy signal |
+|---|---|---|
+| Logs | ClickHouse `<tenant-db>.otel_logs` | `max(Timestamp)` ≈ now, non-zero rows in last 5 min |
+| Metrics | Thanos Query `/api/v1/stores`, receive ingester `/metrics` | `maxTime`/head timestamp advancing, close to now |
+| S3 | `<bucket>/logs/` and `<bucket>/metrics/` prefixes | Recent objects under both prefixes |
+
+---
+
+## Worked example: diagnosing a broken metrics pipeline
+
+The steps above surfaced a real split-brain state on a monitoring cluster: logs healthy,
+metrics stale for four days. This walks through how the checks pinpointed the cause.
+
+**Logs check** — healthy:
+
+```bash
+kubectl exec -n monitoring telemetry-stack-clickhouse-0 -- clickhouse-client --query \
+  "SELECT count(), min(Timestamp), max(Timestamp), now() FROM ace_user_3.otel_logs"
+# → 60.7M rows, max(Timestamp) == now(), 11,638 rows in the last 5 minutes
+```
+
+**Metrics check** — broken:
+
+- `thanos-query` `/api/v1/stores`: the `receive` StoreAPI reported sentinel min/max
+  timestamps (i.e. no live head data at all). The `store` StoreAPI (long-term, backed by S3)
+  reported `maxTime=1783742913242` ms → `2026-07-11T04:08:33Z`, four days stale.
+- `thanos-receive-ingester` `/metrics`: `prometheus_tsdb_head_max_time{tenant="ace.user.3"}`
+  matched that same stale timestamp; a `tenant="default-tenant"` series showed sentinel
+  values, meaning it had never received anything.
+- `thanos-receive-ingester` logs: last `"Uploaded head block"` line was
+  `2026-07-11T08:00:49Z`; every 4h afterward only `"Running pruning job"` appeared, alongside
+  a recurring error:
+  ```
+  msg="Failed to calculate size of \"chunks_head\" dir" err="lstat /var/thanos/receive/ace.user.3/chunks_head: no such file or directory"
+  ```
+
+**Root cause** — checking the Prometheus CR's remote-write config confirmed nothing was
+shipping samples:
+
+```bash
+kubectl get prometheus -n monitoring kube-prometheus-stack-prometheus -o jsonpath='{.spec.remoteWrite}'
+# → (empty)
+```
+
+Prometheus itself was scraping fine locally (`prometheus_remote_storage_highest_timestamp_in_seconds`
+== now), but with `spec.remoteWrite` unset there was nothing pointing samples at the Thanos
+receive router. No `PrometheusAgent` resources existed either, so no other component was
+doing remote-write in its place.
+
+**Fix** — add `spec.remoteWrite` to the `Prometheus` CR, pointing at the Thanos receive
+router's remote-write endpoint (`http://thanos-receive-router-<stack>.monitoring.svc:19291/api/v1/receive`)
+with the tenant header the router's hashring expects (e.g. `THANOS-TENANT: <tenant>`). After
+applying, re-run the `prometheus_tsdb_head_max_time` check above over a couple of minutes to
+confirm the timestamp starts advancing.
+
+**Takeaway:** a healthy logs pipeline says nothing about the metrics pipeline (or vice
+versa) — they're independent paths through the same `TelemetryStack`, so always check both
+per the steps above rather than assuming one implies the other.