Document Observability Integrations (#433)

This has been lagging but it's important to help users understand the config involved and also advertise the functionality.

Author: michaeljguarino

generated/routes.json

@@ -29,7 +29,7 @@
   },
   "/getting-started/first-steps/cli-quickstart": {
     "relPath": "/getting-started/first-steps/cli-quickstart.md",
-    "lastmod": "2025-03-12T14:59:41.000Z"
+    "lastmod": "2025-03-20T18:53:12.000Z"
   },
   "/getting-started/first-steps/existing-cluster": {
     "relPath": "/getting-started/first-steps/existing-cluster.md",
@@ -129,7 +129,7 @@
   },
   "/plural-features/k8s-upgrade-assistant/addon-compatibilities": {
     "relPath": "/plural-features/k8s-upgrade-assistant/addon-compatibilities.md",
-    "lastmod": "2025-03-13T09:28:46.000Z"
+    "lastmod": "2025-03-23T01:07:28.000Z"
   },
   "/plural-features/stacks-iac-management": {
     "relPath": "/plural-features/stacks-iac-management/index.md",
@@ -199,6 +199,22 @@
     "relPath": "/plural-features/plural-ai/cost.md",
     "lastmod": "2025-03-12T14:59:41.000Z"
   },
+  "/plural-features/observability": {
+    "relPath": "/plural-features/observability/index.md",
+    "lastmod": "2025-03-25T00:00:37.000Z"
+  },
+  "/plural-features/observability/prometheus": {
+    "relPath": "/plural-features/observability/prometheus.md",
+    "lastmod": "2025-03-25T00:00:37.000Z"
+  },
+  "/plural-features/observability/logging": {
+    "relPath": "/plural-features/observability/logging.md",
+    "lastmod": "2025-03-25T00:00:37.000Z"
+  },
+  "/plural-features/observability/cost": {
+    "relPath": "/plural-features/observability/cost.md",
+    "lastmod": "2025-03-25T00:00:37.000Z"
+  },
   "/plural-features/pr-automation": {
     "relPath": "/plural-features/pr-automation/index.md",
     "lastmod": "2025-03-12T14:59:41.000Z"
@@ -285,7 +301,7 @@
   },
   "/deployments/cli-quickstart": {
     "relPath": "/getting-started/first-steps/cli-quickstart.md",
-    "lastmod": "2025-03-12T14:59:41.000Z"
+    "lastmod": "2025-03-20T18:53:12.000Z"
   },
   "/deployments/existing-cluster": {
     "relPath": "/getting-started/first-steps/existing-cluster.md",

pages/plural-features/observability/cost.md

@@ -0,0 +1,62 @@
+---
+title: OpenCost Configuration
+description: Configure cost visibility with OpenCost on Plural
+---
+
+## What You'll Get
+
+Plural's leverages OpenCost to gather cost data for any deployed cluster.  This works by:
+
+* deploying an opencost instance per cluster, which acts as a lightweight agent exposing no web ui.
+* configuring an extractor to periodically scrape that instance and ship the data to our central store for historical retrieval and analysis.
+
+This provides a seamless multi-cluster experience for opencost data, and when combined with Plural's built-in deployment knowledge and AI capabilities, can also automate the remediation of overprovisioning with our PR generation capabilities.
+
+* Displaying saturation data for common compute controllers within Kubernetes, like deployments and statefulsets
+* Displaying cluster-level aggregate metrics for CPU, memory, pod saturation, etc.
+* Extracting network metrics for service mesh observability, in particular Istio and Cilium both have their standard network metrics shipped to Prometheus.  We can use them to provide a holistic network graph and also use that information to inform Plural AI.
+
+Once configured, if you go to a compute-related resource and click on the `Metrics` tab, you'll see:
+
+A Full fleet-wide dashboard for cost information:
+
+![](/assets/observability/fleet-cost.png)
+
+A dashboard for a specific cluster:
+
+![](/assets/observability/cluster-cost.png)
+
+A namespace granularity breakdown of costs in a cluster:
+
+![](/assets/observability/namespace-cost.png)
+
+And recommendations to right-size workloads in a cluster:
+
+![](/assets/observability/cost-recs.png)
+
+## Deploy out of our Service Catalog
+
+Plural ships by default with a full [Service catalog](/plural-features/service-catalog) to easily deploy solutions across your stack, among them a setup of OpenCost.  This will:
+
+* set up the opencost agent across your fleet
+* configure the kubecost extractors to ship cost data back to Plural
+
+To deploy out of our storage catalog, go to `Service Catalog` -> `devops` -> `kubecost-setup` and fill out the wizard, it requires no advanced configuration and should just require you to provide a branch name for the PR to configure it.
+
+## Manually Configure against an Existing OpenCost Instance
+
+You can also use an existing OpenCost/Kubecost install with Plural for aggregation.  To do this, simply make sure the following CRD is installed everywhere (installation of the Plural Operator is required):
+
+```yaml
+apiVersion: deployments.plural.sh/v1alpha1
+kind: KubecostExtractor
+metadata:
+  name: default
+spec:
+  interval: "1h"
+  recommendationThreshold: "1"
+  kubecostPort: 9090
+  kubecostServiceRef:
+    name: kubecost-cost-analyzer # this is the kubernetes service hosting the kubecost api
+    namespace: kubecost
+```

pages/plural-features/observability/index.md

@@ -0,0 +1,76 @@
+---
+title: Observability Configuration
+description: Configure metrics and logs provider to directly into the Plural UI 
+---
+
+Plural has the ability to query multiple backends for standard observability data and integrate them directly into the mainline Plural experience.  In particular, these are:
+
+* Prometheus for timeseries metrics
+* Elastic (and others) for log aggregation
+* Kubecost for kubernetes cost observability
+
+All observability implementations are designed to be multi-cluster by default and we also provide out-of-the-box installs in the event a user still hasn't adopted a proper solution for any of the main observability providers.
+
+There are a few major benefits of integrating this data with Plural:
+
+* Single-Pane-Of-Glass for Operations - a unified infrastructure management and observability platform reduces context switching for engineers, and makes it easier for team-members to ramp with their operational responsibilities.
+* AI - this is the most powerful benefit, as Plural gets access to more data, we can integrate it with our unified AI engine and automate more of the tedious root cause analysis and troubleshooting you'd otherwise do manually.
+
+
+## Plural AI + Observability
+
+Generally, Plural AI is a broad RAG system that ultimately calls into mainline LLMs.  This means the more data we can access the more likely we can find that diamond in the rough that solves your specific problem.
+
+In particular, there are a few major workflows our observability integration pairs with Plural AI:
+
+* Log analysis of low-level Kubernetes components: a few tools, especially external-dns and cert-manager, provided very little error reporting outside of their logs.  With an easier way to extract these, we can automatically root cause failures around DNS registration and certificate issuance, which are both commong annoyances with Kubernetes operators.  This is true of a number of other tools as well, especially solutions like CSI drivers.
+* Application Code Failures - common failure modes like failing health checks, 500 error codes, etc, are usually do to bugs in application code and not at the infrastructure level.  With a combination of log analysis and searchable PR data, we can RCA these issues and cross reference them to the offending code changes.
+
+## CRD Configurability
+
+For the most part, these settings are entirely configurable via one of our Kubernetes custom resources, `DeploymentSettings`.  A kitchen-sink example of this is below:
+
+```yaml
+apiVersion: deployments.plural.sh/v1alpha1
+kind: DeploymentSettings
+metadata:
+  name: global # this is a singleton resource that is always at this location
+  namespace: plrl-deploy-operator
+spec:
+  managementRepo: pluralsh/plrl-boot-aws
+  cost: # configuring recommendations from kubecost data
+    recommendationCushion: 15
+    recommendationThreshold: 1
+    
+  ai:
+    enabled: true
+
+    vectorStore: # elastic can be repurposed as a vector store too
+      enabled: true
+      vectorStore: ELASTIC
+      elastic:
+        host: https://{your-elastic-fqdn}
+        user: plrl
+        index: plrl-ai-vectors
+        passwordSecretRef:
+          name: plrl-elastic-user
+          key: password
+    
+  logging:
+    enabled: true
+    driver: ELASTIC
+    elastic:
+      host: https://{your-elastic-fqdn}
+      user: plrl
+      index: plrl-logs-*
+      passwordSecretRef:
+        name: plrl-elastic-user
+        key: password
+
+  prometheusConnection:
+    host: https://{your-prometheus-url}
+    user: plrl
+    passwordSecretRef:
+      name: basic-auth-prom
+      key: password
+```

pages/plural-features/observability/logging.md

@@ -0,0 +1,74 @@
+---
+title: ElasticSearch Configuration
+description: Configure multi-cluster Log Aggregation with Plural
+---
+
+## What You'll Get
+
+Plural's built-in log support allows you to query logs at either the Plural Service or Cluster level, dependent on your permissions to those respective resources.  Service logs are relevant for developer personas, whereas cluster logs are usually useful for platform engineers or kubernetes admins.
+
+Our `Logs` tab features all the standard features like text-based search, facet filtering and time filtering, looking something like so:
+
+![](/assets/getting-started/log-tail.png)
+
+
+And you can tune the log view with the filter modal like:
+
+![](/assets/getting-started/log-filters.png)
+
+
+In addition to the single-pane-of-glass benefits, log data is an incredibly powerful tool for enhancing the dataset used by Plural AI, and so is highly recommended as an addition to a production deployment of Plural.
+
+## Deploy out of our Service Catalog
+
+Plural ships by default with a full [Service catalog](/plural-features/service-catalog) to easily deploy solutions across your stack, among them a setup of ElasticSearch + Logstach using to support robust multi-cluster log aggregation.  This has an additional benefit of also serving as a vector store for additional data used by our AI engine if you wish to enable that.  In total this will set up:
+
+* ElasticSearch using the [ECK operator](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html)
+* LogStach (also using ECK) to aggregate logs at the cluster level and ship them to the singular ElasticSearch Instance
+
+{% callout severity="info" %}
+The system is made to be modular, and can support other logging backends.  If you're interested in using another one, let us know and we'll add it to our roadmap!
+{% /callout %}
+
+To deploy out of our storage catalog, go to `Service Catalog` -> `devops` -> `elastic` and fill out the wizard, it should look something like this:
+
+![prometheus-setup](/assets/observability/elastic-setup.png)
+
+{% callout severity="warning" %}
+We recommend using at least 100Gi for storage if you have a fairly large fleet, this is simply due to the log volume emitted by Kubernetes being quite large at the baseline.  ECK can very easily be tuned post-hoc if you size it too small as well.
+
+Also be sure the cluster you're deploying to has a working ingress setup, since it'll assume external-dns + cert-manager are there to set up DNS and TLS.
+{% /callout %}
+
+The generated PR will have configured:
+
+* a deployment of ElasticSearch to store log and vector data 
+* Logstash for log extraction, filtering and remote shipping to elasticsearch
+* the configuration of the setup in the `DeploymentSettings` CRD to register them with Plural
+
+## Custom Resource Configuration
+
+If you were to do this manually with an existing Prometheus setup, you'd need to manually wire configuration like the following:
+
+```yaml
+apiVersion: deployments.plural.sh/v1alpha1
+kind: DeploymentSettings
+metadata:
+  name: global # this is a singleton resource that is always at this location
+  namespace: plrl-deploy-operator
+spec:
+  prometheusConnection:
+    host: https://{your-prometheus-url}
+    user: plrl # or whatever user you'd want
+    passwordSecretRef:
+      name: basic-auth-prom
+      key: password
+```
+
+## ElasticSearch as a Vector Store
+
+One of the reasons we lean on ElasticSearh as a default log store is its a broadly usable data store with recent support for vector search.  When this is enabled, Plural AI can start doing the following:
+
+* Vectorize and search historical Pull Requests (gathered from [establishing a SCM webhook](/getting-started/how-to-use/scm-connection#add-an-scm-provider-webhook))
+* Vectorize and search historical alert resolutions
+* ...and more coming soon!

pages/plural-features/observability/prometheus.md

@@ -0,0 +1,65 @@
+---
+title: Prometheus Configuration
+description: Configure multi-cluster Prometheus with Plural
+---
+
+## What You'll Get
+
+Plural's leverages Prometheus for a few main purposes:
+
+* Displaying saturation data for common compute controllers within Kubernetes, like deployments and statefulsets
+* Displaying cluster-level aggregate metrics for CPU, memory, pod saturation, etc.
+* Extracting network metrics for service mesh observability, in particular Istio and Cilium both have their standard network metrics shipped to Prometheus.  We can use them to provide a holistic network graph and also use that information to inform Plural AI.
+
+Once configured, if you go to a compute-related resource and click on the `Metrics` tab, you'll see something like:
+
+![](/assets/getting-started/metrics.png)
+
+And cluster saturation views will look like:
+
+![](/assets/getting-started/cluster-metrics.png)
+
+# Deploy out of our Service Catalog
+
+Plural ships by default with a full [Service catalog](/plural-features/service-catalog) to easily deploy solutions across your stack, among them a robust, scale-out setup of Prometheus using [VictoriaMetrics](https://docs.victoriametrics.com/).  This solves a number of key problems with mainline Prometheus:
+
+* Horizontal scaling - Prometheus scales only vertically, which is not an appropriate fit for monitoring large sets of kubernetes clusters which are going to emit a very large set of metrics and increases operational burden. VictoriaMetrics supports a horizontally scalable cluster mode, which solves this entirely
+* Inefficient agent mode - Prometheus wasn't built for a remote-write model, and its agent mode still requires effectively a local prometheus store on-cluster.  `vmagent` is a much better implementation and also adds support for other protocols like statsd.  Remote write is necessary for multi-cluster observabilty since you likely cannot ingress into all clusters to perform metrics scrapes.
+* Better kubernetes operator implementation - the mainline Prometheus operator has a number of longstanding issues, in particular not supporting volume resizing as discussed [here](https://github.com/prometheus-operator/prometheus-operator/issues/4079)
+
+
+To deploy out of our storage catalog, go to `Service Catalog` -> `devops` -> `prometheus-setup` and fill out the wizard, it should look something like this:
+
+![prometheus-setup](/assets/observability/prom-setup.png)
+
+{% callout severity="info" %}
+Be sure to select a cluster with a working ingress setup, since it'll assume external-dns + cert-manager are there to set up DNS and TLS.
+{% /callout %}
+
+The generated PR will have configured:
+
+* a deployment of VictoriaMetrics server to store metrics
+* `vmagent` to ship metrics to the server
+* the configuration of the setup in the `DeploymentSettings` CRD to register them with Plural
+
+
+# Custom Resource Configuration
+
+If you were to do this manually with an existing Prometheus setup, you'd need to manually wire configuration like the following:
+
+```yaml
+apiVersion: deployments.plural.sh/v1alpha1
+kind: DeploymentSettings
+metadata:
+  name: global # this is a singleton resource that is always at this location
+  namespace: plrl-deploy-operator
+spec:
+  prometheusConnection:
+    host: https://{your-prometheus-url}
+    user: plrl
+    passwordSecretRef:
+      name: basic-auth-prom
+      key: password
+```
+
+Also we expect a `cluster` label on all prometheus metrics which matches the cluster handle in Plural.  If not present, we might not be able to properly correlate metrics with clusters/workloads.  This will be configured out-of-the-box by our `vmagent` setup.