Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,11 @@
"configuration/defining-playbooks/creating-notifications.html": "/master/playbook-reference/defining-playbooks/creating-notifications.html",
"configuration/defining-playbooks/external-playbook-repositories.html": "/master/playbook-reference/defining-playbooks/external-playbook-repositories.html",
"configuration/defining-playbooks/index.html": "/master/playbook-reference/defining-playbooks/index.html",
"configuration/defining-playbooks/playbook-advanced.html": "/master/playbook-reference/defining-playbooks/playbook-advanced.html",
"configuration/defining-playbooks/playbook-basics.html": "/master/playbook-reference/defining-playbooks/playbook-basics.html",
"configuration/defining-playbooks/trigger-action-binding.html": "/master/playbook-reference/defining-playbooks/trigger-action-binding.html",
"configuration/defining-playbooks/playbook-advanced.html": "/master/playbook-reference/index.html",
"playbook-reference/defining-playbooks/playbook-advanced.html": "/master/playbook-reference/index.html",
"configuration/defining-playbooks/playbook-basics.html": "/master/playbook-reference/index.html",
"configuration/defining-playbooks/trigger-action-binding.html": "/master/playbook-reference/index.html#matching-actions-to-triggers",
"playbook-reference/defining-playbooks/trigger-action-binding.html": "/master/playbook-reference/index.html#matching-actions-to-triggers",
"configuration/additional-settings.html": "/master/setup-robusta/additional-settings.html",
"developer-guide/writing-playbooks.html": "/master/playbook-reference/defining-playbooks/index.html",
"user-guide/slack.html": "/master/configuration/sinks/slack.html",
Expand All @@ -108,7 +110,8 @@
"configuration/configuring-sinks.html": "/master/notification-routing/configuring-sinks.html",
"how-it-works/privacy-and-security.html": "/master/setup-robusta/privacy-and-security.html",
"how-it-works/index.html": "/master/playbook-reference/what-are-playbooks.html",
"playbook-reference/examples.html": "/master/playbook-reference/prometheus-examples/index.html",
"playbook-reference/examples.html": "/master/playbook-reference/builtin-alert-enrichment.html",
"playbook-reference/prometheus-examples/index.html": "/master/playbook-reference/builtin-alert-enrichment.html",
"tutorials/playbook-track-changes.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
"tutorials/playbook-job-failure.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
"tutorials/playbook-failed-liveness.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
Expand Down Expand Up @@ -167,7 +170,7 @@
"user-guide/embedded-prometheus.html": "/master/configuration/alertmanager-integration/embedded-prometheus.html#enabling-the-embedded-prometheus",
"user-guide/node-selector.html": "/master/setup-robusta/node-selector.html",
"user-guide/interactivity.html": "/master/setup-robusta/additional-settings.html#two-way-interactivity",
"user-guide/flow-control.html": "/master/playbook-reference/defining-playbooks/playbook-advanced.html#using-filters-to-restrict-triggers",
"user-guide/flow-control.html": "/master/playbook-reference/index.html#using-filters-to-restrict-triggers",
"catalog/triggers/index.html": "/master/playbook-reference/triggers/index.html",
"catalog/triggers/kubernetes.html": "/master/playbook-reference/triggers/kubernetes.html",
"catalog/triggers/smart.html": "/master/playbook-reference/triggers/kubernetes.html",
Expand All @@ -189,7 +192,7 @@
"catalog/sinks/webex.html": "/master/configuration/sinks/webex.html",
"catalog/sinks/VictorOps.html": "/master/configuration/sinks/VictorOps.html",
"catalog/sinks/file.html": "/master/configuration/sinks/file.html",
"user-guide/trigger-action-binding.html": "/master/playbook-reference/defining-playbooks/playbook-basics.html#understanding-actions",
"user-guide/trigger-action-binding.html": "/master/playbook-reference/index.html#understanding-actions",
"advanced/privacy-and-security.html": "/master/setup-robusta/privacy-and-security.html",
"advanced/robusta-ui-triggers.html": "/master/setup-robusta/installation/index.html",
"developer-guide/actions/index.html": "/master/playbook-reference/actions/index.html",
Expand All @@ -209,7 +212,9 @@
"configuration/alertmanager-integration/troubleshooting-alertmanager.html": "/master/configuration/exporting/send-alerts-api.html",
"configuration/alertmanager-integration/grafana-alert-manager.html": "/master/configuration/alertmanager-integration/grafana-self-hosted.html",
"configuration/alertmanager-integration/grafana-cloud-mimir.html": "/master/configuration/alertmanager-integration/grafana-cloud.html",
"playbook-reference/what-are-playbooks.html": "/master/playbook-reference/overview.html",
"playbook-reference/what-are-playbooks.html": "/master/playbook-reference/index.html",
"playbook-reference/overview.html": "/master/playbook-reference/index.html",
"playbook-reference/defining-playbooks/playbook-basics.html": "/master/playbook-reference/index.html",
"how-it-works/alert-builtin-enrichment.html": "/master/playbook-reference/builtin-alert-enrichment.html",
"setup-robusta/installation/extend-prometheus-installation.html": "/master/setup-robusta/installation/standalone-installation.html",
"playbook-reference/defining-playbooks/index.html": "/master/playbook-reference/index.html",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,10 @@ Create Custom Prometheus Alerts
You can define new alerts in two ways using Robusta:

1. Prometheus Alerts - Using PromQL
2. Robusta Playbooks - Using :ref:`customPlaybooks YAML <What are Playbooks?>`
2. Robusta Playbooks - Using :ref:`customPlaybooks YAML <Playbook Basics>`

These methods are not mutually exclusive. Robusta playbooks can respond to Prometheus alerts, or they can generate
alerts themselves by listening directly to the Kubernetes APIServer. To better understand the trade-offs, refer to
:ref:`Should I generate alerts with Robusta or with Prometheus? <robusta-or-prometheus-alerts>`
alerts themselves by listening directly to the Kubernetes APIServer.

In this tutorial, we use the first method to generate a custom Prometheus alert using PromQL. In the next tutorial,
we define a custom Robusta playbook that enhances the alert and makes it better.
Expand Down Expand Up @@ -109,4 +108,4 @@ Next Steps

Learn how to enrich Prometheus alerts with more context, so that you can respond faster:

* :ref:`Prometheus Alert Enrichment`
* :ref:`Alert Enrichment <builtin-alert-enrichment>`
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ To enable Robusta to correlate your Grafana alerts with the specific Kubernetes
This is only required for Kubernetes alerts. You can send any alert to the Robusta timeline, including non-Kubernetes alerts.

Option 2: Inline Alert Enrichment and Routing
===========================================
==============================================

Use Robusta to enrich alerts inline with extra context and route them to :doc:`other systems </configuration/sinks/index>` (Slack, Microsoft Teams, etc.). Learn more about :doc:`alert routing </notification-routing/index>`.

Expand Down
2 changes: 2 additions & 0 deletions docs/configuration/holmesgpt/getting-started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,8 @@ Instead of Robusta AI, you can use your own OpenAI, Azure, or AWS Bedrock accoun
name: holmes-secrets
key: awsSecretAccessKey

.. _Reading the Robusta UI Token from a secret in HolmesGPT:

Using Existing Secrets
----------------------

Expand Down
10 changes: 5 additions & 5 deletions docs/configuration/resource-recommender.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
Cost Savings (KRR)
************************************************************

Robustas `KRR <https://github.com/robusta-dev/krr>`_ is a CLI tool for optimizing resource allocation in Kubernetes clusters.
It gathers pod usage data from Prometheus and recommends requests and limits for CPU and memory. This reduces costs and improves performance.
`KRR <https://github.com/robusta-dev/krr>`_ is a CLI tool that optimizes resource allocation in Kubernetes clusters.
It gathers pod usage data from Prometheus and recommends requests and limits for CPU and memory, reducing costs and improving performance.

By optionally integrating KRR with Robusta you can:
Robusta can run KRR scans on a :ref:`schedule <Scheduled>` using playbooks. Because KRR is so popular, it has dedicated documentation here. You can:
Comment thread
aantn marked this conversation as resolved.

1. Get weekly KRR scan reports in Slack via Robusta OSS (disabled by default, see below to configure)
2. View KRR scans from all your clusters in the Robusta UI (enabled by default for UI users)
1. Send weekly scan reports to Slack or other sinks via Robusta OSS (disabled by default, configure below)
2. View scans from all your clusters in the Robusta UI (enabled by default for UI users)


Sending Weekly KRR Scan Reports to Slack
Expand Down
4 changes: 3 additions & 1 deletion docs/help.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
:hide-toc:

.. _Getting Support:

Getting Support
================

Expand Down Expand Up @@ -248,7 +250,7 @@ Holmes

It's often because the ``Robusta UI Token`` is pulled from a secret, and Holmes cannot read it.

See :ref:`Reading the Robusta UI Token from a secret in HolmesGPT` to configure Holmes to read the ``token``
See :ref:`Using Existing Secrets <Reading the Robusta UI Token from a secret in HolmesGPT>` to configure Holmes to read the ``token``

Phase 4: Integration Issues
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
5 changes: 2 additions & 3 deletions docs/how-it-works/coverage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Prometheus Alerts

.. warning::

You must :ref:`send your Prometheus alerts to Robusta by webhook <Integrating with Prometheus>` for these to work.
You must send your Prometheus alerts to Robusta by webhook for these to work. See :doc:`AlertManager Integration </configuration/alertmanager-integration/index>`.
Comment thread
aantn marked this conversation as resolved.

Other errors
----------------
Expand All @@ -39,8 +39,7 @@ Change Tracking
By default all changes to Deployments, DaemonSets, and StatefulSets are sent to the Robusta UI for correlation
with Prometheus alerts and other errors.

These changes are not sent to other sinks (e.g. Slack) by default because they are spammy. :ref:`Routing Cookbook`
explains how to selectively track changes you care about in Slack as well.
These changes are not sent to other sinks (e.g. Slack) by default because they are spammy. See :doc:`Notification Routing </notification-routing/index>` to learn how to selectively track changes you care about in Slack as well.

We also wrote a blog post `Why everyone should track Kubernetes changes and top four ways to do so <https://home.robusta.dev/blog/why-everyone-should-track-and-audit-kubernetes-changes-and-top-ways/>`_

Expand Down
11 changes: 8 additions & 3 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,15 @@
:hidden:

playbook-reference/index
Builtin Alert Enrichment <playbook-reference/builtin-alert-enrichment>
Custom Alert Enrichment <playbook-reference/prometheus-examples/index>
Kubernetes Change Notifications <playbook-reference/kubernetes-examples/kubernetes-change-notifications>
Playbook Notifications <playbook-reference/defining-playbooks/creating-notifications>
Alert Enrichment <playbook-reference/builtin-alert-enrichment>
Automatic Remediation <playbook-reference/automatic-remediation-examples/index>
Change Tracking Playbooks <playbook-reference/kubernetes-examples/kubernetes-change-notifications>
Silencer Playbooks <playbook-reference/defining-playbooks/silencer-playbooks>
Log Based Alerting <playbook-reference/logs-triggers/index>
Cost Savings - KRR <configuration/resource-recommender>
Triggers Reference <playbook-reference/triggers/index>
Actions Reference <playbook-reference/actions/index>

.. toctree::
:maxdepth: 4
Expand Down
2 changes: 1 addition & 1 deletion docs/notification-routing/configuring-sinks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ A Simple Sink Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sinks are defined in Robusta's Helm chart, using the ``sinksConfig`` value.

For example, lets add a :ref:`Microsoft Teams Sink <MS Teams sink>`:
For example, lets add a :ref:`Microsoft Teams Sink <MS Teams>`:

.. code-block:: yaml

Expand Down
2 changes: 1 addition & 1 deletion docs/notification-routing/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Popular Sinks

.. grid-item-card:: PagerDuty
:class-card: sd-bg-light sd-bg-text-light
:link: ../configuration/sinks/pagerduty
:link: ../configuration/sinks/PagerDuty
:link-type: doc

.. grid-item-card:: View All Sinks
Expand Down
1 change: 1 addition & 0 deletions docs/notification-routing/routing-silencing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,5 @@ node that just restarted.
Further Reading
-----------------

* Learn more about :ref:`Silencer Playbooks <Silencer Playbooks>`
* View all :ref:`Prometheus Silencer <Prometheus Silencers>` actions
1 change: 1 addition & 0 deletions docs/playbook-reference/actions/develop-actions/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Please consider sharing your custom actions with the community.
my-first-custom-action
playbook-repositories
loading-custom-actions
Loading External Actions <../../defining-playbooks/external-playbook-repositories>
overriding-builtin-actions
findings-api
triggers-and-events
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _Loading Custom Actions into Robusta:

Loading Custom Actions into Robusta
####################################

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _Creating Playbook Repositories:

Creating Playbook Repositories
################################

Expand Down
2 changes: 2 additions & 0 deletions docs/playbook-reference/actions/index.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
:hide-toc:

.. _Actions Reference:

Actions Reference
==================

Expand Down
46 changes: 40 additions & 6 deletions docs/playbook-reference/builtin-alert-enrichment.rst
Original file line number Diff line number Diff line change
@@ -1,26 +1,60 @@
.. _builtin-alert-enrichment:

Builtin Alert Enrichment
Alert Enrichment
########################################

Robusta takes Prometheus to the next level by correlating alerts with other observability data.
Ever feel overwhelmed by alerts that lack context? Robusta enriches alerts automatically and lets you create custom enrichment rules.

.. note::

Testing out Prometheus alerts
**Looking for automatic AI enrichment?** Check out :doc:`HolmesGPT </configuration/holmesgpt/main-features>` for zero-configuration AI-powered alert enrichment that automatically investigates alerts and provides root cause analysis.

Builtin Alert Enrichment
*********************************
1. Deploy a broken pod that will be stuck in pending state:

Robusta automatically enriches Prometheus alerts with relevant Kubernetes context:

* **Pod events** - Recent events related to the affected pod
* **Pod logs** - Relevant log excerpts from crashing or failing containers
* **Resource metrics** - CPU, memory, and other resource usage data
* **Related Kubernetes objects** - Deployments, ReplicaSets, ConfigMaps, etc.

This happens automatically for common Prometheus alerts without any configuration. To extend it to your own Prometheus alerts you can define custom playbooks.

Testing Alert Enrichment
^^^^^^^^^^^^^^^^^^^^^^^^^^

1. Deploy a broken pod:

.. code-block:: bash
:name: cb-apply-pendingpod

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_resources.yaml

2. Trigger a Prometheus alert immediately, skipping the normal delays:
2. Trigger a Prometheus alert immediately:

.. code-block:: bash
:name: cb-trigger-prometheus-alert

robusta playbooks trigger prometheus_alert alert_name=KubePodCrashLooping namespace=default pod_name=example-pod

.. admonition:: Example Slack Message
.. admonition:: Example Enriched Alert

.. image:: /images/simulatedprometheusalert.png

Custom Alert Enrichment
*********************************

Create custom enrichment rules to:

* Reduce MTTR by automatically gathering system state and logs when alerts fire
* Make faster decisions on which team needs to investigate
* Link alerts to runbooks and documentation for better knowledge sharing

Get started with these examples:

.. toctree::
:maxdepth: 1

prometheus-examples/bash-alert-enrichment
prometheus-examples/link-alert-enrichment
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The following default playbook handles all Prometheus alerts that Robusta receiv

There are additional enrichments for specific alerts. For example:

To define additional playbooks for your own alerts, refer to the :doc:`Custom Alert Enrichment </playbook-reference/prometheus-examples/index>` guide.
To define additional playbooks for your own alerts, refer to the :doc:`Enrich Alerts </playbook-reference/builtin-alert-enrichment>` guide.

Default Prometheus Silencing
--------------------------------
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Creating Notifications
Playbook Notifications
######################

Playbooks can generate notifications to *let a human know* about something in your cluster.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _Loading External Actions:

Loading External Actions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
5 changes: 1 addition & 4 deletions docs/playbook-reference/defining-playbooks/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,8 @@ Learn how to define Robusta playbooks.
.. toctree::
:maxdepth: 1

playbook-basics
creating-notifications
playbook-advanced
trigger-action-binding
external-playbook-repositories
silencer-playbooks

List of All Triggers and Actions
---------------------------------
Expand Down
Loading
Loading