Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,35 @@
"tutorials/alert-custom-prometheus.html": "/master/configuration/alertmanager-integration/alert-custom-prometheus.html",
"catalog/triggers/prometheus.html": "/master/configuration/alertmanager-integration/index.html",
"playbook-reference/prometheus-examples/alert-remediation.html": "/master/playbook-reference/automatic-remediation-examples/index.html",
"configuration/ai-analysis.html": "/master/configuration/holmesgpt/index.html",
"configuration/ai-analysis.html": "/master/configuration/holmesgpt/main-features.html",
"configuration/holmesgpt/index.html": "/master/configuration/holmesgpt/main-features.html",
# AI Analysis pages redirects to holmesgpt.dev (docs have moved there)
"configuration/holmesgpt/builtin_toolsets.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/",
"configuration/holmesgpt/permissions.html": "https://holmesgpt.dev/data-sources/permissions/",
"configuration/holmesgpt/custom_toolsets.html": "https://holmesgpt.dev/data-sources/custom-toolsets/",
"configuration/holmesgpt/remote_mcp_servers.html": "https://holmesgpt.dev/data-sources/remote-mcp-servers/",
# Individual toolset page redirects
"configuration/holmesgpt/toolsets/argocd.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/argocd/",
"configuration/holmesgpt/toolsets/aws.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/aws/",
"configuration/holmesgpt/toolsets/confluence.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/confluence/",
"configuration/holmesgpt/toolsets/coralogix_logs.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/coralogix-logs/",
"configuration/holmesgpt/toolsets/datadog_logs.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/datadog/",
"configuration/holmesgpt/toolsets/datetime.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/datetime/",
"configuration/holmesgpt/toolsets/docker.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/docker/",
"configuration/holmesgpt/toolsets/grafanaloki.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/grafanaloki/",
"configuration/holmesgpt/toolsets/grafanatempo.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/grafanatempo/",
"configuration/holmesgpt/toolsets/helm.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/helm/",
"configuration/holmesgpt/toolsets/internet.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/internet/",
"configuration/holmesgpt/toolsets/kafka.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/kafka/",
"configuration/holmesgpt/toolsets/kubernetes.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/kubernetes/",
"configuration/holmesgpt/toolsets/newrelic.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/newrelic/",
"configuration/holmesgpt/toolsets/notion.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/notion/",
"configuration/holmesgpt/toolsets/opensearch_logs.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/opensearch-logs/",
"configuration/holmesgpt/toolsets/opensearch_status.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/opensearch-status/",
"configuration/holmesgpt/toolsets/prometheus.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/prometheus/",
"configuration/holmesgpt/toolsets/rabbitmq.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/rabbitmq/",
"configuration/holmesgpt/toolsets/robusta.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/robusta/",
"configuration/holmesgpt/toolsets/slab.html": "https://holmesgpt.dev/data-sources/builtin-toolsets/slab/",
"coverage.html": "/master/how-it-works/coverage.html",
"tutorials/python-profiling.html": "/master/playbook-reference/actions/python-troubleshooting.html#python-profiler",
"tutorials/more-tutorials.html": "/master/community-tutorials.html",
Expand Down
126 changes: 63 additions & 63 deletions docs/configuration/alertmanager-integration/_alertmanager-config.rst
Original file line number Diff line number Diff line change
@@ -1,63 +1,63 @@
.. admonition:: AlertManager config for sending alerts to Robusta

.. tab-set::

.. tab-item:: kube-prometheus-stack (Prometheus Operator)

Add the following to your `AlertManager's config Secret <https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/alerting.md#managing-alertmanager-configuration>`_

Do not apply in other ways, they all `have limitations <https://github.com/prometheus-operator/prometheus-operator/issues/3750>`_ and won't forward all alerts.

.. code-block:: yaml

receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
send_resolved: true # (3)
- name: 'default-receiver'

route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (2)
receiver: 'default-receiver'

.. code-annotations::
1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set ``continue: true`` set.
2. Keep sending alerts to receivers defined after Robusta.
3. Important, so Robusta knows when alerts are resolved.

.. tab-item:: Other Prometheus Installations

Add the following to your AlertManager configuration, wherever it is defined.

.. code-block:: yaml

receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
send_resolved: true # (3)

route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (2)

.. code-annotations::
1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set ``continue: true`` set.
2. Keep sending alerts to receivers defined after Robusta.
3. Important, so Robusta knows when alerts are resolved.
Configure your AlertManager to send alerts to Robusta:

.. tab-set::

.. tab-item:: kube-prometheus-stack (Prometheus Operator)

Add the following to your `AlertManager's config Secret <https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/alerting.md#managing-alertmanager-configuration>`_.

Do not apply in other ways, they all `have limitations <https://github.com/prometheus-operator/prometheus-operator/issues/3750>`_ and won't forward all alerts.

.. code-block:: yaml

receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
send_resolved: true # (3)
- name: 'default-receiver'

route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (2)
receiver: 'default-receiver'

.. code-annotations::
1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set ``continue: true``.
2. Keep sending alerts to receivers defined after Robusta.
3. Important, so Robusta knows when alerts are resolved.

.. tab-item:: Other Prometheus Installations

Add the following to your AlertManager configuration, wherever it is defined.

.. code-block:: yaml

receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
send_resolved: true # (3)

route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (2)

Comment thread
pavangudiwada marked this conversation as resolved.
.. code-annotations::
1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set ``continue: true``.
2. Keep sending alerts to receivers defined after Robusta.
3. Important, so Robusta knows when alerts are resolved.
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@

Robusta utilizes the flags API to retrieve data from Prometheus-style metric stores. However, some platforms like Google Managed Prometheus, Azure Managed Prometheus etc, do not implement the flags API.

You can disable the Prometheus flags API check by setting the following option to ``false``.
You can disable the Prometheus flags API check by setting the following option to false.

.. code-block:: yaml

globalConfig:
check_prometheus_flags: true/false
globalConfig:
check_prometheus_flags: true/false
79 changes: 4 additions & 75 deletions docs/configuration/alertmanager-integration/_pull_integration.rst
Original file line number Diff line number Diff line change
@@ -1,81 +1,10 @@
Configure Metric Querying
====================================

Metrics querying lets Robusta pull metrics and create silences.
To enable Robusta to pull metrics and create silences, you need to configure Prometheus and AlertManager URLs.

If Robusta fails to auto-detect the Prometheus and Alertmanager urls - and you see related connection errors in the logs - configure the ``prometheus_url`` and ``alertmanager_url`` in your Helm values and :ref:`update Robusta <Simple Upgrade>`
See :doc:`Prometheus and metrics configuration </configuration/metric-providers-in-cluster>` for detailed instructions.

.. code-block:: yaml
.. note::

globalConfig: # This line should already exist
# Add the lines below
alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093" # (1)
prometheus_url: "http://PROMETHEUS_SERVICE_NAME.NAMESPACE.svc.cluster.local:9090" # (2)

# If Prometheus has data for multiple clusters, tell Robusta how to query data for this cluster only
# prometheus_additional_labels:
# cluster: 'CLUSTER_NAME_HERE'

# If using Grafana alerts, add this too
# grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (3)
# alertmanager_flavor: grafana

# If necessary, see docs below
# prometheus_auth: ...
# alertmanager_auth: ...

# If using a multi-tenant prometheus or alertmanager, pass the org id to all queries
# prometheus_additional_headers:
# X-Scope-OrgID: <org id>
# alertmanager_additional_headers:
# X-Scope-OrgID: <org id>

.. code-annotations::
1. Example: http://alertmanager-Helm_release_name-kube-prometheus-alertmanager.default.svc.cluster.local:9093.
2. Example: http://Helm_Release_Name-kube-prometheus-prometheus.default.svc.cluster.local:9090
3. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.

You can optionally setup authentication, SSL verification, and other parameters described below.

Verify it Works
^^^^^^^^^^^^^^^^^
Open any application in the Robusta UI. If CPU and memory graphs are shown, everything is working.

If you don't use the Robusta UI, trigger a `demo OOMKill alert <https://github.com/robusta-dev/kubernetes-demos/#oomkilled-pod-out-of-memory-kill>`_,
and verify that Robusta sends a Slack/Teams message with a memory graph included. If so, everything is configured properly.

Optional Settings
=============================

Authentication Headers
^^^^^^^^^^^^^^^^^^^^^^^^^^

If Prometheus and/or AlertManager require authentication, add the following to ``generated_values.yaml``:

.. code-block:: yaml

globalConfig:
prometheus_auth: Bearer <YOUR TOKEN> # Replace <YOUR TOKEN> with your actual token or use any other auth header as needed
alertmanager_auth: Basic <USER:PASSWORD base64-encoded> # Replace <USER:PASSWORD base64-encoded> with your actual credentials, base64-encoded, or use any other auth header as needed

These settings may be configured independently.

SSL Verification
^^^^^^^^^^^^^^^^^^^^
By default, Robusta does not verify the SSL certificate of the Prometheus server.

To enable SSL verification, add the following to Robusta's ``generated_values.yaml``:

.. code-block:: yaml

runner:
additional_env_vars:
- name: PROMETHEUS_SSL_ENABLED
value: "true"

If you have a custom Certificate Authority (CA) certificate, add one more setting:

.. code-block:: yaml

runner:
certificate: "<YOUR BASE-64 ENCODED DATA>" # base64-encoded certificate value
Robusta will attempt to auto-detect Prometheus and AlertManager URLs in your cluster. Manual configuration is only needed if auto-detection fails.
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,4 @@ If everything is setup properly, this alert will reach Robusta. It will show up

Robusta enriches alerts with Kubernetes and log data using Prometheus labels for mapping.
Standard label names are used by default. If your setup differs, you can
:ref:`customize this mapping <Relabel Prometheus Alerts>` to fit your environment.
`customize this mapping </setup-robusta/additional-settings.html#alert-label-mapping>`_ to fit your environment.
10 changes: 3 additions & 7 deletions docs/configuration/alertmanager-integration/alert-manager.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
In-cluster Prometheus
In-cluster AlertManager Integration
****************************************

Here's how to integrate an existing Prometheus with Robusta in the same cluster:
This guide shows how to send alerts from an existing AlertManager to Robusta in the same cluster.

* Send alerts to Robusta by adding a receiver to AlertManager
* Point Robusta at Prometheus so it can query metrics and silence alerts
* Robusta will attempt auto-detection, so this is not always necessary!

If your Prometheus is in a different cluster, refer to :ref:`External Prometheus`.
If your AlertManager is in a different cluster, refer to :ref:`External Prometheus`.
Comment thread
pavangudiwada marked this conversation as resolved.

Send Alerts to Robusta
============================
Expand Down
Loading