Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docs/configuration/holmesgpt/toolsets/coralogix_logs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ Configuration
Advanced Configuration
^^^^^^^^^^^^^^^^^^^^^^

**Frequent logs and archive**
Frequent logs and archive
****************************

By default, holmes fetched the logs from the `Frequent search <https://coralogix.com/docs/user-guides/account-management/tco-optimizer/logs/#frequent-search-data-high-priority>`_
tier and only fetch logs from the `Archive` tier if the frequent search returned no result.
Expand Down Expand Up @@ -98,7 +99,8 @@ Here is a description of each possible log retrieval methodology:
- **FREQUENT_SEARCH_FALLBACK** Search logs in the archive first. If there are no results, fallback to searching the frequent logs.
- **BOTH_FREQUENT_SEARCH_AND_ARCHIVE** Always use both the frequent search and the archive to fetch logs. The result contains merged data which is deduplicated and sorted by timestamp.

**Search labels**
Search labels
***************

You can tweak the labels used by the toolset to identify kubernetes resources. This is **optional** and only needed if your
logs settings for ``pod``, ``namespace``, ``application`` and ``subsystem`` differ from the defaults in the example below.
Expand All @@ -124,7 +126,8 @@ You can verify what labels to use by attempting to run a query in the coralogix
:align: center


**Disabling the default toolset**
Disabling the default toolset
*********************************

If Coralogix is your primary datasource for logs, it is **advised** to disable the default HolmesGPT logging
tool by disabling the ``kubernetes/logs`` toolset. Without this. HolmesGPT may still use kubectl to
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/holmesgpt/toolsets/grafanaloki.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ You can find the Grafana URL required for Loki in your Grafana cloud account set


Obtaining the datasource UID
-----------------------
-----------------------------------

You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to
fetch the logs and it needs to know the UID of this datasource.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/holmesgpt/toolsets/grafanatempo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ You can find the Grafana URL required for Tempo in your Grafana cloud account se
:align: center

Obtaining the datasource UID
-----------------------
--------------------------------

You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to
fetch the traces and it needs to know the UID of this datasource.
Expand Down
8 changes: 4 additions & 4 deletions docs/configuration/holmesgpt/toolsets/notion.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@ Notion
=======================

Notion Integration for HolmesGPT
-------------
---------------------------------------

Enabling this toolset allows HolmesGPT to fetch pages from Notion, making it useful when providing Notion-based runbooks.

Setup Instructions
-------------
-----------------------

1. **Create a Webhook Integration**
- Go to the Notion Developer Portal.
Expand All @@ -25,7 +25,7 @@ Setup Instructions


Configuration
-------------
------------------

.. code-block:: yaml

Expand All @@ -46,7 +46,7 @@ Configuration
.. include:: ./_toolset_configuration.inc.rst

Capabilities
------------
----------------
.. include:: ./_toolset_capabilities.inc.rst

.. list-table::
Expand Down
12 changes: 7 additions & 5 deletions docs/configuration/holmesgpt/toolsets/prometheus.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
.. _toolset_prometheus:

Prometheus
==========
=============

By enabling this toolset, HolmesGPT will be able to generate graphs from prometheus metrics as well as help you write and
validate prometheus queries. HolmesGPT can also detect memory leak patterns, CPU throttling, lagging queues, and high
latency issues.

Prior to generating a PromQL query, HolmesQPT tends to list the available metrics. This is done to ensure the metrics used
Prior to generating a PromQL query, HolmesGPT tends to list the available metrics. This is done to ensure the metrics used
in PromQL are actually available.

Configuration
Expand Down Expand Up @@ -47,7 +47,8 @@ Configuration

It is also possible to set the ``PROMETHEUS_URL`` environment variable instead of the above ``prometheus_url`` config key.

**Advanced configuration**
Advanced configuration
******************************************

Below is the full list of options for this toolset:

Expand All @@ -74,7 +75,8 @@ Below is the full list of options for this toolset:
- **fetch_metadata_with_series_api** Uses the `series API <https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers>`_ instead of the `metadata API <https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata>`_. You should only set this value to `true` if the metadata API is disabled or not working. HolmesGPT's ability to select the right metric will be negatively impacted because the series API does not return key metadata like the metrics/series description or their type (gauge, histogram, etc.).
- **tool_calls_return_data** Defaults to ``true``. If ``false``, no prometheus data will be returned to HolmesGPT. Set it to ``false`` if you frequently reach the token limit when using this toolset. Setting this setting to ``false`` will also disable HolmesGPT's ability to analyze prometheus data.

**Finding the prometheus URL**
Finding the prometheus URL
******************************************

The best way to find the prometheus URL is to use "ask holmes". This only works if your cluster is live and already connected to Robusta.

Expand All @@ -85,7 +87,7 @@ If not, you can often find the prometheus URL by running the following command (
kubectl get svc --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"."}{.metadata.namespace}{".svc.cluster.local:"}{.spec.ports[0].port}{"\n"}{end}' | grep prometheus | grep -Ev 'operat|alertmanager|node|coredns|kubelet|kube-scheduler|etcd|controller' | awk '{print "http://"$1}'

Capabilities
------------
-----------------
.. include:: ./_toolset_capabilities.inc.rst

.. list-table::
Expand Down
2 changes: 1 addition & 1 deletion docs/how-it-works/architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Bundled Prometheus (Optional)

Install Robusta with :ref:`Prometheus included <embedded Prometheus stack>`. This is powered by ``kube-prometheus-stack``.

Alternatively, you can :ref:`integrate an existing Prometheus with Robusta <Integrating AlertManager and Prometheus>`.
Alternatively, you can :ref:`integrate an existing Prometheus with Robusta <Integrating with Prometheus>`.

Web UI (Optional)
^^^^^^^^^^^^^^^^^^^^^^
Expand Down
4 changes: 2 additions & 2 deletions docs/how-it-works/coverage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Prometheus Alerts

.. warning::

You must :ref:`send your Prometheus alerts to Robusta by webhook <Integrating AlertManager and Prometheus>` for these to work.
You must :ref:`send your Prometheus alerts to Robusta by webhook <Integrating with Prometheus>` for these to work.

Other errors
----------------
Expand All @@ -39,7 +39,7 @@ Change Tracking
By default all changes to Deployments, DaemonSets, and StatefulSets are sent to the Robusta UI for correlation
with Prometheus alerts and other errors.

These changes are not sent to other sinks (e.g. Slack) by default because they are spammy. :ref:`Automation Basics`
These changes are not sent to other sinks (e.g. Slack) by default because they are spammy. :ref:`Routing Cookbook`
explains how to selectively track changes you care about in Slack as well.

We also wrote a blog post `Why everyone should track Kubernetes changes and top four ways to do so <https://home.robusta.dev/blog/why-everyone-should-track-and-audit-kubernetes-changes-and-top-ways/>`_
Expand Down
62 changes: 62 additions & 0 deletions docs/notification-routing/notification-routing-examples.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
:hide-toc:

Routing Cookbook
===================================

Expand All @@ -15,3 +17,63 @@ Routing Cookbook
disable-oomkill-notifications

In this section you'll find example configurations for common routing patterns.


.. grid:: 1 1 2 3
:gutter: 3

.. grid-item-card:: :octicon:`book;1em;` Routing by Namespace
:class-card: sd-bg-light sd-bg-text-light
:link: routing-by-namespace
:link-type: doc

Route notifications based on Kubernetes namespaces.

.. grid-item-card:: :octicon:`book;1em;` Routing by Alert Name
:class-card: sd-bg-light sd-bg-text-light
:link: routing-by-type
:link-type: doc

Route notifications based on alert types.

.. grid-item-card:: :octicon:`book;1em;` Route by Time of Day
:class-card: sd-bg-light sd-bg-text-light
:link: implementing-monitoring-shifts
:link-type: doc

Implement monitoring shifts for better alert management.

.. grid-item-card:: :octicon:`book;1em;` Routing to Multiple Slack Channels
:class-card: sd-bg-light sd-bg-text-light
:link: routing-to-multiple-slack-channels
:link-type: doc

Send notifications to multiple Slack channels.

.. grid-item-card:: :octicon:`book;1em;` Routing Exclusion
:class-card: sd-bg-light sd-bg-text-light
:link: routing-exclusion
:link-type: doc

Exclude specific alerts from being routed.

.. grid-item-card:: :octicon:`book;1em;` Dropping Specific Alerts
:class-card: sd-bg-light sd-bg-text-light
:link: routing-by-severity
:link-type: doc

Route notifications based on alert severity.

.. grid-item-card:: :octicon:`book;1em;` Excluding "Resolved" Notifications
:class-card: sd-bg-light sd-bg-text-light
:link: excluding-resolved
:link-type: doc

Exclude resolved alerts from notifications.

.. grid-item-card:: :octicon:`book;1em;` Disable "OOMKill" Notifications
:class-card: sd-bg-light sd-bg-text-light
:link: disable-oomkill-notifications
:link-type: doc

Disable notifications for OOMKill events.
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Trigger the newly added playbook by simulating a Prometheus alert.


Reference Alert Metadata in Remediation Jobs
-------------------------------------------
--------------------------------------------------

When remediating based on alerts, you can access all the alert metadata like name, namespace, cluster name, pod, node and more as environment variables.

Expand Down
Loading
Loading