|
| 1 | +.. _toolset_prometheus: |
| 2 | + |
| 3 | +Prometheus |
| 4 | +========== |
| 5 | + |
| 6 | +By enabling this toolset, HolmesGPT will be able to generate graphs from prometheus metrics as well as help you write and |
| 7 | +validate prometheus queries. |
| 8 | + |
| 9 | +There is also an option for Holmes to analyze prometheus metrics. When enabled, HolmesGPT can detect memory leak patterns, |
| 10 | +CPU throttling, high latency for your APIs, etc. The configuration field to enable prometheus metrics analysis is |
| 11 | +``tool_calls_return_data``. |
| 12 | + |
| 13 | +Configuration |
| 14 | +------------- |
| 15 | + |
| 16 | +.. md-tab-set:: |
| 17 | + |
| 18 | + .. md-tab-item:: Robusta Helm Chat |
| 19 | + |
| 20 | + .. code-block:: yaml |
| 21 | +
|
| 22 | + holmes: |
| 23 | + toolsets: |
| 24 | + prometheus/metrics: |
| 25 | + enabled: true |
| 26 | + config: |
| 27 | + prometheus_url: ... |
| 28 | + metrics_labels_time_window_hrs: 48 # default value |
| 29 | + metrics_labels_cache_duration_hrs: 12 # default value |
| 30 | + fetch_labels_with_labels_api: false # default value |
| 31 | + fetch_metadata_with_series_api: false # default value |
| 32 | + tool_calls_return_data: false # default value |
| 33 | + headers: |
| 34 | + Authorization: "Basic <base_64_encoded_string>" |
| 35 | +
|
| 36 | +
|
| 37 | + .. include:: ./_toolset_configuration.inc.rst |
| 38 | + |
| 39 | + .. md-tab-item:: Holmes CLI |
| 40 | + |
| 41 | + Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist: |
| 42 | + |
| 43 | + .. code-block:: yaml |
| 44 | +
|
| 45 | + toolsets: |
| 46 | + prometheus/metrics: |
| 47 | + enabled: true |
| 48 | + config: |
| 49 | + prometheus_url: ... |
| 50 | + metrics_labels_time_window_hrs: 48 # default value |
| 51 | + metrics_labels_cache_duration_hrs: 12 # default value |
| 52 | + fetch_labels_with_labels_api: false # default value |
| 53 | + fetch_metadata_with_series_api: false # default value |
| 54 | + tool_calls_return_data: false # default value |
| 55 | + headers: |
| 56 | + Authorization: "Basic <base_64_encoded_string>" |
| 57 | +
|
| 58 | +It is also possible to set the ``PROMETHEUS_URL`` environment variable instead of the above ``prometheus_url`` config key. |
| 59 | + |
| 60 | +Prior to generating a PromQL query, HolmesQPT tends to list the available metrics. This is done to ensure the metrics used |
| 61 | +in PromQL are actually available. |
| 62 | + |
| 63 | +Below is the full list of options for this toolset: |
| 64 | + |
| 65 | +- **metrics_labels_time_window_hrs** Represents the time window, in hours, over which labels are fetched. This avoids fetching obsolete labels. Set it to ``null`` to let HolmesGPT fetch labels regardless of when they were generated. |
| 66 | +- **metrics_labels_cache_duration_hrs** How long are labels cached, in hours. Set it to ``null`` to disable caching. |
| 67 | +- **fetch_labels_with_labels_api** Uses prometheus `labels API <https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names>`_ to fetch labels instead of the `series API <https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers>`_. In some cases setting to True can improve the performance of the toolset, however there will be an increased number of HTTP calls to prometheus. You can experiment with both as they are functionally identical. |
| 68 | +- **fetch_metadata_with_series_api** Uses the `series API <https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers>`_ instead of the `metadata API <https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata>`_. You should only set this value to `true` if the metadata API is disabled or not working. HolmesGPT's ability to select the right metric will be negatively impacted because the series API does not return key metadata like the metrics/series description or their type (gauge, histogram, etc.). |
| 69 | +- **tool_calls_return_data** Experimental. If true, the prometheus data will be available to HolmesGPT. In some cases, HolmesGPT will be able to detect memory leaks or other anomalies. This is disabled by default to reduce the likelyhood of reaching the input token limit. |
| 70 | +- **headers** Extra headers to pass to all prometheus http requests. Use this to pass authentication. Prometheus `supports basic authentication <https://prometheus.io/docs/guides/basic-auth/>`_. |
| 71 | + |
| 72 | +Capabilities |
| 73 | +------------ |
| 74 | +.. include:: ./_toolset_capabilities.inc.rst |
| 75 | + |
| 76 | +.. list-table:: |
| 77 | + :header-rows: 1 |
| 78 | + :widths: 30 70 |
| 79 | + |
| 80 | + * - Tool Name |
| 81 | + - Description |
| 82 | + * - list_available_metrics |
| 83 | + - List all the available metrics to query from prometheus, including their types (counter, gauge, histogram, summary) and available labels. |
| 84 | + * - execute_prometheus_instant_query |
| 85 | + - Execute an instant PromQL query |
| 86 | + * - execute_prometheus_range_query |
| 87 | + - Execute a PromQL range query |
0 commit comments