From 4ec919490d6157fbe00a434f168534700c9a9995 Mon Sep 17 00:00:00 2001 From: Pavan Gudiwada Date: Thu, 8 May 2025 08:16:40 +0530 Subject: [PATCH 1/2] Warning and navigation fixes --- .../holmesgpt/toolsets/coralogix_logs.rst | 9 +- .../holmesgpt/toolsets/grafanaloki.rst | 2 +- .../holmesgpt/toolsets/grafanatempo.rst | 2 +- .../holmesgpt/toolsets/notion.rst | 8 +- .../holmesgpt/toolsets/prometheus.rst | 12 +- docs/how-it-works/architecture.rst | 2 +- docs/how-it-works/coverage.rst | 4 +- .../notification-routing-examples.rst | 62 +++++ .../job-to-remediate-alert.rst | 2 +- .../logs-triggers/index.rst | 258 +++++++++--------- 10 files changed, 214 insertions(+), 147 deletions(-) diff --git a/docs/configuration/holmesgpt/toolsets/coralogix_logs.rst b/docs/configuration/holmesgpt/toolsets/coralogix_logs.rst index bc926f7b9..1d5de74a8 100644 --- a/docs/configuration/holmesgpt/toolsets/coralogix_logs.rst +++ b/docs/configuration/holmesgpt/toolsets/coralogix_logs.rst @@ -68,7 +68,8 @@ Configuration Advanced Configuration ^^^^^^^^^^^^^^^^^^^^^^ -**Frequent logs and archive** +Frequent logs and archive +**************************** By default, holmes fetched the logs from the `Frequent search `_ tier and only fetch logs from the `Archive` tier if the frequent search returned no result. @@ -98,7 +99,8 @@ Here is a description of each possible log retrieval methodology: - **FREQUENT_SEARCH_FALLBACK** Search logs in the archive first. If there are no results, fallback to searching the frequent logs. - **BOTH_FREQUENT_SEARCH_AND_ARCHIVE** Always use both the frequent search and the archive to fetch logs. The result contains merged data which is deduplicated and sorted by timestamp. -**Search labels** +Search labels +*************** You can tweak the labels used by the toolset to identify kubernetes resources. This is **optional** and only needed if your logs settings for ``pod``, ``namespace``, ``application`` and ``subsystem`` differ from the defaults in the example below. @@ -124,7 +126,8 @@ You can verify what labels to use by attempting to run a query in the coralogix :align: center -**Disabling the default toolset** +Disabling the default toolset +********************************* If Coralogix is your primary datasource for logs, it is **advised** to disable the default HolmesGPT logging tool by disabling the ``kubernetes/logs`` toolset. Without this. HolmesGPT may still use kubectl to diff --git a/docs/configuration/holmesgpt/toolsets/grafanaloki.rst b/docs/configuration/holmesgpt/toolsets/grafanaloki.rst index 26e63b14c..730373936 100644 --- a/docs/configuration/holmesgpt/toolsets/grafanaloki.rst +++ b/docs/configuration/holmesgpt/toolsets/grafanaloki.rst @@ -35,7 +35,7 @@ You can find the Grafana URL required for Loki in your Grafana cloud account set Obtaining the datasource UID ------------------------ +----------------------------------- You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to fetch the logs and it needs to know the UID of this datasource. diff --git a/docs/configuration/holmesgpt/toolsets/grafanatempo.rst b/docs/configuration/holmesgpt/toolsets/grafanatempo.rst index 3e8217a57..928c18ff7 100644 --- a/docs/configuration/holmesgpt/toolsets/grafanatempo.rst +++ b/docs/configuration/holmesgpt/toolsets/grafanatempo.rst @@ -25,7 +25,7 @@ You can find the Grafana URL required for Tempo in your Grafana cloud account se :align: center Obtaining the datasource UID ------------------------ +-------------------------------- You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to fetch the traces and it needs to know the UID of this datasource. diff --git a/docs/configuration/holmesgpt/toolsets/notion.rst b/docs/configuration/holmesgpt/toolsets/notion.rst index 0ce7310af..203d87028 100644 --- a/docs/configuration/holmesgpt/toolsets/notion.rst +++ b/docs/configuration/holmesgpt/toolsets/notion.rst @@ -2,12 +2,12 @@ Notion ======================= Notion Integration for HolmesGPT -------------- +--------------------------------------- Enabling this toolset allows HolmesGPT to fetch pages from Notion, making it useful when providing Notion-based runbooks. Setup Instructions -------------- +----------------------- 1. **Create a Webhook Integration** - Go to the Notion Developer Portal. @@ -25,7 +25,7 @@ Setup Instructions Configuration -------------- +------------------ .. code-block:: yaml @@ -46,7 +46,7 @@ Configuration .. include:: ./_toolset_configuration.inc.rst Capabilities ------------- +---------------- .. include:: ./_toolset_capabilities.inc.rst .. list-table:: diff --git a/docs/configuration/holmesgpt/toolsets/prometheus.rst b/docs/configuration/holmesgpt/toolsets/prometheus.rst index 1e58104e5..5b900b8db 100644 --- a/docs/configuration/holmesgpt/toolsets/prometheus.rst +++ b/docs/configuration/holmesgpt/toolsets/prometheus.rst @@ -1,13 +1,13 @@ .. _toolset_prometheus: Prometheus -========== +============= By enabling this toolset, HolmesGPT will be able to generate graphs from prometheus metrics as well as help you write and validate prometheus queries. HolmesGPT can also detect memory leak patterns, CPU throttling, lagging queues, and high latency issues. -Prior to generating a PromQL query, HolmesQPT tends to list the available metrics. This is done to ensure the metrics used +Prior to generating a PromQL query, HolmesGPT tends to list the available metrics. This is done to ensure the metrics used in PromQL are actually available. Configuration @@ -47,7 +47,8 @@ Configuration It is also possible to set the ``PROMETHEUS_URL`` environment variable instead of the above ``prometheus_url`` config key. -**Advanced configuration** +Advanced configuration +****************************************** Below is the full list of options for this toolset: @@ -74,7 +75,8 @@ Below is the full list of options for this toolset: - **fetch_metadata_with_series_api** Uses the `series API `_ instead of the `metadata API `_. You should only set this value to `true` if the metadata API is disabled or not working. HolmesGPT's ability to select the right metric will be negatively impacted because the series API does not return key metadata like the metrics/series description or their type (gauge, histogram, etc.). - **tool_calls_return_data** Defaults to ``true``. If ``false``, no prometheus data will be returned to HolmesGPT. Set it to ``false`` if you frequently reach the token limit when using this toolset. Setting this setting to ``false`` will also disable HolmesGPT's ability to analyze prometheus data. -**Finding the prometheus URL** +Finding the prometheus URL +****************************************** The best way to find the prometheus URL is to use "ask holmes". This only works if your cluster is live and already connected to Robusta. @@ -85,7 +87,7 @@ If not, you can often find the prometheus URL by running the following command ( kubectl get svc --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"."}{.metadata.namespace}{".svc.cluster.local:"}{.spec.ports[0].port}{"\n"}{end}' | grep prometheus | grep -Ev 'operat|alertmanager|node|coredns|kubelet|kube-scheduler|etcd|controller' | awk '{print "http://"$1}' Capabilities ------------- +----------------- .. include:: ./_toolset_capabilities.inc.rst .. list-table:: diff --git a/docs/how-it-works/architecture.rst b/docs/how-it-works/architecture.rst index 839f314b0..249afb441 100644 --- a/docs/how-it-works/architecture.rst +++ b/docs/how-it-works/architecture.rst @@ -21,7 +21,7 @@ Bundled Prometheus (Optional) Install Robusta with :ref:`Prometheus included `. This is powered by ``kube-prometheus-stack``. -Alternatively, you can :ref:`integrate an existing Prometheus with Robusta `. +Alternatively, you can :ref:`integrate an existing Prometheus with Robusta `. Web UI (Optional) ^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/how-it-works/coverage.rst b/docs/how-it-works/coverage.rst index a885f1ef2..295a1b8cf 100644 --- a/docs/how-it-works/coverage.rst +++ b/docs/how-it-works/coverage.rst @@ -20,7 +20,7 @@ Prometheus Alerts .. warning:: - You must :ref:`send your Prometheus alerts to Robusta by webhook ` for these to work. + You must :ref:`send your Prometheus alerts to Robusta by webhook ` for these to work. Other errors ---------------- @@ -39,7 +39,7 @@ Change Tracking By default all changes to Deployments, DaemonSets, and StatefulSets are sent to the Robusta UI for correlation with Prometheus alerts and other errors. -These changes are not sent to other sinks (e.g. Slack) by default because they are spammy. :ref:`Automation Basics` +These changes are not sent to other sinks (e.g. Slack) by default because they are spammy. :ref:`Routing Cookbook` explains how to selectively track changes you care about in Slack as well. We also wrote a blog post `Why everyone should track Kubernetes changes and top four ways to do so `_ diff --git a/docs/notification-routing/notification-routing-examples.rst b/docs/notification-routing/notification-routing-examples.rst index 916343060..8338094d0 100644 --- a/docs/notification-routing/notification-routing-examples.rst +++ b/docs/notification-routing/notification-routing-examples.rst @@ -1,3 +1,5 @@ +:hide-toc: + Routing Cookbook =================================== @@ -15,3 +17,63 @@ Routing Cookbook disable-oomkill-notifications In this section you'll find example configurations for common routing patterns. + + +.. grid:: 1 1 2 3 + :gutter: 3 + + .. grid-item-card:: :octicon:`book;1em;` Routing by Namespace + :class-card: sd-bg-light sd-bg-text-light + :link: routing-by-namespace + :link-type: doc + + Route notifications based on Kubernetes namespaces. + + .. grid-item-card:: :octicon:`book;1em;` Routing by Alert Name + :class-card: sd-bg-light sd-bg-text-light + :link: routing-by-type + :link-type: doc + + Route notifications based on alert types. + + .. grid-item-card:: :octicon:`book;1em;` Route by Time of Day + :class-card: sd-bg-light sd-bg-text-light + :link: implementing-monitoring-shifts + :link-type: doc + + Implement monitoring shifts for better alert management. + + .. grid-item-card:: :octicon:`book;1em;` Routing to Multiple Slack Channels + :class-card: sd-bg-light sd-bg-text-light + :link: routing-to-multiple-slack-channels + :link-type: doc + + Send notifications to multiple Slack channels. + + .. grid-item-card:: :octicon:`book;1em;` Routing Exclusion + :class-card: sd-bg-light sd-bg-text-light + :link: routing-exclusion + :link-type: doc + + Exclude specific alerts from being routed. + + .. grid-item-card:: :octicon:`book;1em;` Dropping Specific Alerts + :class-card: sd-bg-light sd-bg-text-light + :link: routing-by-severity + :link-type: doc + + Route notifications based on alert severity. + + .. grid-item-card:: :octicon:`book;1em;` Excluding "Resolved" Notifications + :class-card: sd-bg-light sd-bg-text-light + :link: excluding-resolved + :link-type: doc + + Exclude resolved alerts from notifications. + + .. grid-item-card:: :octicon:`book;1em;` Disable "OOMKill" Notifications + :class-card: sd-bg-light sd-bg-text-light + :link: disable-oomkill-notifications + :link-type: doc + + Disable notifications for OOMKill events. \ No newline at end of file diff --git a/docs/playbook-reference/automatic-remediation-examples/job-to-remediate-alert.rst b/docs/playbook-reference/automatic-remediation-examples/job-to-remediate-alert.rst index 04d17313f..30d54ff31 100644 --- a/docs/playbook-reference/automatic-remediation-examples/job-to-remediate-alert.rst +++ b/docs/playbook-reference/automatic-remediation-examples/job-to-remediate-alert.rst @@ -35,7 +35,7 @@ Trigger the newly added playbook by simulating a Prometheus alert. Reference Alert Metadata in Remediation Jobs -------------------------------------------- +-------------------------------------------------- When remediating based on alerts, you can access all the alert metadata like name, namespace, cluster name, pod, node and more as environment variables. diff --git a/docs/playbook-reference/logs-triggers/index.rst b/docs/playbook-reference/logs-triggers/index.rst index 02b5648e0..1a591e11d 100644 --- a/docs/playbook-reference/logs-triggers/index.rst +++ b/docs/playbook-reference/logs-triggers/index.rst @@ -1,4 +1,4 @@ -:hide-toc: +.. :hide-toc: Automate Responses to Application Logs ========================================== @@ -13,7 +13,7 @@ To achieve this, we’ll use: - **Prometheus**: Stores the metrics and triggers alerts based on them. (We're going to use Robusta's bundled Prometheus here, but it can work with any other Prometheus distribution) - **Robusta**: Executes an automated playbook when an alert is fired, such as restarting the affected pod. - .. image:: /images/logs-to-metrics.png + .. image:: /images/logs-to-metrics.png Let's get started! Here's a video explainer of how it works: @@ -24,10 +24,10 @@ Let's get started! Here's a video explainer of how it works: **Step 1: Create a namespace for the demo** ---------------------------------------------------- - .. code-block:: yaml +.. code-block:: yaml - kubectl create namespace log-triggers - kubectl config set-context --current --namespace log-triggers + kubectl create namespace log-triggers + kubectl config set-context --current --namespace log-triggers **Step 2: Parse Logs into Metrics with Fluent Bit** ---------------------------------------------------- @@ -40,136 +40,136 @@ First, lets configure Fluent Bit to monitor your pod logs and generate Prometheu This is our Fluent Bit configuration (``fluentbit-values.yaml``): - .. code-block:: yaml - - config: - service: | - [SERVICE] - Flush 1 - Daemon Off - Log_Level info - HTTP_Server On - HTTP_Listen 0.0.0.0 - HTTP_Port 2020 - - inputs: | - [INPUT] - Name tail - Tag kube.* - Path /var/log/containers/*.log - Parser json_message - DB /var/log/flb_kube.db - Mem_Buf_Limit 5MB - Skip_Long_Lines On - Refresh_Interval 10 - - [INPUT] - Name dummy - Tag dummy.alive - Dummy {"log":"keepalive"} - - parsers: | - [PARSER] - Name wrap_raw_line - Format regex - Regex ^(?.*)$ - - filters: | - [FILTER] - Name kubernetes - Match kube.* - K8S-Logging.Parser On - K8S-Logging.Exclude On - - [FILTER] - name log_to_metrics - match * - tag log_metrics - metric_mode counter - metric_name mysql_connection_error - metric_description MySql connection errors - regex log .*mysql connection error.* - add_label pod $kubernetes['pod_name'] - add_label namespace $kubernetes['namespace_name'] - add_label container $kubernetes['container_name'] - - [FILTER] - name log_to_metrics - match * - tag log_metrics - metric_mode counter - metric_name dns_error - metric_description DNS Resolution errors - regex log .*dns error.* - add_label pod $kubernetes['pod_name'] - add_label namespace $kubernetes['namespace_name'] - add_label container $kubernetes['container_name'] - - [FILTER] - Name log_to_metrics - Match dummy.alive - Metric_Name fluentbit_keepalive - Metric_Description Dummy metric to keep /metrics available - Metric_Mode counter - Tag log_metrics - Regex log .*keepalive.* - Flush_Interval_Sec 10 # Process and flush metrics every 60 seconds - - outputs: | - [OUTPUT] - Name prometheus_exporter - Match log_metrics - - [OUTPUT] - Name stdout - Match log_metrics - - # export metrics - metrics: - enabled: true - - extraPorts: - - name: metrics - targetPort: metrics - protocol: TCP - port: 2021 - containerPort: 2021 - - serviceMonitor: - enabled: true - additionalEndpoints: - - port: metrics - path: /metrics - honorLabels: true # important - keep the original label on the metrics (pod, namespace, container) - - - .. note:: - By default, the ``log_to_metrics`` FILTER, adds the ``log_metric_counter_`` prefix to every metric - - .. raw:: html - -
- Understanding the Configuration -
    -
  • The tail INPUT section defines all Kubernetes container logs as input
    • -
  • The dummy INPUT section defines a keepalive input - it's required to create at least 1 active metric
    • -
  • The kubernetes FILTER section is for adding the Kubernetes labels to the log lines/li> -
  • The 1st log-to-metrics FILTER - match any log line containing "mysql connection error", and increase the ``mysql_connection_error`` counter. Add the pod labels to the metric
    • -
  • The 2nd log-to-metrics FILTER - match any log line containing "dns error", and increase the ``dns_error`` counter. Add the pod labels to the metric
    • -
  • The 3rd log-to-metrics FILTER - for the keepalive metric
    • -
  • The prometheus_exporter OUTPUT is for exporting the Prometheus metrics
    • -
  • The stdout OUTPUT is used for debugging. It prints the metrics to the fluentbit pod logs. Not required for production deployment
    • -
-
+ .. code-block:: yaml + + config: + service: | + [SERVICE] + Flush 1 + Daemon Off + Log_Level info + HTTP_Server On + HTTP_Listen 0.0.0.0 + HTTP_Port 2020 + + inputs: | + [INPUT] + Name tail + Tag kube.* + Path /var/log/containers/*.log + Parser json_message + DB /var/log/flb_kube.db + Mem_Buf_Limit 5MB + Skip_Long_Lines On + Refresh_Interval 10 + + [INPUT] + Name dummy + Tag dummy.alive + Dummy {"log":"keepalive"} + + parsers: | + [PARSER] + Name wrap_raw_line + Format regex + Regex ^(?.*)$ + + filters: | + [FILTER] + Name kubernetes + Match kube.* + K8S-Logging.Parser On + K8S-Logging.Exclude On + + [FILTER] + name log_to_metrics + match * + tag log_metrics + metric_mode counter + metric_name mysql_connection_error + metric_description MySql connection errors + regex log .*mysql connection error.* + add_label pod $kubernetes['pod_name'] + add_label namespace $kubernetes['namespace_name'] + add_label container $kubernetes['container_name'] + + [FILTER] + name log_to_metrics + match * + tag log_metrics + metric_mode counter + metric_name dns_error + metric_description DNS Resolution errors + regex log .*dns error.* + add_label pod $kubernetes['pod_name'] + add_label namespace $kubernetes['namespace_name'] + add_label container $kubernetes['container_name'] + + [FILTER] + Name log_to_metrics + Match dummy.alive + Metric_Name fluentbit_keepalive + Metric_Description Dummy metric to keep /metrics available + Metric_Mode counter + Tag log_metrics + Regex log .*keepalive.* + Flush_Interval_Sec 10 # Process and flush metrics every 60 seconds + + outputs: | + [OUTPUT] + Name prometheus_exporter + Match log_metrics + + [OUTPUT] + Name stdout + Match log_metrics + + # export metrics + metrics: + enabled: true + + extraPorts: + - name: metrics + targetPort: metrics + protocol: TCP + port: 2021 + containerPort: 2021 + + serviceMonitor: + enabled: true + additionalEndpoints: + - port: metrics + path: /metrics + honorLabels: true # important - keep the original label on the metrics (pod, namespace, container) + + + .. note:: + By default, the ``log_to_metrics`` FILTER, adds the ``log_metric_counter_`` prefix to every metric + + .. raw:: html + +
+ Understanding the Configuration +
    +
  • The tail INPUT section defines all Kubernetes container logs as input
    • +
  • The dummy INPUT section defines a keepalive input - it's required to create at least 1 active metric
    • +
  • The kubernetes FILTER section is for adding the Kubernetes labels to the log lines/li> +
  • The 1st log-to-metrics FILTER - match any log line containing "mysql connection error", and increase the ``mysql_connection_error`` counter. Add the pod labels to the metric
    • +
  • The 2nd log-to-metrics FILTER - match any log line containing "dns error", and increase the ``dns_error`` counter. Add the pod labels to the metric
    • +
  • The 3rd log-to-metrics FILTER - for the keepalive metric
    • +
  • The prometheus_exporter OUTPUT is for exporting the Prometheus metrics
    • +
  • The stdout OUTPUT is used for debugging. It prints the metrics to the fluentbit pod logs. Not required for production deployment
    • +
+
Let's deploy the Fluent Bit DaemonSet: - .. code-block:: bash + .. code-block:: bash - helm repo add fluent https://fluent.github.io/helm-charts && helm repo update - helm install metrics-fluent-bit fluent/fluent-bit -f ./fluentbit-values.yaml + helm repo add fluent https://fluent.github.io/helm-charts && helm repo update + helm install metrics-fluent-bit fluent/fluent-bit -f ./fluentbit-values.yaml From 3fe53e70d2de54bf84729a14bf7da5f7c988e6ae Mon Sep 17 00:00:00 2001 From: Pavan Gudiwada Date: Thu, 8 May 2025 08:22:03 +0530 Subject: [PATCH 2/2] Minor TOC fixes --- .../logs-triggers/index.rst | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/playbook-reference/logs-triggers/index.rst b/docs/playbook-reference/logs-triggers/index.rst index 1a591e11d..fd55fe278 100644 --- a/docs/playbook-reference/logs-triggers/index.rst +++ b/docs/playbook-reference/logs-triggers/index.rst @@ -1,5 +1,3 @@ -.. :hide-toc: - Automate Responses to Application Logs ========================================== @@ -21,7 +19,7 @@ Let's get started! Here's a video explainer of how it works:
-**Step 1: Create a namespace for the demo** +Step 1: Create a namespace for the demo ---------------------------------------------------- .. code-block:: yaml @@ -29,7 +27,7 @@ Let's get started! Here's a video explainer of how it works: kubectl create namespace log-triggers kubectl config set-context --current --namespace log-triggers -**Step 2: Parse Logs into Metrics with Fluent Bit** +Step 2: Parse Logs into Metrics with Fluent Bit ---------------------------------------------------- First, lets configure Fluent Bit to monitor your pod logs and generate Prometheus metrics for specific log patterns. @@ -173,7 +171,7 @@ Let's deploy the Fluent Bit DaemonSet: -**Step 3: Configure Prometheus** +Step 3: Configure Prometheus ---------------------------------------------------- In this step, we will configure Prometheus to: @@ -181,7 +179,7 @@ In this step, we will configure Prometheus to: 1. **Collect metrics from Fluent Bit** via a `ServiceMonitor` 2. **Configure an alert** based on the metrics extracted from the logs -1. **Configure Prometheus to read the new ServiceMonitor** +1. Configure Prometheus to read the new ServiceMonitor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Assuming you're using Robusta's bundled Prometheus, add this to your ``generated_values.yaml``: @@ -202,7 +200,7 @@ To apply it, upgrade with helm: helm upgrade robusta robusta/robusta -f generated_values.yaml -set clusterName=YOUR_CLUSTER -2. **Configure an Alert** +2. Configure an Alert ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is the alerting rule that will be used to trigger an alert when a `MySqlConnectionErrors` is detected in the logs (``mysql-alert.yaml``): @@ -241,7 +239,7 @@ To apply it run: kubectl apply -f mysql-alert.yaml -**Step 4: Adding a Robusta playbook** +Step 4: Adding a Robusta playbook ---------------------------------------------------- Now, we'd like to configure an automated action that will run each time this alert is fired. @@ -269,12 +267,13 @@ To apply it, upgrade with helm: helm upgrade robusta robusta/robusta -f generated_values.yaml -set clusterName=YOUR_CLUSTER -**Step 5: See It in Action** +Step 5: See It in Action ---------------------------------------------------- Let’s test the full automation pipeline by generating a log line that simulates a MySQL connection error. -1. **Deploy a demo pod** +1. Deploy a demo pod +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use this manifest to deploy a demo pod that prints to the logs whatever is sent to its API (``postlog.yaml``): @@ -326,7 +325,8 @@ Apply it to your cluster: kubectl apply -f postlog.yaml -2. **Generate MySQL errors in the logs** +2. Generate MySQL errors in the logs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Call the pod’s API to print some simulated MySQL errors. @@ -351,7 +351,8 @@ Then, after 60 seconds, with 10 log lines: This will produce 10 log lines containing the error. Fluent Bit will match the log lines and emit metrics, which Prometheus will collect. -3. **Trigger the alert and observe the automation** +3. Trigger the alert and observe the automation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Wait a few minutes (typically up to 5) for the alert to fire. This delay is due to the ``for`` condition in the alert and Prometheus' ``group_interval``.