From 5a40a65e9796de89f0ddf9943f120e7587caa3b8 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon Date: Thu, 2 Jul 2026 00:59:23 -0400 Subject: [PATCH 1/3] first draft --- .../alerting-privileges.md | 59 +++++++++++++++++-- .../quick-start-alerting.md | 8 +-- .../setup-alerting.md | 37 +++++++++--- 3 files changed, 85 insertions(+), 19 deletions(-) diff --git a/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md b/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md index ba2ad6297b..9a717fb480 100644 --- a/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md +++ b/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md @@ -12,22 +12,69 @@ description: "Privilege requirements for the {{alerting-v2}}: what {{kib}} featu # {{alerting-v2-cap}} privileges [alerting-privileges] -Privileges for the {{alerting-v2}} control which users can manage rules, alerts, action policies, and query alert data. The {{alerting-v2}} are part of Kibana and are marked **Experimental** in the UI. +Privileges for the {{alerting-v2}} control what you can do with rules, alerts, action policies, and alert data queries. The {{alerting-v2}} are part of Kibana and are marked **Experimental** in the UI. - +In Kibana role management, the {{alerting-v2}} privileges appear under the **Alerting** category. There are four privileges, each independent. Holding one does not grant another. For example, if you have **Action Policies: All**, you cannot create or edit rules unless you also have **Rules: All**. ## Manage rules [alerting-manage-rules-privileges] -[CONTENT NEEDED] +The **Rules** privilege controls who can create and manage rules. + +| Level | What you can do | +|---|---| +| **All** | Create, edit, delete, enable, and disable rules | +| **Read** | View rules and their configuration | + +If you need to create or modify rules, use **Rules: All**. If you only need to view existing rules and their configuration, use **Rules: Read**. ## Manage action policies [action-policy-management] -[CONTENT NEEDED] +The **Action Policies** privilege controls who can manage the policies that route alert episode notifications. + +| Level | What you can do | +|---|---| +| **All** | Create, update, delete, snooze, and unsnooze action policies | +| **Read** | View action policies | + +If you need to create or configure action policies, use **Action Policies: All**. Having **Rules: All** does not include the ability to manage action policies unless you also have **Action Policies: All**. ## Manage alerts [alerting-manage-alerts-privileges] -[CONTENT NEEDED] +The **Alerts** privilege controls who can take triage actions on alert episodes. + +| Level | What you can do | +|---|---| +| **All** | Acknowledge, snooze, assign, tag, activate, and deactivate alert episodes | +| **Read** | View alert episodes | + +If you need to acknowledge or snooze active episodes, use **Alerts: All**. If you only need to monitor the alerts list, use **Alerts: Read**. + +## View execution history [alerting-execution-history-privileges] + +The **Execution history** privilege controls who can view rule execution history. + +| Level | What you can do | +|---|---| +| **All** | View rule execution history | +| **Read** | View rule execution history | + +Execution history is read-only. Both **All** and **Read** grant the same access. There is no write surface for execution history. ## Query alert data in Discover [alerting-discover-privileges] -[CONTENT NEEDED] +To query alert data in Discover using {{esql}}, you need both Kibana feature access and Elasticsearch index access. + +### Kibana feature access + +You need **Discover: Read** access in the space where you plan to run queries. + +### Elasticsearch index access + +The {{alerting-v2}} writes alert data to two hidden system data streams: `.rule-events` and `.alert-actions`. To query them in Discover, you need the `read` index privilege on each data stream. + +| Data stream | Required privilege | +|---|---| +| `.rule-events` | `read` | +| `.alert-actions` | `read` | + +Because these are hidden system data streams, request access through a custom role with the appropriate index privileges. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md b/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md index 4aa05a6017..9385b6ffca 100644 --- a/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md +++ b/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md @@ -155,11 +155,11 @@ By completing this tutorial, you saw the core experimental alerting model in act ## Next steps - \ No newline at end of file diff --git a/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md b/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md index 99a7fef25c..81531c4785 100644 --- a/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md +++ b/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md @@ -12,17 +12,39 @@ description: "Get the {{alerting-v2}} running in your space: enable the UI, conf # Set up the {{alerting-v2}} [alerting-setup] -Set up is part of the {{alerting-v2}} in Kibana. Before you can create your first rule, the {{alerting-v2}} need to be enabled in your space and a few background systems need to be in place. Rules rely on two data streams to store their output, API keys to run with the right privileges, and space scoping to keep objects organized. Getting these right upfront means your rules will run cleanly and their output will be queryable from the start. +Before you can create your first rule, the {{alerting-v2}} in {{kib}} must be enabled in your space and a few background systems need to be in place. + +Rules rely on two data streams to store their output, API keys to run with the right privileges, and space scoping to keep objects organized. Getting these right upfront means your rules will run cleanly and their output will be queryable from the start. If you want to jump straight to creating a rule, go to [Quick start](quick-start-alerting.md). For privilege requirements, refer to [{{alerting-v2-cap}} privileges](alerting-privileges.md). ## Enable the {{alerting-v2}} [alerting-set-up] +To access the {{alerting-v2}}, a {{kib}} administrator must turn on the `alerting:v2:enabled` advanced setting. The steps differ by deployment type. -The {{alerting-v2}} are available in technical preview in {{stack}} 9.5 and in {{serverless-short}}. +### Enable on {{stack}} [alerting-enable-stack] - +1. Go to **Stack Management → Advanced Settings**. +2. Under **Global settings**, toggle on **alerting:v2:enabled**. + +### Enable on {{serverless-short}} [alerting-enable-serverless] + +{{serverless-short}} has no Global Advanced Settings UI. Use Dev Tools to call the global settings API: + +```json +POST kbn:/internal/kibana/global_settings +{ + "changes": { + "alerting:v2:enabled": true + } +} +``` + +:::{note} +The `/internal/kibana/global_settings` endpoint is an internal API and may change without notice. There is currently no public equivalent. +::: + +When the `alerting:v2:enabled` setting is turned off, rule and dispatcher execution stops, the APIs and UIs are hidden, and existing rules and action policies are paused. Turning it back on resumes execution. Disabling does not delete any data. ## Where rule events are stored @@ -42,9 +64,6 @@ FROM $`.rule-events` | LIMIT 10 ``` - - After your first rule runs, use the query above in Discover to confirm documents are appearing. If nothing appears after a few seconds, check that the rule is enabled and that your ES|QL query returns results when run independently. ## Spaces [spaces-for-alerting] @@ -61,9 +80,9 @@ Saving a rule or action policy automatically creates an API key that is used to When you're ready to go further, these can be done in any order: \ No newline at end of file From 44db48c4069ab3770c9958550d0264b17a5fe81b Mon Sep 17 00:00:00 2001 From: Nastasha Solomon Date: Thu, 2 Jul 2026 18:54:03 -0400 Subject: [PATCH 2/3] fix privs and setup --- docset.yml | 2 + .../alerting-privileges.md | 80 --------- .../get-started.md | 18 ++ .../get-started/configure-access.md | 124 +++++++++++++ .../get-started/create-your-first-rule.md | 168 ++++++++++++++++++ .../get-started/setup.md | 92 ++++++++++ .../quick-start-alerting.md | 165 ----------------- .../setup-alerting.md | 88 --------- explore-analyze/toc.yml | 8 +- 9 files changed, 409 insertions(+), 336 deletions(-) delete mode 100644 explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md create mode 100644 explore-analyze/alerting/kibana-alerting-experimental/get-started.md create mode 100644 explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md create mode 100644 explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md create mode 100644 explore-analyze/alerting/kibana-alerting-experimental/get-started/setup.md delete mode 100644 explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md delete mode 100644 explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md diff --git a/docset.yml b/docset.yml index a6a20b4619..1d5ec39c60 100644 --- a/docset.yml +++ b/docset.yml @@ -125,6 +125,8 @@ subs: maint-windows-cap: "Maintenance windows" alerting-v2: "experimental alerting features" alerting-v2-cap: "Experimental alerting features" + alerting-v2-system: "experimental alerting system" + alerting-v2-system-cap: "Experimental alerting system" custom-roles-app: "Custom Roles" data-source: "data view" data-sources: "data views" diff --git a/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md b/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md deleted file mode 100644 index 9a717fb480..0000000000 --- a/explore-analyze/alerting/kibana-alerting-experimental/alerting-privileges.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -navigation_title: Privileges -applies_to: - stack: experimental 9.5+ - serverless: experimental -products: - - id: kibana - - id: cloud-serverless -description: "Privilege requirements for the {{alerting-v2}}: what {{kib}} feature access and {{es}} index access each role needs to manage rules, policies, alerts, and query alert data." ---- - -# {{alerting-v2-cap}} privileges [alerting-privileges] - - -Privileges for the {{alerting-v2}} control what you can do with rules, alerts, action policies, and alert data queries. The {{alerting-v2}} are part of Kibana and are marked **Experimental** in the UI. - -In Kibana role management, the {{alerting-v2}} privileges appear under the **Alerting** category. There are four privileges, each independent. Holding one does not grant another. For example, if you have **Action Policies: All**, you cannot create or edit rules unless you also have **Rules: All**. - -## Manage rules [alerting-manage-rules-privileges] - -The **Rules** privilege controls who can create and manage rules. - -| Level | What you can do | -|---|---| -| **All** | Create, edit, delete, enable, and disable rules | -| **Read** | View rules and their configuration | - -If you need to create or modify rules, use **Rules: All**. If you only need to view existing rules and their configuration, use **Rules: Read**. - -## Manage action policies [action-policy-management] - -The **Action Policies** privilege controls who can manage the policies that route alert episode notifications. - -| Level | What you can do | -|---|---| -| **All** | Create, update, delete, snooze, and unsnooze action policies | -| **Read** | View action policies | - -If you need to create or configure action policies, use **Action Policies: All**. Having **Rules: All** does not include the ability to manage action policies unless you also have **Action Policies: All**. - -## Manage alerts [alerting-manage-alerts-privileges] - -The **Alerts** privilege controls who can take triage actions on alert episodes. - -| Level | What you can do | -|---|---| -| **All** | Acknowledge, snooze, assign, tag, activate, and deactivate alert episodes | -| **Read** | View alert episodes | - -If you need to acknowledge or snooze active episodes, use **Alerts: All**. If you only need to monitor the alerts list, use **Alerts: Read**. - -## View execution history [alerting-execution-history-privileges] - -The **Execution history** privilege controls who can view rule execution history. - -| Level | What you can do | -|---|---| -| **All** | View rule execution history | -| **Read** | View rule execution history | - -Execution history is read-only. Both **All** and **Read** grant the same access. There is no write surface for execution history. - -## Query alert data in Discover [alerting-discover-privileges] - -To query alert data in Discover using {{esql}}, you need both Kibana feature access and Elasticsearch index access. - -### Kibana feature access - -You need **Discover: Read** access in the space where you plan to run queries. - -### Elasticsearch index access - -The {{alerting-v2}} writes alert data to two hidden system data streams: `.rule-events` and `.alert-actions`. To query them in Discover, you need the `read` index privilege on each data stream. - -| Data stream | Required privilege | -|---|---| -| `.rule-events` | `read` | -| `.alert-actions` | `read` | - -Because these are hidden system data streams, request access through a custom role with the appropriate index privileges. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started.md new file mode 100644 index 0000000000..41ba64b8ae --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started.md @@ -0,0 +1,18 @@ +--- +navigation_title: Get started +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana + - id: cloud-serverless +description: "Enable the experimental alerting system in Kibana, configure role access, and create your first rule." +--- + +# Get started with the {{alerting-v2-system-cap}} [alerting-v2-get-started] + +Enable the feature, configure access, and create your first rule. + +- [Turn on the {{alerting-v2-system-cap}}](get-started/setup.md): Enable the `alerting:v2:enabled` advanced setting and confirm the system is accessible in your space. +- [Configure access](get-started/configure-access.md): Set up a role with the {{kib}} feature privileges and {{es}} index privileges needed to create rules and query alert data. +- [Create your first rule](get-started/create-your-first-rule.md): A hands-on tutorial that walks you through loading sample data, creating a rule, and watching the alert lifecycle play out from breach to recovery. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md new file mode 100644 index 0000000000..3e51b69df1 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md @@ -0,0 +1,124 @@ +--- +navigation_title: Configure access +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana + - id: cloud-serverless +description: "Privilege requirements for Kibana experimental alerting features: which Kibana feature privileges and Elasticsearch index privileges each role needs to manage rules, action policies, alerts, and query rule events and alert actions." +--- + +# Configure access to the {{alerting-v2-system}} [alerting-privileges] + +To use the {{alerting-v2-system}}, your role needs specific {{kib}} feature privileges and, if you're querying alerting data in Discover, {{es}} index privileges. [Create or update a role](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-role-management.md) and add the privileges that match the tasks your team performs. + +This page is organized by user activity. Most privileges are set under the **Alerting** category in {{kib}} role management. Exceptions are noted in each section. + +:::{note} +This page covers access to the {{alerting-v2-system}} features and data. Depending on how your rules and notifications are configured, your role might also need `read` index privileges on the indices their rules query and **Actions and Connectors: All** (under **Management**) to create or edit workflow connectors. Refer to [{{kib}} role management](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-role-management.md) for guidance on building roles that combine privileges across features. +::: + +## Quick reference [alerting-quick-reference] + +The following table shows the minimum privileges required for each activity. Higher privilege levels include the access shown here. Refer to the following sections for the full breakdown. + +| To... | Minimum required | +|---|---| +| Author and manage rules | **Rules: All** (under **Alerting**) | +| Monitor rule execution | **Execution history: Read** (under **Alerting**) | +| Triage alert episodes | **Alerts: All** (under **Alerting**) | +| Configure notifications | **Action Policies: All** (under **Alerting**) + **Workflows: Read** (under **Analytics > Workflows**) | +| Query rule and episode data in Discover | **Discover: Read** (under **Analytics > Discover**) + `read` index privilege on the relevant data stream | + +## Author and monitor rules [alerting-authoring-monitoring-privileges] + +These privileges control who can create rules and review their execution history. + +### Rules [alerting-manage-rules-privileges] + +The **Rules** privilege controls who can create and manage rules. + +| Level | What you can do | +|---|---| +| **All** | Create, edit, delete, enable, and disable rules | +| **Read** | View rules and their configuration | + +:::{note} +**Rules: All** also grants access to the **Alerts** menu in Discover, which routes rule creation to the {{alerting-v2-system}} rule form when the system is enabled in your space. +::: + +### View rule execution history [alerting-execution-history-privileges] + +The **Execution history** privilege controls who can view rule execution history. Execution history is read-only; both **All** and **Read** grant the same access. There is no write surface for execution history. + +| Level | What you can do | +|---|---| +| **All** | View rule execution history | +| **Read** | View rule execution history | + +## Triaging alerts [alerting-triage-privileges] + +The **Alerts** privilege controls who can take triage actions on alert episodes. + +| Level | What you can do | +|---|---| +| **All** | Acknowledge, snooze, assign, tag, activate, and deactivate alert episodes | +| **Read** | View alert episodes | + +## Configure notifications [alerting-notifications-privileges] + +These privileges control who can set up the action policies and workflows that route alert episode notifications. + +### Action policies [action-policy-management] + +The **Action Policies** privilege controls who can manage the policies that route alert episode notifications. + +| Level | What you can do | +|---|---| +| **All** | Create, update, delete, snooze, and unsnooze action policies | +| **Read** | View action policies | + +:::{note} +Having **Action Policies: All** does not include the ability to create or edit rules. Add **Rules: All** if rule management is also required. +::: + +### Workflows [alerting-workflows-access] + +Action policies route notifications through workflows. The **Workflows** privilege is set under **Analytics > Workflows** in {{kib}} role management. To create or manage action policies, your role also needs access to the workflows they reference. + +| Level | What you can do | +|---|---| +| **All** | Create and edit workflows; view and select existing workflows in action policies | +| **Read** | View and select existing workflows in action policies | + +## Query rule output and episode data [alerting-data-investigation-privileges] + +The {{alerting-v2-system}} writes rule output and episode data to three queryable data sources. To query them in Discover using {{esql}}, your role needs {{kib}} feature access and {{es}} index access. + +### {{kib}} feature access + +Set Discover privileges under **Analytics > Discover** in {{kib}} role management. + +| Level | What you can do | +|---|---| +| **All** | Run {{esql}} queries against rule events, alert actions, and execution history in Discover | +| **Read** | Run {{esql}} queries against rule events, alert actions, and execution history in Discover | + +Both levels grant the same query access. There is no write surface for any of these data sources in Discover. + +### {{es}} index access + +Each data source requires a separate `read` index privilege: + +| Data source | What it stores | Required privilege | +|---|---|---| +| `.rule-events` | A record for every rule evaluation; one document per result row, per run | `read` | +| `.alert-actions` | Episode action records: acknowledge, snooze, resolve, assign, and other triage operations | `read` | +| `.kibana-event-log-*` | Action policy dispatch outcomes written by the dispatcher: `dispatched`, `throttled`, and `unmatched` | `read` | + +Because `.rule-events` and `.alert-actions` are hidden system data streams, request access through a custom role with the appropriate index privileges. + + diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md new file mode 100644 index 0000000000..501a6865c4 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md @@ -0,0 +1,168 @@ +--- +navigation_title: Create your first rule +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana + - id: cloud-serverless +description: "Create your first Kibana rule using the experimental alerting features: enable the feature flag, write an ES|QL query in the YAML editor, and verify the rule is running by querying the rule events data stream." +--- + +# Create your first rule [alerting-quick-start] + +In this tutorial, you'll load sample error logs, create a rule that detects errors by service, and watch the alert lifecycle play out — from breach to recovery. Along the way, you'll learn the core concepts of the {{alerting-v2-system}}. + +## Prerequisites [alerting-quick-start-prerequisites] + +- **Enable the {{alerting-v2-system}}:** The feature must be turned on in your space before you can create rules or view the UI. Refer to [Turn on the {{alerting-v2-system}}](setup.md) for instructions. +- **Configure access:** Your role must include the following privileges to complete this tutorial: + - **Rules: All** (under **Alerting** in Kibana role management) to create and manage rules + - **Discover: Read** and `read` index privilege on `.rule-events` to query rule output in Discover + + For a full breakdown of privilege requirements, refer to [{{alerting-v2-system-cap}} privileges](configure-access.md). + +## Tutorial [alerting-quick-start-tutorial] + +:::::{stepper} + +::::{step} Create a sample data stream + +Run this in Dev Tools to create a data stream called `logs-tutorial` and populate it with sample error logs. This is the data your rule will query. + +```json +POST logs-tutorial/_bulk +{ "index": {} } +{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Connection timeout" } +{ "index": {} } +{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Database query failed" } +{ "index": {} } +{ "@timestamp": "2026-04-21T21:52:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Null pointer exception" } +{ "index": {} } +{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "ERROR", "service_name": "payments", "message": "Payment gateway unreachable" } +{ "index": {} } +{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "ERROR", "service_name": "payments", "message": "Transaction rollback failed" } +{ "index": {} } +{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "INFO", "service_name": "checkout", "message": "Request received" } +{ "index": {} } +{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "INFO", "service_name": "payments", "message": "Health check passed" } +``` + +:::{note} +`logs-tutorial` is automatically created as a data stream because the `logs-*` naming pattern matches Elasticsearch's default index template. You don't need to create it manually beforehand. +::: + +Confirm the data was indexed. You should see `errors: false` and `result: "created"` for each document in the response. + +:::: + +::::{step} Create the rule + +Go to **Alerting V2 Preview** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). From the rules list, select the option to create a new rule. When the rule creation panel opens, select **Create ES|QL rule**. + +Next, paste the following {{esql}} query into the **Query sandbox** textbox: + +```esql +FROM logs-tutorial +| WHERE log_level == "ERROR" +| STATS error_count = COUNT() BY service_name +| WHERE error_count >= 2 +``` + +Use the query sandbox to verify the query returns the results you expect before applying it to the rule form. The sandbox lets you run the query against current data using the time field selector and date picker to control the time range. Select **Search** (or press ⌘↵) to execute. When the results look correct, select **Apply changes** to populate the form. + +After applying the query, set the remaining fields in the form: + +| Field | Value | +|---|---| +| Time field | `@timestamp` | +| Schedule | `5s` | +| Lookback | `24h` | +| Group by | `service_name` | + + + +Click **Save**. The rule is enabled automatically. + +:::: + +::::{step} Confirm the rule is running + +Wait 5 seconds, then run the following in Discover using the ES|QL mode. Replace `` with the `tutorial-error-rate` rule ID. + +:::{tip} +After saving the rule, open its details page. The rule ID is the string at the end of the browser URL. +::: + +```esql +FROM .rule-events +| WHERE rule.id == "" +| SORT @timestamp DESC +| LIMIT 10 +``` + +Check the following in the query results: + +| Field | What you should see | +|---|---| +| `@timestamp` | Recent timestamps updating every 5 seconds | +| `status` | `breached` confirms the query is finding matches | +| `episode.status` | `active` confirms episodes have opened for both services | +| `data.service_name` | `checkout` and `payments` confirms grouping is working | + +:::: + +::::{step} Confirm two episodes are open + +In Discover, run: + +```esql +FROM .rule-events +| WHERE rule.id == "" +| STATS latest = MAX(@timestamp) BY episode.id, episode.status, data.service_name +| SORT latest DESC +``` + +You should see two rows: one episode for `checkout` and one for `payments`, both with `episode.status: active`. + +:::: + +::::{step} Trigger recovery + +Delete the test data to clear the breach condition. Run the following in Dev Tools: + +```json +POST logs-tutorial/_delete_by_query +{ + "query": { "match_all": {} } +} +``` + +Wait up to 5 seconds, then re-run the query from the previous step in Discover. Both episodes should now show `episode.status: inactive`. + +:::: + +::::: + +## Key concepts demonstrated + +This tutorial introduces the core concepts of the {{alerting-v2-system}}: + +* **Rules**: Query your data on a schedule and detect when conditions are met. Your rule ran every 5 seconds, checking for services with 2 or more errors in the last 24 hours. +* **Grouping**: Creates independent tracked series from a single rule. Because the rule grouped by `service_name`, `checkout` and `payments` were tracked separately — each with its own episode. +* **Episodes**: Follow a lifecycle driven by the rule's findings. Both episodes moved to `active` when errors existed, and recovered to `inactive` automatically when the data was deleted. +* **Rule events**: The underlying record of every evaluation. Every run wrote documents to `.rule-events`, giving you a full queryable history of what the rule found and when. + +## What's next + + + + diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started/setup.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started/setup.md new file mode 100644 index 0000000000..23370b6ad6 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started/setup.md @@ -0,0 +1,92 @@ +--- +navigation_title: Set up +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana + - id: cloud-serverless +description: "Enable and disable the experimental alerting system in Kibana: turn on the alerting:v2:enabled advanced setting, confirm the system is accessible, and understand what happens when the setting is turned off." +--- + +# Set up the {{alerting-v2-system}} [alerting-setup] + +This page explains how to enable the {{alerting-v2-system}} in your space, confirm it's accessible, and turn it off when needed. Enabling and disabling the system requires a {{kib}} administrator. Confirming accessibility can be done by any user with space access. + +## Turn on the system [alerting-setup-turn-on] + +The {{alerting-v2-system}} is controlled by the `alerting:v2:enabled` advanced setting in {{kib}}. This setting is off by default. Turn it on to make rules, action policies, and the alerting UI available in your space. The steps differ by deployment type. + +::::{tab-set} +:::{tab-item} {{stack}} +:sync: stack + +**Requirement:** Turning on this setting requires the `kibana_admin` role or equivalent {{stack-manage-app}} access. + +**Steps:** + +1. Go to the **Advanced Settings** menu using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +2. Under **Global settings**, toggle on **alerting:v2:enabled**. +::: + +:::{tab-item} {{serverless-short}} +:sync: serverless + +**Requirement:** Turning on this setting requires the `admin` project role. + +**Steps:** + +{{serverless-short}} has no Global Advanced Settings UI. Use Dev Tools to call the global settings API: + +```json +POST kbn:/internal/kibana/global_settings +{ + "changes": { + "alerting:v2:enabled": true + } +} +``` + +:::{note} +The `/internal/kibana/global_settings` endpoint is an internal API and may change without notice. There is currently no public equivalent. +::: +::: +:::: + +### Confirm the UI is accessible [alerting-setup-confirm] + +After turning on the setting, verify the {{alerting-v2-system}} is accessible in your space: + +1. Use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) and enter `Alerting v2 preview`. +2. Select the menu item from the results. + +If the menu item doesn't appear immediately, refresh the page and search again. It may take a moment for the UI to reflect the updated setting. + +## Turn off the system [alerting-setup-turn-off] + +To turn off the {{alerting-v2-system}}, set `alerting:v2:enabled` to `false`: + +- **{{stack}}:** Go to the same **Advanced Settings** page and toggle off **alerting:v2:enabled**. +- **{{serverless-short}}:** Call the global settings API with `"alerting:v2:enabled": false`. + +When the setting is off: + +- Rule and action policy execution stops. +- The APIs and UI are hidden. +- Existing rules and action policies are paused. + +Turning the setting back on resumes execution. Turning it off does not delete any data — your rules and action policies remain as {{kib}} saved objects, and existing documents in `.rule-events` and `.alert-actions` are preserved. + +## Next steps + +After turning on the system, configure role access so your team can use it: + +- **[Configure access](configure-access.md):** Create or update a role with access to the {{alerting-v2-system}} features and the data streams they write to. Users need at minimum **Rules: All** to create rules and `read` index access on `.rule-events` to query rule output in Discover. + + + diff --git a/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md b/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md deleted file mode 100644 index 9385b6ffca..0000000000 --- a/explore-analyze/alerting/kibana-alerting-experimental/quick-start-alerting.md +++ /dev/null @@ -1,165 +0,0 @@ ---- -navigation_title: Quick start -applies_to: - stack: experimental 9.5+ - serverless: experimental -products: - - id: kibana - - id: cloud-serverless -description: "Create your first rule in the {{alerting-v2}}: enable the feature, write an {{esql}} query in the YAML editor, and confirm the rule is running by querying the rule events data stream." ---- - -# Quick start: Your first rule [alerting-quick-start] - - -This quick start is part of the {{alerting-v2}} in Kibana. It walks you through the core model in action. You'll load sample error logs, create a rule that groups results by service, watch episodes open as the rule finds breaches, and then trigger recovery by deleting the data. By the end, you'll have learned about rules, grouping, episodes, and the alert lifecycle play out from start to finish. - -## Step 1: Create a sample data stream - -Run this in Dev Tools to create a data stream called `logs-tutorial` and populate it with sample error logs. This is the data your rule will query. - -```json -POST logs-tutorial/_bulk -{ "index": {} } -{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Connection timeout" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Database query failed" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:52:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Null pointer exception" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "ERROR", "service_name": "payments", "message": "Payment gateway unreachable" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "ERROR", "service_name": "payments", "message": "Transaction rollback failed" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "INFO", "service_name": "checkout", "message": "Request received" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "INFO", "service_name": "payments", "message": "Health check passed" } -``` - -:::{note} -`logs-tutorial` is automatically created as a data stream because the `logs-*` naming pattern matches Elasticsearch's default index template. You don't need to create it manually beforehand. -::: - -Confirm the data was indexed. You should see `errors: false` and `result: "created"` for each document in the response. - - -## Step 2: Verify the data is queryable - -Run this in Dev Tools to confirm the rule will find it: - -```esql -FROM logs-tutorial -| WHERE log_level == "ERROR" -| STATS error_count = COUNT() BY service_name -| WHERE error_count >= 2 -``` - -You should see two rows: one for `checkout` (3 errors) and one for `payments` (2 errors). If you see this, the rule will trigger for both services. - -## Step 3: Create the rule - -Go to **Management > V2 Alerting Preview** and create a new rule using the YAML editor with the following configuration: - - - -```yaml -kind: alert -metadata: - name: tutorial-error-rate - tags: - - tutorial -time_field: '@timestamp' -schedule: - every: 5s - lookback: 24h -evaluation: - query: - base: |- - FROM logs-tutorial - | WHERE log_level == "ERROR" - | STATS error_count = COUNT() BY service_name - | WHERE error_count >= 2 -grouping: - fields: - - service_name -``` - - - -Save the rule. It will be enabled automatically. - - -## Step 4: Confirm the rule is running - -Wait 5 seconds, then run the following in Discover using the ES|QL mode. Replace `` with the `tutorial-error-rate` rule ID. - -```esql -FROM .rule-events -| WHERE rule.id == "" -| SORT @timestamp DESC -| LIMIT 10 -``` - -:::{tip} -After saving the rule, open its details page. The rule ID is the string at the end of the browser URL. -::: - -Check the following in the query results: - -| Field | What you should see | -|---|---| -| `@timestamp` | Recent timestamps updating every 5 seconds | -| `status` | `breached` confirms the query is finding matches | -| `episode.status` | `active` confirms episodes have opened for both services | -| `data.service_name` | `checkout` and `payments` confirms grouping is working | - - -## Step 5: Confirm two episodes are open - -In Discover, run: - -```esql -FROM .rule-events -| WHERE rule.id == "" -| STATS latest = MAX(@timestamp) BY episode.id, episode.status, data.service_name -| SORT latest DESC -``` - -You should see two rows: one episode for `checkout` and one for `payments`, both with `episode.status: active`. - - -## Step 6: Trigger recovery - -Delete the test data to clear the breach condition. Run the following in Dev Tools: - -```json -POST logs-tutorial/_delete_by_query -{ - "query": { "match_all": {} } -} -``` - -Wait up to 5 seconds, then re-run the query from Step 5 in Discover. Both episodes should now show `episode.status: inactive`. - -## What you learned - -By completing this tutorial, you saw the core experimental alerting model in action end to end: - -- **Rules query your data on a schedule.** Your rule ran every 5 seconds, checking for services with 2 or more errors in the last 24 hours. -- **Grouping creates independent series.** Because the rule grouped by `service_name`, `checkout` and `payments` were tracked separately. Each got its own episode. -- **Episodes follow a lifecycle.** When the error logs existed, both episodes moved to `active`. When you deleted the logs, both recovered and moved to `inactive` automatically, no manual intervention required. -- **Rule events are the underlying record.** Every evaluation wrote documents to `.rule-events`, giving you a full queryable history of what the rule found and when. - -## Next steps - - - - \ No newline at end of file diff --git a/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md b/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md deleted file mode 100644 index 81531c4785..0000000000 --- a/explore-analyze/alerting/kibana-alerting-experimental/setup-alerting.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -navigation_title: Set up -applies_to: - stack: experimental 9.5+ - serverless: experimental -products: - - id: kibana - - id: cloud-serverless -description: "Get the {{alerting-v2}} running in your space: enable the UI, confirm data streams, then understand spaces, API keys, and privileges." ---- - -# Set up the {{alerting-v2}} [alerting-setup] - - -Before you can create your first rule, the {{alerting-v2}} in {{kib}} must be enabled in your space and a few background systems need to be in place. - -Rules rely on two data streams to store their output, API keys to run with the right privileges, and space scoping to keep objects organized. Getting these right upfront means your rules will run cleanly and their output will be queryable from the start. - -If you want to jump straight to creating a rule, go to [Quick start](quick-start-alerting.md). For privilege requirements, refer to [{{alerting-v2-cap}} privileges](alerting-privileges.md). - -## Enable the {{alerting-v2}} [alerting-set-up] - -To access the {{alerting-v2}}, a {{kib}} administrator must turn on the `alerting:v2:enabled` advanced setting. The steps differ by deployment type. - -### Enable on {{stack}} [alerting-enable-stack] - -1. Go to **Stack Management → Advanced Settings**. -2. Under **Global settings**, toggle on **alerting:v2:enabled**. - -### Enable on {{serverless-short}} [alerting-enable-serverless] - -{{serverless-short}} has no Global Advanced Settings UI. Use Dev Tools to call the global settings API: - -```json -POST kbn:/internal/kibana/global_settings -{ - "changes": { - "alerting:v2:enabled": true - } -} -``` - -:::{note} -The `/internal/kibana/global_settings` endpoint is an internal API and may change without notice. There is currently no public equivalent. -::: - -When the `alerting:v2:enabled` setting is turned off, rule and dispatcher execution stops, the APIs and UIs are hidden, and existing rules and action policies are paused. Turning it back on resumes execution. Disabling does not delete any data. - -## Where rule events are stored - -The {{alerting-v2}} automatically create and manage two data streams when the first rule runs. You don't need to create them manually. - -| Data stream | What it stores | -|---|---| -| `.rule-events` | A record for every rule evaluation. One document per result row, per run. Never updated in place. | -| `.alert-actions` | Records for acknowledge, snooze, deactivate, fire, suppress, and other audit and suppression tracking. | - -Both data streams are hidden system data streams. To query them in Discover, prefix the name with `$`: - -```esql -FROM $`.rule-events` -| WHERE rule.id == "" -| SORT @timestamp DESC -| LIMIT 10 -``` - -After your first rule runs, use the query above in Discover to confirm documents are appearing. If nothing appears after a few seconds, check that the rule is enabled and that your ES|QL query returns results when run independently. - -## Spaces [spaces-for-alerting] - -Rules and action policies are space-scoped. Objects you create in one space are not visible in another. Alert events are stored globally, but the UI filters what you see by space. - -## API keys [alerting-privileges] - - -Saving a rule or action policy automatically creates an API key that is used to run it. The key inherits the privileges of the user who saved the object. If those privileges change over time, update the key from the rule or policy management UI. - -## Next steps - -When you're ready to go further, these can be done in any order: - - - \ No newline at end of file diff --git a/explore-analyze/toc.yml b/explore-analyze/toc.yml index 0910fd85f8..4845da2d8d 100644 --- a/explore-analyze/toc.yml +++ b/explore-analyze/toc.yml @@ -382,9 +382,11 @@ toc: - file: report-and-share/reporting-troubleshooting-pdf.md - file: alerting.md children: - - file: alerting/kibana-alerting-experimental/quick-start-alerting.md - - file: alerting/kibana-alerting-experimental/setup-alerting.md - - file: alerting/kibana-alerting-experimental/alerting-privileges.md + - file: alerting/kibana-alerting-experimental/get-started.md + children: + - file: alerting/kibana-alerting-experimental/get-started/setup.md + - file: alerting/kibana-alerting-experimental/get-started/configure-access.md + - file: alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md - file: alerting/alerts.md children: - file: alerting/alerts/alerting-getting-started.md From 78ee550b5a4cfe8c6d048ee33c9571797e16b2fe Mon Sep 17 00:00:00 2001 From: Nastasha Solomon Date: Thu, 2 Jul 2026 19:59:59 -0400 Subject: [PATCH 3/3] updates tutorial --- .../get-started.md | 4 +- .../get-started/configure-access.md | 4 +- .../get-started/create-your-first-rule.md | 374 +++++++++++++----- .../get-started/setup.md | 10 +- 4 files changed, 290 insertions(+), 102 deletions(-) diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started.md index 41ba64b8ae..b87fdfed6c 100644 --- a/explore-analyze/alerting/kibana-alerting-experimental/get-started.md +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started.md @@ -9,9 +9,9 @@ products: description: "Enable the experimental alerting system in Kibana, configure role access, and create your first rule." --- -# Get started with the {{alerting-v2-system-cap}} [alerting-v2-get-started] +# Get started with the {{alerting-v2-system}} [alerting-v2-get-started] -Enable the feature, configure access, and create your first rule. +Use these guides to get the {{alerting-v2-system}} running in your space, set up role access for your team, and create your first rule. - [Turn on the {{alerting-v2-system-cap}}](get-started/setup.md): Enable the `alerting:v2:enabled` advanced setting and confirm the system is accessible in your space. - [Configure access](get-started/configure-access.md): Set up a role with the {{kib}} feature privileges and {{es}} index privileges needed to create rules and query alert data. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md index 3e51b69df1..0ef728f0f7 100644 --- a/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started/configure-access.md @@ -41,7 +41,7 @@ The **Rules** privilege controls who can create and manage rules. | Level | What you can do | |---|---| -| **All** | Create, edit, delete, enable, and disable rules | +| **All** | Create, edit, delete, enable, and turn off rules | | **Read** | View rules and their configuration | :::{note} @@ -113,7 +113,7 @@ Each data source requires a separate `read` index privilege: | Data source | What it stores | Required privilege | |---|---|---| -| `.rule-events` | A record for every rule evaluation; one document per result row, per run | `read` | +| `.rule-events` | A record for every rule evaluation — one document per result row per run | `read` | | `.alert-actions` | Episode action records: acknowledge, snooze, resolve, assign, and other triage operations | `read` | | `.kibana-event-log-*` | Action policy dispatch outcomes written by the dispatcher: `dispatched`, `throttled`, and `unmatched` | `read` | diff --git a/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md b/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md index 501a6865c4..43f0732552 100644 --- a/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md +++ b/explore-analyze/alerting/kibana-alerting-experimental/get-started/create-your-first-rule.md @@ -6,141 +6,331 @@ applies_to: products: - id: kibana - id: cloud-serverless -description: "Create your first Kibana rule using the experimental alerting features: enable the feature flag, write an ES|QL query in the YAML editor, and verify the rule is running by querying the rule events data stream." +description: "Step-by-step tutorial for creating your first rule in Kibana's experimental alerting system: load sample latency data, write an ES|QL detection query, configure the rule, and observe the alert episode lifecycle from breach to automatic recovery." --- # Create your first rule [alerting-quick-start] -In this tutorial, you'll load sample error logs, create a rule that detects errors by service, and watch the alert lifecycle play out — from breach to recovery. Along the way, you'll learn the core concepts of the {{alerting-v2-system}}. +In this tutorial, you'll create an index of synthetic latency data, build a rule that detects when P95 latency for a service exceeds 2 seconds, and watch the episode lifecycle — from breach detection through automatic recovery. Along the way, you'll learn how the {{alerting-v2-system}} tracks conditions over time. ## Prerequisites [alerting-quick-start-prerequisites] -- **Enable the {{alerting-v2-system}}:** The feature must be turned on in your space before you can create rules or view the UI. Refer to [Turn on the {{alerting-v2-system}}](setup.md) for instructions. -- **Configure access:** Your role must include the following privileges to complete this tutorial: - - **Rules: All** (under **Alerting** in Kibana role management) to create and manage rules +- **Enable the {{alerting-v2-system}}:** The feature must be turned on in your space before you can create rules or view the UI. Refer to [Set up the {{alerting-v2-system}}](setup.md) for instructions. +- **Configure access:** Your role must include the following privileges to complete this tutorial. Refer to [{{kib}} role management](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-role-management.md) for instructions on creating or updating a role. + - **Rules: All** (under **Alerting** in {{kib}} role management) to create and manage rules - **Discover: Read** and `read` index privilege on `.rule-events` to query rule output in Discover - For a full breakdown of privilege requirements, refer to [{{alerting-v2-system-cap}} privileges](configure-access.md). - ## Tutorial [alerting-quick-start-tutorial] :::::{stepper} -::::{step} Create a sample data stream +::::{step} Create the index -Run this in Dev Tools to create a data stream called `logs-tutorial` and populate it with sample error logs. This is the data your rule will query. +Run the following in **Dev Tools** to create the index that your rule will query. Unlike data streams, this index requires explicit creation because it uses a custom mapping. ```json -POST logs-tutorial/_bulk -{ "index": {} } -{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Connection timeout" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Database query failed" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:52:00.000Z", "log_level": "ERROR", "service_name": "checkout", "message": "Null pointer exception" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "ERROR", "service_name": "payments", "message": "Payment gateway unreachable" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "ERROR", "service_name": "payments", "message": "Transaction rollback failed" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:50:00.000Z", "log_level": "INFO", "service_name": "checkout", "message": "Request received" } -{ "index": {} } -{ "@timestamp": "2026-04-21T21:51:00.000Z", "log_level": "INFO", "service_name": "payments", "message": "Health check passed" } +PUT checkout-service-logs +{ + "mappings": { + "properties": { + "@timestamp": { "type": "date" }, + "service.name": { "type": "keyword" }, + "transaction.name": { "type": "keyword" }, + "latency_ms": { "type": "float" } + } + } +} ``` +Confirm the response shows `"acknowledged": true` before proceeding. + +:::: + +::::{step} Load the sample data + +Expand the drop-down below (at the end of this step) to copy the full bulk request, then run it in Dev Tools. It populates the index with ~40 minutes of synthetic latency data for a `checkout` service covering three phases: + +- **Healthy** (first ~16 minutes): P95 well under 2 seconds +- **Degraded** (next ~15 minutes): P95 well over 2 seconds, spanning 3 consecutive 5-minute windows +- **Recovered** (final ~10 minutes): P95 returns to healthy levels + +The response should show `errors: false` for all documents. The dataset gives you: healthy through `22:12`, degraded `22:13`–`22:27` (spanning 3 consecutive 5-minute windows), recovered from `22:28` onward. + :::{note} -`logs-tutorial` is automatically created as a data stream because the `logs-*` naming pattern matches Elasticsearch's default index template. You don't need to create it manually beforehand. +The timestamps in this request are fixed to `2026-07-02`. For the full episode lifecycle to play out as described, load this data and create the rule within 60 minutes, or adjust the timestamps to a recent window before running. For example, open the request in a text editor and use find-and-replace to substitute `2026-07-02` with today's date in `YYYY-MM-DD` format, keeping the time portion (`T21:57` through `T22:37`) intact so the degraded window (`T22:13`–`T22:27`) falls within the past 60 minutes in UTC. ::: -Confirm the data was indexed. You should see `errors: false` and `result: "created"` for each document in the response. +::::{dropdown} Bulk request: 80 synthetic latency events (healthy → degraded → recovered) +```json +POST checkout-service-logs/_bulk +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T21:57:01.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":468} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T21:57:31.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":336} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T21:58:01.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":367} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T21:58:31.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":372} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T21:59:01.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":497} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T21:59:31.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":305} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:00:01.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":384} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:00:31.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":406} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:01:01.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":427} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:01:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":461} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:02:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":448} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:02:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":527} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:03:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":272} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:03:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":477} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:04:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":355} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:04:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":466} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:05:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":528} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:05:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":280} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:06:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":278} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:06:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":252} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:07:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":443} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:07:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":447} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:08:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":504} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:08:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":260} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:09:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":353} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:09:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":508} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:10:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":381} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:10:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":315} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:11:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":284} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:11:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":261} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:12:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":371} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:12:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":329} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:13:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2704} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:13:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2909} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:14:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2898} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:14:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3094} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:15:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3701} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:15:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3977} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:16:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2368} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:16:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3954} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:17:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2610} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:17:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2491} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:18:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3751} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:18:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3909} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:19:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3903} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:19:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3905} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:20:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2292} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:20:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":4429} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:21:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":4147} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:21:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2462} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:22:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2733} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:22:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2869} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:23:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":4323} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:23:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3802} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:24:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3105} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:24:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2335} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:25:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3649} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:25:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":4320} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:26:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2671} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:26:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3438} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:27:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":2251} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:27:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":3235} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:28:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":525} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:28:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":458} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:29:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":448} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:29:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":453} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:30:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":435} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:30:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":463} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:31:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":319} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:31:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":421} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:32:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":353} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:32:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":378} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:33:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":369} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:33:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":490} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:34:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":284} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:34:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":359} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:35:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":468} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:35:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":292} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:36:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":427} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:36:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":423} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:37:02.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":259} +{"index":{"_index":"checkout-service-logs"}} +{"@timestamp":"2026-07-02T22:37:32.000Z","service.name":"checkout","transaction.name":"POST /checkout","latency_ms":309} +``` +:::: + :::: ::::{step} Create the rule -Go to **Alerting V2 Preview** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). From the rules list, select the option to create a new rule. When the rule creation panel opens, select **Create ES|QL rule**. +Go to **Alerting V2 Preview** using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). From the rules list, select the option to create a new rule. When the rule creation panel opens, select **Create ES|QL rule** to open the rule authoring flyout. -Next, paste the following {{esql}} query into the **Query sandbox** textbox: +1. Paste the following {{esql}} query into the **Query sandbox**. The query computes P95 latency per service, assigns a severity tier (`critical`, `high`, or `low`) based on the result, and returns only rows where P95 exceeds 2000 ms. An empty result means no breach. -```esql -FROM logs-tutorial -| WHERE log_level == "ERROR" -| STATS error_count = COUNT() BY service_name -| WHERE error_count >= 2 -``` + ```esql + FROM checkout-service-logs + | STATS p95_latency_ms = PERCENTILE(latency_ms, 95) BY service.name + | EVAL severity = CASE( + p95_latency_ms >= 4000, "critical", + p95_latency_ms >= 2000, "high", + "low" + ) + | WHERE p95_latency_ms > 2000 + ``` -Use the query sandbox to verify the query returns the results you expect before applying it to the rule form. The sandbox lets you run the query against current data using the time field selector and date picker to control the time range. Select **Search** (or press ⌘↵) to execute. When the results look correct, select **Apply changes** to populate the form. + :::{note} + Do not add a `WHERE @timestamp` clause to this query. Both the query sandbox and the rule executor automatically inject the time-window filter based on the date range you select in the sandbox or the rule's schedule and lookback once it's running. + ::: -After applying the query, set the remaining fields in the form: +2. Set the sandbox date range to **Last 1 hour** and run the query. This preset gives comfortable coverage of the full dataset without being wide enough to pull in data from a previous run-through. -| Field | Value | -|---|---| -| Time field | `@timestamp` | -| Schedule | `5s` | -| Lookback | `24h` | -| Group by | `service_name` | +3. Confirm the query results. You should see one row for `service.name: checkout` with `p95_latency_ms` above 2000 and `severity: high` or `critical`. When you narrow the range to cover only the healthy periods before or after the degraded window, the row disappears. P95 drops below the threshold, which is the `no_breach` recovery condition. - +4. When the results look correct, select **Apply changes** to populate the rule form, then select **Next**. -Click **Save**. The rule is enabled automatically. + :::{note} + The sandbox time controls set the preview range only. They don't carry over to the rule's schedule or lookback window once the rule is running. + ::: -:::: +5. Complete the **Alert Condition** step (pane 1 of 4). The query you applied from the sandbox auto-fills **Mode**, **Time field**, and **Group fields**. Set the remaining fields, then select **Next**: -::::{step} Confirm the rule is running + - Under **Alert delay**, select **Breaches** and set the count to `2`. This requires the breach to persist across 2 consecutive evaluations before the episode moves to `active`. + - Set **Schedule** to every `5` minutes. + - Set **Lookback Window** to the last `2` hours. This ensures the rule can reach the pre-loaded sample data regardless of when you complete the tutorial. A shorter lookback would only work if the data timestamps fell within the last few minutes. -Wait 5 seconds, then run the following in Discover using the ES|QL mode. Replace `` with the `tutorial-error-rate` rule ID. +6. On the **Recovery Condition** step, confirm the defaults, then select **Next**: -:::{tip} -After saving the rule, open its details page. The rule ID is the string at the end of the browser URL. -::: + - **Recovery**: `Default recovery` + - **Recovery delay**: `Immediate` (No delay, recovers on first non-breach) -```esql -FROM .rule-events -| WHERE rule.id == "" -| SORT @timestamp DESC -| LIMIT 10 -``` + These defaults produce the automatic recovery behavior this tutorial demonstrates: the episode closes as soon as a scheduled run returns no breaching rows. + +7. On the **Details & Artifacts** step, enter the following, then select **Next**: -Check the following in the query results: + - **Name**: Checkout Service Latency + - **Description**: `Detects when P95 latency for the checkout service exceeds 2 seconds. Groups by service name and assigns severity: critical at 4000 ms, high at 2000 ms.` -| Field | What you should see | -|---|---| -| `@timestamp` | Recent timestamps updating every 5 seconds | -| `status` | `breached` confirms the query is finding matches | -| `episode.status` | `active` confirms episodes have opened for both services | -| `data.service_name` | `checkout` and `payments` confirms grouping is working | +8. On the **Actions** step, do not create an action policy (rules can run without notifications or actions setup). Select **Create rule** to create and enable the rule. :::: -::::{step} Confirm two episodes are open +::::{step} Confirm the rule is evaluating -In Discover, run: +1. Open Discover and switch to {{esql}} mode. -```esql -FROM .rule-events -| WHERE rule.id == "" -| STATS latest = MAX(@timestamp) BY episode.id, episode.status, data.service_name -| SORT latest DESC -``` +2. Set the date range to **Today** or **Last 1 hour**. The default "Last 15 minutes" is often too narrow to catch the first evaluation. + +3. Wait one schedule interval (5 minutes), then run the following query. Replace `` with the ID from the rule's detail page URL. -You should see two rows: one episode for `checkout` and one for `payments`, both with `episode.status: active`. + ```esql + FROM .rule-events + | WHERE rule.id == "" + | SORT @timestamp DESC + | LIMIT 10 + ``` + + :::{tip} + Open the rule's detail page. The rule ID is the last path segment in the browser URL. If you see 0 results, confirm the date range is wide enough and that at least one schedule interval has elapsed since you created the rule. + ::: + +4. Check these fields in the results: + + | Field | What to look for | + |---|---| + | `@timestamp` | A recent timestamp confirming the rule has run | + | `status` | `breached` when the query found rows above the threshold; `no_breach` when it found none | + | `episode.status` | `pending` for the first two consecutive breaching runs; `active` once the breach persists | + | `data.service.name` | `checkout` — confirms the grouping field was captured | + | `data.severity` | `high` or `critical` depending on the P95 value in the evaluated window | :::: -::::{step} Trigger recovery +::::{step} Observe the episode lifecycle -Delete the test data to clear the breach condition. Run the following in Dev Tools: +1. As the rule continues to run every 5 minutes, its lookback window slides forward in time. Once the degraded data at `22:13`–`22:27` is no longer within the 2-hour window, the rule returns no breach and the episode recovers automatically. -```json -POST logs-tutorial/_delete_by_query -{ - "query": { "match_all": {} } -} -``` +2. Run the following in Discover to track the episode: + + ```esql + FROM .rule-events + | WHERE rule.id == "" + | STATS latest = MAX(@timestamp) BY episode.id, episode.status, data.service.name + | SORT latest DESC + ``` -Wait up to 5 seconds, then re-run the query from the previous step in Discover. Both episodes should now show `episode.status: inactive`. +3. Confirm the episode moves from `active` to `inactive` without any manual intervention. This is the `no_breach` recovery strategy in action. :::: @@ -148,20 +338,18 @@ Wait up to 5 seconds, then re-run the query from the previous step in Discover. ## Key concepts demonstrated -This tutorial introduces the core concepts of the {{alerting-v2-system}}: - -* **Rules**: Query your data on a schedule and detect when conditions are met. Your rule ran every 5 seconds, checking for services with 2 or more errors in the last 24 hours. -* **Grouping**: Creates independent tracked series from a single rule. Because the rule grouped by `service_name`, `checkout` and `payments` were tracked separately — each with its own episode. -* **Episodes**: Follow a lifecycle driven by the rule's findings. Both episodes moved to `active` when errors existed, and recovered to `inactive` automatically when the data was deleted. -* **Rule events**: The underlying record of every evaluation. Every run wrote documents to `.rule-events`, giving you a full queryable history of what the rule found and when. - -## What's next +- **Rules**: Run an {{esql}} query on a schedule and detect when the results meet a condition. Each run computes P95 latency over the lookback window and finds a breach when the result exceeds 2000 ms. +- **Severity tiers**: The `CASE()` expression assigns `high` or `critical` based on the P95 value. These values are stored in `.rule-events` as `data.severity` and are queryable from Discover. +- **Episode lifecycle**: Episodes don't open on the first breach. With **Alert delay** set to **Breaches: 2**, the condition must persist for two consecutive evaluations before the episode moves to `active`. This filters out transient spikes. +- **Automatic recovery**: With **Recovery** set to **Default recovery**, the episode closes as soon as a scheduled run returns no breaching rows. No separate recovery query or manual step was required. +- **Rule events**: Every evaluation writes a document to `.rule-events`, giving you a full queryable history of what the rule found, when the episode opened, and when it recovered.