Skip to content

Commit d163be1

Browse files
nastasha-solomonclaude
nastasha-solomon
and
claude
committed
Add experimental alerting features alerts pages
Adds four pages under kibana-alerting-experimental/alerts/: - alerts.md: section index covering alert lifecycle and .alert-actions - alerts/view-and-manage-alerts.md: triage, acknowledge, snooze, bulk actions - alerts/alert-states-and-fields-reference.md: episode states and field reference - alerts/query-alerts-and-signals-in-discover.md: ES|QL query guide Incorporates additions from PR #6395. Cross-references to pages in other PRs are commented out with TODO notes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 8f8c4d0 commit d163be1

5 files changed

Lines changed: 381 additions & 0 deletions

File tree

‎explore-analyze/alerting/kibana-alerting-experimental/alerts.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
---
2+
navigation_title: Alerts
3+
applies_to:
4+
stack: unavailable
5+
serverless: preview
6+
products:
7+
- id: kibana
8+
description: "Alert episodes in experimental alerting features: lifecycle states, series and episodes, signals versus alerts, and where to find them."
9+
---
10+
11+
# experimental alerting features alerts
12+
13+
When a rule fires repeatedly on the same problem, a flat list of events doesn't tell you when the issue started, whether it's still happening, or how long it's been going on. Alert episodes fill that gap. Each episode is a persistent record of one issue on one series, from first breach through recovery, with every evaluation appended to the same history. Nothing is overwritten.
14+
15+
<!--[CONTENT NEEDED for M2: UI. Once the navigation and page name have been confirmed, add instructions for opening the Alerts page.]
16+
-->
17+
18+
## Alert lifecycle [alert-lifecycle]
19+
20+
Every alert episode moves through these states:
21+
22+
```
23+
inactive → pending → active → recovering → inactive
24+
```
25+
26+
| State | What it means |
27+
| --- | --- |
28+
| Inactive | Problem fully resolved. You get a recovery notification. |
29+
| Pending | Errors detected, but the system is waiting to confirm it's a real problem before fully alerting. |
30+
| Active | Problem confirmed and ongoing. This is when you get notified. |
31+
| Recovering | Errors have stopped, but the system is waiting to confirm it's truly resolved. |
32+
33+
<!-- TODO: Uncomment when PR #6523 (rules) is merged:
34+
Activation and recovery thresholds control how many consecutive evaluations must agree, or how long the condition must persist, before transitioning. Refer to [Configure a rule](rules/configure-a-rule.md#activation-recovery-thresholds) to learn more about these settings.
35+
-->
36+
Activation and recovery thresholds control how many consecutive evaluations must agree, or how long the condition must persist, before transitioning.
37+
38+
### Example: First breach opens, first clear closes
39+
40+
A checkout-latency rule runs in Alert mode every 5 minutes. Latency breaches at 14:05 and clears at 14:50:
41+
42+
1. **14:00** — Routine check. p95 is within budget. The episode is `inactive`.
43+
2. **14:05** — p95 jumps to 3.1s. The first breach is detected. With no activation threshold, the episode opens immediately as `active`.
44+
3. **14:10–14:45** — Every evaluation finds high latency. The same episode stays `active`. No new episodes are created.
45+
4. **14:50** — p95 drops back under 2s. With no recovery threshold, the episode resolves immediately to `inactive`.
46+
47+
One problem is tracked in one episode, even though the rule evaluated many times while the condition was ongoing.
48+
49+
<!-- TODO: Add image once alert-episode-example-without-threshold.png is available in explore-analyze/images/
50+
:::{image} ../../images/alert-episode-example-without-threshold.png
51+
:alt: Timeline of a checkout-latency alert episode without thresholds. At 14:05, p95 jumps to 3.1s and the episode opens immediately as active. It stays active through 14:45. At 14:50, p95 drops back under 2s and the episode resolves immediately as inactive.
52+
:::
53+
-->
54+
55+
### Example: Waiting for confirmation before opening and closing
56+
57+
The same checkout-latency rule, now with an activation threshold of 2 consecutive breaches and a recovery threshold of 2 consecutive clears:
58+
59+
1. **14:00** — Routine check. p95 is within budget. The episode is `inactive`.
60+
2. **14:05** — p95 jumps to 3.1s. The first breach is detected. The episode is created in `pending` and the system starts counting consecutive breaches.
61+
3. **14:10** — p95 is still elevated. The second consecutive breach meets the activation threshold. The episode moves from `pending` to `active`, and the engineer is paged.
62+
4. **14:10–14:45** — Latency stays elevated. The episode remains `active`.
63+
5. **14:50** — p95 drops back under 2s. The first clean check moves the episode to `recovering`. The system starts counting consecutive clears.
64+
6. **14:55** — A second consecutive clear meets the recovery threshold. The episode moves from `recovering` to `inactive`.
65+
66+
Thresholds prevent brief spikes from opening episodes and transient dips from closing them prematurely. The episode waits in `pending` until the problem is confirmed, and waits in `recovering` until the resolution is confirmed.
67+
68+
<!-- TODO: Add image once alert-episode-example-with-activation-threshold.png is available in explore-analyze/images/
69+
:::{image} ../../images/alert-episode-example-with-activation-threshold.png
70+
:alt: Timeline of a checkout-latency alert episode with activation threshold of 2 and recovery threshold of 2. At 14:05, the first breach puts the episode in pending. At 14:10, the second consecutive breach moves it to active. At 14:50, the first clean check moves it to recovering. At 14:55, the second consecutive clear resolves it to inactive.
71+
:::
72+
-->
73+
74+
## Series
75+
76+
A series is the ongoing relationship between a rule and one specific thing it monitors.
77+
78+
Your rule monitors services. Each service it tracks has its own series, one for `checkout-service`, one for `payment-service`, and so on. A series exists for as long as that rule keeps monitoring that service.
79+
80+
Think of it like a patient's medical file. The file exists as long as the patient is in the system. Individual health incidents come and go, but the file persists.
81+
82+
<!-- TODO: Uncomment when PR #6523 (rules) is merged:
83+
For the fields that identify a series in alert event documents, refer to [Rule event and field reference](rules/rule-event-field-reference.md#rule-reference).
84+
-->
85+
86+
### How series and episodes relate
87+
88+
An episode lives inside a series. A series can contain many episodes over its lifetime, one for each time that service had a problem.
89+
90+
```
91+
Series: checkout-service
92+
│
93+
├── Episode 1: errors on April 10 (active → inactive)
94+
├── Episode 2: errors on April 15 (active → inactive)
95+
└── Episode 3: errors on April 18 (active right now)
96+
```
97+
98+
The series is the container. Episodes are the individual problems that happened within it. When the series breaches again after recovering, a new episode starts.
99+
100+
This means you can track "the checkout service was broken from 02:14 to 03:21" and "the payment service was broken at the same time" as separate episodes, even when both come from the same rule.
101+
102+
:::{tip}
103+
Snooze operates at the series level, not the episode level. If you snooze `checkout-service`, you're silencing all notifications from that series for the next X hours, regardless of how many new episodes start during that time. You're quieting a specific ongoing situation, not a single alert.
104+
:::
105+
106+
### A practical way to think about it
107+
108+
| Concept | Analogy |
109+
| --- | --- |
110+
| Rule | A security camera watching the building |
111+
| Series | The camera's feed for one specific door |
112+
| Episode | A specific incident caught on that feed |
113+
| Rule events | The individual video frames |
114+
115+
The camera runs continuously (rule), always watching door 3 (series). One night someone breaks in. That's an episode. The frames captured during the break-in are the rule events.
116+
117+
## Signals versus alerts
118+
119+
Every time a rule finds a match, it writes a document to `.rule-events`. Whether that document is a signal or an alert depends on the rule's mode, and that choice determines whether the system only records what happened or actively tracks it through to resolution.
120+
121+
A **signal** is a one-time observation. The system writes it and moves on, no lifecycle, no notifications, no follow-up. An **alert** participates in an episode. The system links it to every other document from the same problem, tracks the lifecycle states, and routes notifications through action policies.
122+
123+
| Type | What it is | When it's created |
124+
| --- | --- | --- |
125+
| Signal | A point-in-time record that the query matched (`type: signal`). Stored in `.rule-events`. | Rules in Detect mode |
126+
| Alert | A lifecycle-tracked episode with `type: alert` and `episode.*` fields. Stored in `.rule-events`. | Rules in Alert mode |
127+
128+
A rule in Detect mode only writes signals. It never opens episodes, so action policies have nothing to match against.
129+
130+
## Where alerts live
131+
132+
Alert events are stored in `.rule-events`. Triage actions (acknowledge, snooze, resolve) are stored in `.alert-actions`. Both are queryable in Discover.
133+
134+
Every time you take an action on an episode — acknowledging it, snoozing it, resolving it, editing its tags — {{kib}} writes a new document to `.alert-actions`. These documents are append-only and can be queried in Discover for auditing and metrics such as mean time to acknowledge (MTTA). For field definitions, refer to [Alert states and fields reference](alerts/alert-states-and-fields-reference.md#alert-states-reference).
135+
136+
<!--[CONTENT NEEDED for M2: UI. "V2 Alerting Preview" is a development-phase navigation label. Once the navigation and page name have been confirmed, add instructions for opening the Alerts page.]
137+
-->
138+
139+
### Data stream storage and retention
140+
141+
Both `.rule-events` and `.alert-actions` are data streams, append-only, time-series stores optimized for writes. On every rule evaluation, {{kib}} writes a **new document** to `.rule-events` rather than updating the previous one. Each document is a point-in-time snapshot. The `episode.status` field records the lifecycle state the episode was in at that exact evaluation. Nothing is overwritten.
142+
143+
Because every evaluation produces its own document, you can reconstruct the full history of an episode by querying all documents that share the same `episode.id`. Refer to [Query alerts and signals in Discover](alerts/query-alerts-and-signals-in-discover.md#explore-alerts-discover) for example queries.
144+
145+
Retention is managed automatically through ILM. Older backing indices move through storage tiers and are deleted when the retention window expires. You do not need to manually remove documents. {{kib}} manages versioning, retention, and lifecycle for both streams. Do not change their mappings or index settings.
146+
147+
## Related pages
148+
149+
- **[View and manage alerts](alerts/view-and-manage-alerts.md):** Open the alert episodes table, triage active episodes, and acknowledge, snooze, or resolve them.
150+
- **[Query alerts and signals in Discover](alerts/query-alerts-and-signals-in-discover.md):** Use {{esql}} to query `.rule-events` and `.alert-actions` for ad hoc analysis and dashboards.
151+
- **[Alert states and fields reference](alerts/alert-states-and-fields-reference.md):** Look up lifecycle states, field names, and episode document structure.
152+
<!-- TODO: Uncomment when PR #6525 (workflows/notifications) is merged:
153+
- **[Notifications](notifications.md):** Set up action policies to route alert episodes to the right people and channels.
154+
- **[Notification gating](notifications/notification-gating.md):** Understand how acknowledge, snooze, and deactivate control whether an episode triggers a notification.
155+
-->

‎explore-analyze/alerting/kibana-alerting-experimental/alerts/alert-states-and-fields-reference.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
navigation_title: Alert states and fields
3+
applies_to:
4+
stack: unavailable
5+
serverless: preview
6+
products:
7+
- id: kibana
8+
description: "Reference for episode status, `.rule-events` row status, and `.alert-actions` document fields in the experimental alerting features."
9+
---
10+
11+
# Alert states and fields reference [alert-states-reference]
12+
13+
14+
Alert states and fields are part of the experimental alerting features in Kibana. Use these tables when you read alert UI state, query `.rule-events` or `.alert-actions` in Discover, or align API payloads with what operators see. For triage controls (acknowledge, snooze, resolve, tags) and how they map to storage, refer to [Alert actions](view-and-manage-alerts.md#alert-actions).
15+
<!-- TODO: Uncomment when PR #6523 (rules) is merged:
16+
For rule evaluation fields on `.rule-events`, refer to [Rule event and field reference](../rules/rule-event-field-reference.md#rule-reference).
17+
-->
18+
19+
## Episode status
20+
21+
The `episode.status` field appears on documents with `type: alert` in `.rule-events`. It represents the current lifecycle state of the alert episode.
22+
23+
| Value | Description |
24+
|---|---|
25+
| `inactive` | Episode not in an active breach state in the lifecycle model. |
26+
| `pending` | Condition met but activation thresholds not yet satisfied. |
27+
| `active` | Episode is actively breaching per rule logic. |
28+
| `recovering` | Condition clearing but recovery thresholds not yet satisfied. |
29+
30+
<!--[CONTENT NEEDED for M2: M2 adds two first-class severity fields to the episode document:
31+
32+
- `episode.severity` - The severity from the most recent rule event (current state). Updated each evaluation.
33+
- `episode.severity_max` - The highest severity seen over the episode's lifetime. Never decreases; enables "peaked at CRITICAL" display in the episode detail UI.
34+
35+
Add a new table or rows for these fields once M2 ships. The episode detail UI is also expected to change to surface these fields directly. Several M2 details are still open: the enforced value set (or lack thereof), whether de-escalation triggers policy re-evaluation, and whether manual override is supported. Do not document specifics until resolved.]
36+
-->
37+
38+
## Rule event status
39+
40+
The `status` field appears on all documents in `.rule-events`, for both `type: signal` and `type: alert`. It reflects the outcome of a single rule evaluation row, independent of the episode lifecycle.
41+
42+
| Value | Description |
43+
|---|---|
44+
| `breached` | Condition met for this evaluation row. |
45+
| `recovered` | Recovery path satisfied for this evaluation row. |
46+
| `no_data` | No-data handling produced a no-data style outcome for this evaluation. |
47+
48+
## Alert action document fields
49+
50+
When a user or the system records an action on an alert episode, {{kib}} writes a document to `.alert-actions`. Use this stream for triage history, operational metrics such as mean time to acknowledge (MTTA), and auditing. It does not store what your rule query returned on each run — that output is in `.rule-events`.
51+
52+
| Field | Type | Description |
53+
|---|---|---|
54+
| `@timestamp` | date | When the action was recorded. |
55+
| `episode.id` | keyword | Target episode. |
56+
| `rule.id` | keyword | Rule that owns the episode. |
57+
| `action.type` | keyword | The action type, for example: <br>- `acknowledge`: User acknowledged the alert.<br>- `snooze`: Notifications snoozed for a period.<br>- `tag`: Tag applied to the alert.<br>- `fire`: Notification or escalation fired for the episode.<br>- `unmatched`: No action policy matched the episode, so no workflow ran for it under these policies. <br><br> For the full set of action types and UI behavior, refer to [Alert actions](view-and-manage-alerts.md#alert-actions). |
58+
| `episode.status_count` | long | Count of consecutive evaluations in the current `episode.status`. Only set when `episode.status` is `pending` or `recovering`.<br>For example, if the episode stays `pending` for three rule evaluations in a row, the value is `3`. |

‎explore-analyze/alerting/kibana-alerting-experimental/alerts/query-alerts-and-signals-in-discover.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
navigation_title: Query alerts in Discover
3+
applies_to:
4+
stack: unavailable
5+
serverless: preview
6+
products:
7+
- id: kibana
8+
description: "Use {{esql}} in Discover against `.rule-events` and `.alert-actions`: sample queries, trends, and MTTA-style analysis for the experimental alerting features."
9+
---
10+
11+
# Query alerts and signals in Discover [explore-alerts-discover]
12+
13+
14+
Querying alerts and signals in Discover is part of the experimental alerting features in Kibana. Discover gives you direct {{esql}} access to everything the experimental alerting features record, including rule evaluation history, episode progressions, triage actions, and operational metrics like mean time to acknowledge.
15+
16+
The Alerts UI shows current episode state. Discover lets you go further: ask arbitrary questions, spot trends over time, replay how a specific incident unfolded, or correlate alert history with other data in your environment.
17+
18+
To use this page, open Discover, select {{esql}}, paste a query from the examples below, then adjust the time range and placeholders (`YOUR_RULE_ID`, `YOUR_GROUP_HASH`) to match your environment.
19+
20+
<!--[CONTENT NEEDED: The queries on this page use `.rule-events` and `.alert-actions` directly. Confirm whether these will remain the intended query surface, or whether users should query an ES|QL view or a stable user-facing data stream instead. Update all examples accordingly before publishing.]
21+
22+
<!--[CONTENT NEEDED for M2: Review and expand the query examples below once M2 field renames (`group_hash` → `series.key`, new `series.tracked_by`, `episode.severity`, `episode.severity_max`) are finalized. Add examples that take advantage of the new first-class severity and series fields.]
23+
-->
24+
25+
For field names, types, and episode fields, refer to [Alert states and fields reference](alert-states-and-fields-reference.md#alert-states-reference). For triage in the product UI, refer to [View, manage, and reference alerts](view-and-manage-alerts.md).
26+
<!-- TODO: Uncomment when PR #6523 (rules) is merged and restore full sentence:
27+
For field names, types, and episode fields, refer to [Alert states and fields reference](alert-states-and-fields-reference.md#alert-states-reference) and [Rule event and field reference](../rules/rule-event-field-reference.md#rule-reference). For triage in the product UI, refer to [View, manage, and reference alerts](view-and-manage-alerts.md).
28+
-->
29+

0 commit comments

Comments
 (0)