Skip to content

Commit 5ff3386

Browse files
Avi-RobustaAvi-Robusta
authored
[ROB-2217] Docs for Additional alert forwarding (#1930)
* added newrelic integration * added pagerduty alerts * dynatrace support * removed unneeded generated docs
1 parent dbdbaba commit 5ff3386

4 files changed

Lines changed: 371 additions & 0 deletions

File tree

‎docs/configuration/alertmanager-integration/dynatrace.rst‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
Dynatrace Integration with Robusta
2+
==================================
3+
4+
.. note::
5+
This feature is available with the Robusta SaaS platform and self-hosted commercial plans. It is not available in the open-source version.
6+
7+
This guide explains how to forward **Dynatrace problem notifications** to Robusta via a webhook.
8+
9+
Requirements
10+
------------
11+
12+
- Robusta is deployed and running.
13+
- You have access to the Robusta UI (to create an API key and find your ``account_id``).
14+
- You have admin access to the Dynatrace platform (to configure Problem notifications).
15+
16+
Step 1: Get Robusta Account ID and API Key
17+
------------------------------------------
18+
19+
1. In the Robusta UI, go to **Settings → API Keys**.
20+
2. Click **New API Key**, select **Alerts: Write** permissions, and **Save**.
21+
3. Copy the generated API key — you will use it as a Bearer token in Dynatrace.
22+
4. Find your ``account_id``:
23+
- In Robusta, the ``account_id`` appears in your **generated_values.yaml** file (from installation), or
24+
- In the Robusta UI under **Settings → Workspace**.
25+
26+
Step 2: Create a Dynatrace Problems Webhook
27+
-------------------------------------------
28+
29+
1. In Dynatrace, open the **command palette** (**Cmd+K** on macOS / **Ctrl+K** on Windows/Linux).
30+
2. Search for **problem notification** and open **Problem notifications**.
31+
3. Click **Add notification** and choose **Webhook**.
32+
4. Configure the **URL**:
33+
34+
``https://api.robusta.dev/integrations/generic/dynatrace``
35+
36+
5. Set the **Custom payload** to the Dynatrace macro:
37+
38+
.. code-block:: json
39+
40+
{ProblemDetailsJSONv2}
41+
42+
6. Add the following **HTTP headers**:
43+
44+
.. code-block:: http
45+
46+
Authorization: Bearer <api-key>
47+
account-id: <account_id>
48+
49+
Replace ``<api-key>`` with the Robusta API key from Step 1 and ``<account_id>`` with your Robusta account ID.
50+
51+
7. Save the webhook notification.
52+

‎docs/configuration/alertmanager-integration/newrelic.rst‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
New Relic Integration with Robusta
2+
==================================
3+
4+
.. note::
5+
This feature is available with the Robusta SaaS platform and self-hosted commercial plans. It is not available in the open-source version.
6+
7+
This guide explains how to route New Relic alerts to Robusta's UI via a webhook.
8+
9+
Requirements
10+
------------
11+
12+
- Robusta must already be deployed and running in your environment.
13+
- You have admin access to the Robusta UI (to create API keys and view your ``account_id``).
14+
- You have admin access to New Relic Alerts (to create Destinations and Workflows).
15+
16+
Integration Steps
17+
-----------------
18+
19+
We will configure:
20+
21+
1. A Robusta API key and account ID.
22+
2. A New Relic **Webhook Destination** pointing to Robusta.
23+
3. A New Relic **Workflow** that always sends notifications to Robusta using a custom payload template.
24+
25+
Step 1: Get Account ID and API Key
26+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
27+
28+
Obtain your Robusta ``account_id`` and create an API key:
29+
30+
1. In the Robusta UI, navigate to **Settings → API Keys**.
31+
2. Click **New API Key**.
32+
3. Name the key **New Relic**, grant it **Read/Write** access to alerts, and click **Generate API Key**.
33+
4. Copy and securely store the generated API key — you’ll use it as a **Bearer token** in New Relic.
34+
5. In **Settings → Workspace**, copy your **account_id**.
35+
36+
Step 2: Create a Webhook Destination in New Relic
37+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38+
39+
In New Relic:
40+
41+
1. Go to **Alerts → Destinations**.
42+
2. Click **New destination** → choose **Webhook**.
43+
3. Configure:
44+
45+
- **URL**: ``https://api.robusta.dev/integrations/generic/newrelic``
46+
- **Authentication**: **Bearer token**
47+
- **Token**: paste the **Robusta API key** from Step 1.
48+
49+
4. Save the destination. (Optional) Use **Send test notification** to verify connectivity.
50+
51+
Step 3: Configure a Workflow to Send Alerts
52+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
53+
54+
1. Go to **Alerts → Workflows**.
55+
2. Use an existing workflow or click **New workflow**.
56+
3. **When**: set to **Always send notifications** (or choose your preferred filter).
57+
4. **Then**: add an **Action → Webhook** and select the **Robusta** destination created in Step 2.
58+
5. **Template**: choose **Custom payload** and paste the JSON template below.
59+
6. Replace ``ACCOUNT_ID_HERE`` with your **account_id** from Step 1.
60+
7. Save and enable the workflow.
61+
62+
Webhook Payload Template (JSON)
63+
-------------------------------
64+
65+
Paste this into the **Template** field for the webhook action. Replace ``ACCOUNT_ID_HERE`` with your actual account ID.
66+
67+
.. code-block:: json
68+
69+
{
70+
"account_id": "ACCOUNT_ID_HERE",
71+
"title": {{#if annotations}}{{#if annotations.title}}{{ json annotations.title.[0] }}{{else}}"N/A"{{/if}}{{else}}"N/A"{{/if}},
72+
"description": {{#if annotations}}{{#if annotations.description}}{{ json annotations.description.[0] }}{{else}}{{#if annotations.title}}{{ json annotations.title.[0] }}{{else}}"N/A"{{/if}}{{/if}}{{else}}"N/A"{{/if}},
73+
"source": "newrelic",
74+
"priority": {{#if priority}}{{ json priority }}{{else if severity}}{{ json severity }}{{else}}null{{/if}},
75+
"aggregation_key": {{#if accumulations.conditionName}}{{ json accumulations.conditionName.[0] }}{{else}}null{{/if}},
76+
"starts_at": {{#if createdAt}}{{#timezone createdAt 'UTC'}}{{/timezone}}{{else}}"2025-09-22 00:00:00 UTC"{{/if}},
77+
"ends_at": {{#eq state "CLOSED"}}{{#if updatedAt}}{{#timezone updatedAt 'UTC'}}{{/timezone}}{{else}}null{{/if}}{{else}}null{{/eq}},
78+
"cluster": {{#if [latest clusterName]}}
79+
{{ json [latest clusterName] }}
80+
{{else if accumulations}}
81+
{{#if accumulations.tag}}
82+
{{#if accumulations.tag.k8s.clusterName}}
83+
{{ json accumulations.tag.k8s.clusterName.[0] }}
84+
{{else if accumulations.tag.cluster}}
85+
{{ json accumulations.tag.cluster.[0] }}
86+
{{else if tag.k8s.clusterName}}
87+
{{ json tag.k8s.clusterName.[0] }}
88+
{{else}}
89+
"external"
90+
{{/if}}
91+
{{else}}
92+
"external"
93+
{{/if}}
94+
{{else}}
95+
"external"
96+
{{/if}},
97+
"raw_alert_data": {{ json . }}
98+
}
99+
100+
Optional: Field Mapping Notes
101+
-----------------------------
102+
103+
- ``title`` / ``description`` are sourced from ``annotations`` when present, otherwise fall back to ``"N/A"``.
104+
- ``priority`` prefers ``priority`` and falls back to ``severity`` when available.
105+
- ``aggregation_key`` maps to the Alert name to help group similar alerts in Robusta.
106+
- ``starts_at`` / ``ends_at`` are normalized to **UTC** by New Relic templating.
107+
- ``cluster`` attempts multiple common locations for Kubernetes cluster identifiers and defaults to ``"external"``.
108+
- ``raw_alert_data`` includes the full New Relic payload for troubleshooting in Robusta.
109+
110+
Validation
111+
----------
112+
113+
- Trigger a test alert that matches the workflow (or use **Send test notification** on the Destination).
114+
- In Robusta's UI, verify the alert appears with the expected title, description, priority, and cluster.

‎docs/configuration/alertmanager-integration/pagerduty-alerting.rst‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
PagerDuty Integration
2+
****************************************
3+
4+
PagerDuty can send both incidents and full alert-level data to Robusta for enrichment and resolution tracking.
5+
6+
This guide explains how to set up both integrations, and how to forward alerts from sources like Alertmanager using PagerDuty AIOps Event Orchestration.
7+
8+
After completing this tutorial, we recommend confirming that findings appear correctly in the Robusta UI timeline.
9+
10+
Prerequisite
11+
=================================
12+
* A Robusta account with API access.
13+
* For alert-level forwarding: an AIOps-enabled PagerDuty plan.
14+
* A Robusta cluster name, as defined in your ``generated_values.yaml``.
15+
16+
Send Incidents to Robusta
17+
============================
18+
19+
To send PagerDuty incidents (triggered/resolved) to Robusta:
20+
21+
Step 1: Create a Robusta API Key
22+
---------------------------------
23+
1. In the Robusta UI, navigate to **Settings → API Keys**.
24+
2. Click **New API Key**.
25+
3. Name the key ``PagerDuty``, grant it ``Read/Write`` access to alerts, then click **Generate API Key**.
26+
4. Copy and securely save the generated API key.
27+
28+
Step 2: Get Your Account ID
29+
-------------------------------
30+
1. From your Robusta ``generated_values.yaml`` file, locate and copy the ``account_id``.
31+
It will look like:
32+
33+
.. code-block::
34+
35+
account_id: caa68d87-XXXX-XXXX-XXXX-a6514ccb11eb
36+
37+
Step 3: Create a Webhook in PagerDuty
38+
-------------------------------------------
39+
1. In PagerDuty, go to **Integrations → Generic Webhooks v3**.
40+
2. Click **New Webhook**.
41+
42+
Step 4: Configure the Webhook
43+
----------------------------------------
44+
Set the following values:
45+
46+
* **Webhook URL**:
47+
48+
.. code-block::
49+
50+
https://api.robusta.dev/integrations/generic/pagerduty/incidents
51+
52+
* **Custom Headers**:
53+
54+
.. code-block::
55+
56+
account-id: <your_account_id>
57+
Authorization: Bearer <ROBUSTA_API_KEY>
58+
59+
Replace the placeholders with values from Step 1 and Step 2.
60+
61+
Once configured, PagerDuty will begin sending incident-level data to Robusta. These will appear in the Robusta UI timeline.
62+
63+
Send Alerts to Robusta (AIOps Plans Only)
64+
==============================================
65+
66+
For Robusta AIOps users, you can send full alert-level data in addition to incidents.
67+
68+
Step 1: Go to AIOps → Event Orchestration
69+
----------------------------------------------
70+
1. In PagerDuty, go to **AIOps → Event Orchestration**.
71+
2. Click **New Orchestration** and name it ``Robusta``.
72+
73+
Forwarding Alerts via Event Orchestration
74+
----------------------------------------------
75+
76+
**Recommended for Alertmanager Users**
77+
78+
If you already use PagerDuty Event Orchestration to forward alerts from systems like Alertmanager, we recommend forwarding those alerts **directly to Robusta**.
79+
80+
Why this matters
81+
~~~~~~~~~~~~~~~~
82+
Robusta’s AIOps integration works best when it receives structured alert data directly. Routing alerts through intermediate services or limited integrations may strip important context.
83+
84+
Step 1: Create an Integration for Alertmanager
85+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
86+
1. In Event Orchestration, create a new **Integration** named ``Alertmanager``.
87+
2. Use the following webhook URL:
88+
89+
.. code-block::
90+
91+
https://api.robusta.dev/integrations/generic/pagerduty/alerts
92+
93+
Step 2: Configure a Service Route
94+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
95+
1. At the end of your orchestration rule chain, add a **Service Route**.
96+
2. Route events with the source ``Alertmanager`` to the ``Robusta`` webhook.
97+
98+
This ensures full alert payloads are delivered directly to Robusta.
99+
100+
Step 2: Add a New Rule
101+
----------------------------
102+
1. Under your ``Robusta`` orchestration, add a new rule.
103+
2. For **"When this rule is applied"**, select **Always, for all events**.
104+
3. Click **Next**.
105+
106+
Step 3: Add a Webhook Action
107+
-------------------------------
108+
1. Under **Automation → Webhook Actions**, enable:
109+
``Use webhooks if an event reaches this rule``.
110+
111+
2. Configure the webhook as follows:
112+
113+
* **Name**: ``Robusta``
114+
115+
* **URL**:
116+
117+
.. code-block::
118+
119+
https://api.robusta.dev/integrations/generic/pagerduty/alerts
120+
121+
* **Headers**:
122+
123+
.. code-block::
124+
125+
account-id: <your_account_id>
126+
Authorization: Bearer <ROBUSTA_API_KEY>
127+
128+
* **Body (key-value pairs)**:
129+
130+
.. code-block:: json
131+
132+
{
133+
"custom_details": "{{event.custom_details}}",
134+
"summary": "{{event.summary}}",
135+
"source": "{{event.source}}",
136+
"dedup_key": "{{event.dedup_key}}",
137+
"severity": "{{event.severity}}"
138+
}
139+
140+
3. Click **Save**.
141+
142+
Step 4: Route Matching
143+
---------------------------
144+
In the routing configuration, route it to the the service of your choice or you can do it dynamically by setting the route to:
145+
146+
.. code-block::
147+
148+
event.source
149+
150+
Using this regular expression:
151+
152+
.. code-block::
153+
154+
.*
155+
156+
This ensures all alert sources are routed properly to their Pagerduty Services.
157+
158+
Verify it Works
159+
=============================
160+
161+
To confirm the integration:
162+
163+
* Trigger an alert from Alertmanager, Nagios, or another connected system.
164+
* Check that the alert appears in the Robusta UI timeline under the correct cluster.
165+
* Confirm the incident is routed to the correct Service in Pagerduty.
166+
167+
Optional: Cluster Name via Query Param
168+
============================================
169+
170+
You can specify the target cluster using a query parameter in the webhook URL:
171+
172+
.. code-block::
173+
174+
https://api.robusta.dev/integrations/generic/pagerduty/incidents?cluster=your-cluster-name
175+
176+
This is useful for multi-cluster setups where Robusta should assign findings to a specific cluster.
177+
178+
For example:
179+
180+
.. code-block::
181+
182+
https://api.robusta.dev/integrations/generic/pagerduty/incidents?cluster=test-cluster
183+
184+
This will create findings in ``test-cluster``.

‎docs/configuration/index.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,32 @@ Connect your monitoring system to Robusta, to enrich alerts and apply automation
2424

2525
Forward Nagios alerts by webhook
2626

27+
.. grid-item-card:: :octicon:`bell;1em;` NewRelic
28+
:class-card: sd-bg-light sd-bg-text-light
29+
:link: alertmanager-integration/newrelic
30+
:link-type: doc
31+
32+
Forward NewRelic alerts by webhook
33+
2734
.. grid-item-card:: :octicon:`bell;1em;` SolarWinds
2835
:class-card: sd-bg-light sd-bg-text-light
2936
:link: alertmanager-integration/solarwinds
3037
:link-type: doc
3138

3239
Forward SolarWinds alerts by webhook
3340

41+
.. grid-item-card:: :octicon:`bell;1em;` Dynatrace
42+
:class-card: sd-bg-light sd-bg-text-light
43+
:link: alertmanager-integration/dynatrace
44+
:link-type: doc
45+
46+
Forward Dynatrace alerts by webhook
47+
48+
.. grid-item-card:: :octicon:`bell;1em;` Pagerduty
49+
:class-card: sd-bg-light sd-bg-text-light
50+
:link: alertmanager-integration/pagerduty-alerting
51+
:link-type: doc
52+
53+
Forward PagerDuty incidents and alerts by webhook
54+
3455
**Have alerts elsewhere?** Send alerts via the generic :doc:`HTTP webhook endpoint <exporting/custom-webhooks>`.

0 commit comments

Comments
 (0)