Skip to content

Commit 8013f54

Browse files
aantnaantn
authored
remove fluff (#1948)
1 parent 2ace013 commit 8013f54

59 files changed

Lines changed: 1059 additions & 1635 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/conf.py‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -109,10 +109,15 @@
109109
"how-it-works/privacy-and-security.html": "/master/setup-robusta/privacy-and-security.html",
110110
"how-it-works/index.html": "/master/playbook-reference/what-are-playbooks.html",
111111
"playbook-reference/examples.html": "/master/playbook-reference/prometheus-examples/index.html",
112-
"tutorials/playbook-track-changes.html": "/master/playbook-reference/kubernetes-examples/playbook-failed-liveness.html",
113-
"tutorials/playbook-job-failure.html": "/master/playbook-reference/kubernetes-examples/playbook-job-failure.html",
114-
"tutorials/playbook-failed-liveness.html": "/master/playbook-reference/kubernetes-examples/playbook-failed-liveness.html",
115-
"tutorials/playbook-track-secrets.html": "/master/playbook-reference/kubernetes-examples//playbook-track-secrets.html",
112+
"tutorials/playbook-track-changes.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
113+
"tutorials/playbook-job-failure.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
114+
"tutorials/playbook-failed-liveness.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
115+
"tutorials/playbook-track-secrets.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
116+
"playbook-reference/kubernetes-examples/playbook-failed-liveness.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
117+
"playbook-reference/kubernetes-examples/playbook-job-failure.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
118+
"playbook-reference/kubernetes-examples/playbook-track-changes.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
119+
"playbook-reference/kubernetes-examples/playbook-track-secrets.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
120+
"playbook-reference/kubernetes-examples/track-kubernetes-changes.html": "/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html",
116121
"tutorials/alert-remediation.html": "/master/playbook-reference/prometheus-examples/alert-remediation.html",
117122
"tutorials/alert-custom-enrichment.html": "/master/playbook-reference/prometheus-examples/alert-custom-enrichment.html",
118123
"catalog/sinks/slack.html": "/master/configuration/sinks/slack.html",
@@ -201,7 +206,16 @@
201206
"user-guide/robusta-cli.html": "/master/setup-robusta/installation/index.html",
202207
"advanced/index.html": "/master/setup-robusta/installation/index.html",
203208
"configuration/exporting/exporting-data.html": "/master/configuration/exporting/send-alerts-api.html",
204-
"configuration/alertmanager-integration/troubleshooting-alertmanager.html": "/master/configuration/exporting/send-alerts-api.html"
209+
"configuration/alertmanager-integration/troubleshooting-alertmanager.html": "/master/configuration/exporting/send-alerts-api.html",
210+
"configuration/alertmanager-integration/grafana-alert-manager.html": "/master/configuration/alertmanager-integration/grafana-self-hosted.html",
211+
"configuration/alertmanager-integration/grafana-cloud-mimir.html": "/master/configuration/alertmanager-integration/grafana-cloud.html",
212+
"playbook-reference/what-are-playbooks.html": "/master/playbook-reference/overview.html",
213+
"how-it-works/alert-builtin-enrichment.html": "/master/playbook-reference/builtin-alert-enrichment.html",
214+
"setup-robusta/installation/extend-prometheus-installation.html": "/master/setup-robusta/installation/standalone-installation.html",
215+
"playbook-reference/defining-playbooks/index.html": "/master/playbook-reference/index.html",
216+
"configuration/alertmanager-integration/alert-custom-prometheus.html": "/master/configuration/alertmanager-integration/embedded-prometheus.html#creating-custom-prometheus-alerts",
217+
"configuration/alertmanager-integration/index.html": "/master/configuration/index.html",
218+
"notification-routing/notification-routing-examples.html": "/master/notification-routing/index.html"
205219
}
206220

207221

‎docs/configuration/alertmanager-integration/_prometheus_flags_check.rst‎

Lines changed: 0 additions & 8 deletions
This file was deleted.

‎docs/configuration/alertmanager-integration/alert-manager.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
In-cluster AlertManager Integration
2-
****************************************
1+
AlertManager - in-cluster
2+
**************************
33

44
This guide shows how to send alerts from an existing AlertManager to Robusta in the same cluster.
55

6-
If your AlertManager is in a different cluster, refer to :ref:`External Prometheus`.
6+
If your AlertManager is in a different cluster, refer to :doc:`AlertManager - external <outofcluster-prometheus>`.
77

88
Send Alerts to Robusta
99
============================

‎docs/configuration/alertmanager-integration/customize-labels-priorities.rst‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
Customize Labels and Priorities
2+
=================================
3+
4+
Relabel Prometheus Alerts
5+
--------------------------
6+
7+
When sending Prometheus alerts to Robusta, alerts are mapped onto related Kubernetes resources, when possible. This mapping relies on the alerts having the following labels:
8+
9+
+---------------------------+-------------------------------------------+
10+
| Kubernetes Resource | Alert Labels |
11+
+===========================+===========================================+
12+
| Deployment | deployment, namespace |
13+
+---------------------------+-------------------------------------------+
14+
| DaemonSet | daemonset, namespace |
15+
+---------------------------+-------------------------------------------+
16+
| StatefulSet | statefulset, namespace |
17+
+---------------------------+-------------------------------------------+
18+
| Job | job_name, namespace |
19+
+---------------------------+-------------------------------------------+
20+
| Pod | pod, namespace |
21+
+---------------------------+-------------------------------------------+
22+
| HorizontalPodAutoscaler | horizontalpodautoscaler, namespace |
23+
+---------------------------+-------------------------------------------+
24+
| Node | node or instance (fallback if node |
25+
| | doesn't exist) |
26+
+---------------------------+-------------------------------------------+
27+
28+
If your alerts have different labels, you can change the mapping with the ``alertRelabel`` helm value.
29+
30+
A relabeling has 3 attributes:
31+
32+
* ``source``: The label's name on your alerts (which differs from the expected value in the above table)
33+
* ``target``: The standard label name that Robusta expects (a value from the table above)
34+
* ``operation``: Either ``add`` (default) or ``replace``. If ``add``, your custom mapping will be recognized in addition to Robusta's default mapping.
35+
36+
For example:
37+
38+
.. code-block:: yaml
39+
40+
alertRelabel:
41+
- source: "pod_name"
42+
target: "pod"
43+
operation: "add"
44+
- source: "deployment_name"
45+
target: "deployment"
46+
operation: "replace"
47+
- source: "job_name"
48+
target: "job"
49+
50+
Mapping Custom Alert Severity
51+
------------------------------
52+
53+
To help you prioritize alerts from different sources, Robusta maps alert severity to four standard levels:
54+
55+
* **HIGH** - requires your immediate attention - may indicate a service outage
56+
* **LOW** - minor problems and areas for improvement (e.g. performance) - to be reviewed periodically on a weekly or bi-weekly cadence
57+
* **INFO** - you probably want to be aware of these, but do not necessarily need to take action
58+
* **DEBUG** - debug only - can be ignored unless you're actively debugging an issue
59+
60+
You are free to interpret these levels differently, but the above is a good starting point for most companies.
61+
62+
Prometheus alerts are normalized to the above levels as follows:
63+
64+
+----------------------+--------------------+
65+
| Prometheus Severity | Robusta Severity |
66+
+======================+====================+
67+
| critical | HIGH |
68+
+----------------------+--------------------+
69+
| high | HIGH |
70+
+----------------------+--------------------+
71+
| medium | HIGH |
72+
+----------------------+--------------------+
73+
| error | HIGH |
74+
+----------------------+--------------------+
75+
| warning | LOW |
76+
+----------------------+--------------------+
77+
| low | LOW |
78+
+----------------------+--------------------+
79+
| info | INFO |
80+
+----------------------+--------------------+
81+
| debug | DEBUG |
82+
+----------------------+--------------------+
83+
84+
Prometheus alerts with a severity not in the above list are mapped to Robusta's INFO level.
85+
86+
You can map your own Prometheus severities, using the ``custom_severity_map`` Helm value. For example:
87+
88+
.. code-block:: yaml
89+
90+
globalConfig:
91+
custom_severity_map:
92+
# maps a p1 value on your own alerts to Robusta's HIGH value
93+
p1: high
94+
# maps a p2 value on your own alerts to Robusta's LOW value
95+
p2: low
96+
97+
The mapped values must be one of: ``high``, ``low``, ``info``, and ``debug``.

‎docs/configuration/alertmanager-integration/dynatrace.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,13 +35,13 @@ Step 2: Create a Dynatrace Problems Webhook
3535

3636
5. Set the **Custom payload** to the Dynatrace macro:
3737

38-
.. code-block:: json
38+
.. code-block:: text
3939
4040
{ProblemDetailsJSONv2}
4141
4242
6. Add the following **HTTP headers**:
4343

44-
.. code-block:: http
44+
.. code-block:: text
4545
4646
Authorization: Bearer <api-key>
4747
account-id: <account_id>

‎docs/configuration/alertmanager-integration/eks-managed-prometheus.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Since AWS Managed Prometheus doesn't have a built-in AlertManager, you'll need t
1818

1919
1. Set up Amazon Managed Grafana with your AMP workspace
2020
2. Configure Grafana alerts to send to Robusta
21-
3. See :doc:`grafana-alert-manager` for detailed Grafana alerting setup
21+
3. See :doc:`grafana-self-hosted` for detailed Grafana alerting setup
2222

2323
Configure Metric Querying
2424
=========================

‎docs/configuration/alertmanager-integration/embedded-prometheus.rst‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,76 @@ To allow the Grafana dashboard to persist after the Grafana instance restarts, y
5151
5252
Apply the change by performing a :ref:`Helm Upgrade <Simple Upgrade>`.
5353

54+
Creating Custom Prometheus Alerts
55+
----------------------------------
56+
57+
Prometheus Alerts are defined on Kubernetes using the PrometheusRule CRD.
58+
59+
Prerequisites
60+
^^^^^^^^^^^^^
61+
62+
Enable global rule selection for the Prometheus operator. Add the following config to your ``generated_values.yaml``. (By default Prometheus Operator picks up only certain new alerts, here we tell it to pick up all new alerts)
63+
64+
.. code-block:: yaml
65+
66+
kube-prometheus-stack:
67+
prometheus:
68+
prometheusSpec:
69+
ruleNamespaceSelector: {} # (1)
70+
ruleSelector: {} # (2)
71+
ruleSelectorNilUsesHelmValues: false # (3)
72+
73+
.. code-annotations::
74+
1. Add a namespace if you want Prometheus to identify rules created in specific namespaces. Leave ``{}`` to detect rules from any namespace.
75+
2. Add a label if you want Prometheus to detect rules with a specific selector. Leave ``{}`` to detect rules with any label.
76+
3. When set to `false`, Prometheus detects rules that are created directly, not just rules created using helm values file.
77+
78+
Defining an Alert
79+
^^^^^^^^^^^^^^^^^
80+
81+
As an example, we'll define an alert to find Pods with CPU usage over their request.
82+
83+
Save the following YAML into ``my_alert.yaml`` and run ``kubectl apply -f my_alert.yaml``
84+
85+
.. code-block:: yaml
86+
87+
apiVersion: monitoring.coreos.com/v1
88+
kind: PrometheusRule
89+
metadata:
90+
name: container-cpu-alert
91+
labels:
92+
prometheus: kube-prometheus
93+
role: alert-rules
94+
spec:
95+
groups:
96+
- name: container-cpu-usage
97+
rules:
98+
- alert: KubeContainerCPURequestAlert
99+
expr: |
100+
(rate(container_cpu_usage_seconds_total{container="stress"}[5m]) /
101+
on (container) kube_pod_container_resource_requests{resource="cpu", container="stress"}) > 0.75
102+
for: 1m
103+
labels:
104+
severity: warning
105+
annotations:
106+
summary: "Container CPU usage is above 75% of request for 5 minutes"
107+
description: "The container is using more than 75% of its requested CPU for 5 minutes."
108+
109+
Testing the Alert
110+
^^^^^^^^^^^^^^^^^
111+
112+
To test the alert, deploy a pod that uses more CPU than its request.
113+
114+
.. code-block:: bash
115+
116+
kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/cpu_throttling/throttling.yaml
117+
118+
You will know the alert was defined successfully when Prometheus fires an alert and you receive a notification in all configured sinks.
119+
120+
.. image:: /images/container_cpu_request_alert.png
121+
:width: 600
122+
:align: center
123+
54124
Troubleshooting
55125
---------------------
56126

0 commit comments

Comments
 (0)