|
| 1 | +Customize Labels and Priorities |
| 2 | +================================= |
| 3 | + |
| 4 | +Relabel Prometheus Alerts |
| 5 | +-------------------------- |
| 6 | + |
| 7 | +When sending Prometheus alerts to Robusta, alerts are mapped onto related Kubernetes resources, when possible. This mapping relies on the alerts having the following labels: |
| 8 | + |
| 9 | ++---------------------------+-------------------------------------------+ |
| 10 | +| Kubernetes Resource | Alert Labels | |
| 11 | ++===========================+===========================================+ |
| 12 | +| Deployment | deployment, namespace | |
| 13 | ++---------------------------+-------------------------------------------+ |
| 14 | +| DaemonSet | daemonset, namespace | |
| 15 | ++---------------------------+-------------------------------------------+ |
| 16 | +| StatefulSet | statefulset, namespace | |
| 17 | ++---------------------------+-------------------------------------------+ |
| 18 | +| Job | job_name, namespace | |
| 19 | ++---------------------------+-------------------------------------------+ |
| 20 | +| Pod | pod, namespace | |
| 21 | ++---------------------------+-------------------------------------------+ |
| 22 | +| HorizontalPodAutoscaler | horizontalpodautoscaler, namespace | |
| 23 | ++---------------------------+-------------------------------------------+ |
| 24 | +| Node | node or instance (fallback if node | |
| 25 | +| | doesn't exist) | |
| 26 | ++---------------------------+-------------------------------------------+ |
| 27 | + |
| 28 | +If your alerts have different labels, you can change the mapping with the ``alertRelabel`` helm value. |
| 29 | + |
| 30 | +A relabeling has 3 attributes: |
| 31 | + |
| 32 | +* ``source``: The label's name on your alerts (which differs from the expected value in the above table) |
| 33 | +* ``target``: The standard label name that Robusta expects (a value from the table above) |
| 34 | +* ``operation``: Either ``add`` (default) or ``replace``. If ``add``, your custom mapping will be recognized in addition to Robusta's default mapping. |
| 35 | + |
| 36 | +For example: |
| 37 | + |
| 38 | +.. code-block:: yaml |
| 39 | +
|
| 40 | + alertRelabel: |
| 41 | + - source: "pod_name" |
| 42 | + target: "pod" |
| 43 | + operation: "add" |
| 44 | + - source: "deployment_name" |
| 45 | + target: "deployment" |
| 46 | + operation: "replace" |
| 47 | + - source: "job_name" |
| 48 | + target: "job" |
| 49 | +
|
| 50 | +Mapping Custom Alert Severity |
| 51 | +------------------------------ |
| 52 | + |
| 53 | +To help you prioritize alerts from different sources, Robusta maps alert severity to four standard levels: |
| 54 | + |
| 55 | +* **HIGH** - requires your immediate attention - may indicate a service outage |
| 56 | +* **LOW** - minor problems and areas for improvement (e.g. performance) - to be reviewed periodically on a weekly or bi-weekly cadence |
| 57 | +* **INFO** - you probably want to be aware of these, but do not necessarily need to take action |
| 58 | +* **DEBUG** - debug only - can be ignored unless you're actively debugging an issue |
| 59 | + |
| 60 | +You are free to interpret these levels differently, but the above is a good starting point for most companies. |
| 61 | + |
| 62 | +Prometheus alerts are normalized to the above levels as follows: |
| 63 | + |
| 64 | ++----------------------+--------------------+ |
| 65 | +| Prometheus Severity | Robusta Severity | |
| 66 | ++======================+====================+ |
| 67 | +| critical | HIGH | |
| 68 | ++----------------------+--------------------+ |
| 69 | +| high | HIGH | |
| 70 | ++----------------------+--------------------+ |
| 71 | +| medium | HIGH | |
| 72 | ++----------------------+--------------------+ |
| 73 | +| error | HIGH | |
| 74 | ++----------------------+--------------------+ |
| 75 | +| warning | LOW | |
| 76 | ++----------------------+--------------------+ |
| 77 | +| low | LOW | |
| 78 | ++----------------------+--------------------+ |
| 79 | +| info | INFO | |
| 80 | ++----------------------+--------------------+ |
| 81 | +| debug | DEBUG | |
| 82 | ++----------------------+--------------------+ |
| 83 | + |
| 84 | +Prometheus alerts with a severity not in the above list are mapped to Robusta's INFO level. |
| 85 | + |
| 86 | +You can map your own Prometheus severities, using the ``custom_severity_map`` Helm value. For example: |
| 87 | + |
| 88 | +.. code-block:: yaml |
| 89 | +
|
| 90 | + globalConfig: |
| 91 | + custom_severity_map: |
| 92 | + # maps a p1 value on your own alerts to Robusta's HIGH value |
| 93 | + p1: high |
| 94 | + # maps a p2 value on your own alerts to Robusta's LOW value |
| 95 | + p2: low |
| 96 | +
|
| 97 | +The mapped values must be one of: ``high``, ``low``, ``info``, and ``debug``. |
0 commit comments