1- .. :hide-toc:
2-
31Automate Responses to Application Logs
42==========================================
53
@@ -21,15 +19,15 @@ Let's get started! Here's a video explainer of how it works:
2119
2220 <div style =" position : relative ; height : 0 ; padding-bottom : 56.25% ;" > <iframe src =" https://www.youtube.com/embed/14Z4hVhlkWE" frameborder =" 0" allowfullscreen style =" position : absolute ; top : 0 ; left : 0 ; width : 100% ; height : 100% ;" ></iframe ></div >
2321
24- ** Step 1: Create a namespace for the demo **
22+ Step 1: Create a namespace for the demo
2523----------------------------------------------------
2624
2725.. code-block :: yaml
2826
2927 kubectl create namespace log-triggers
3028 kubectl config set-context --current --namespace log-triggers
3129
32- ** Step 2: Parse Logs into Metrics with Fluent Bit **
30+ Step 2: Parse Logs into Metrics with Fluent Bit
3331----------------------------------------------------
3432
3533First, lets configure Fluent Bit to monitor your pod logs and generate Prometheus metrics for specific log patterns.
@@ -173,15 +171,15 @@ Let's deploy the Fluent Bit DaemonSet:
173171
174172
175173
176- ** Step 3: Configure Prometheus **
174+ Step 3: Configure Prometheus
177175----------------------------------------------------
178176
179177In this step, we will configure Prometheus to:
180178
1811791. **Collect metrics from Fluent Bit ** via a `ServiceMonitor `
1821802. **Configure an alert ** based on the metrics extracted from the logs
183181
184- 1. ** Configure Prometheus to read the new ServiceMonitor **
182+ 1. Configure Prometheus to read the new ServiceMonitor
185183~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
186184
187185Assuming you're using Robusta's bundled Prometheus, add this to your ``generated_values.yaml ``:
@@ -202,7 +200,7 @@ To apply it, upgrade with helm:
202200 helm upgrade robusta robusta/robusta -f generated_values.yaml -set clusterName=YOUR_CLUSTER
203201
204202
205- 2. ** Configure an Alert **
203+ 2. Configure an Alert
206204~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
207205
208206This is the alerting rule that will be used to trigger an alert when a `MySqlConnectionErrors ` is detected in the logs (``mysql-alert.yaml ``):
@@ -241,7 +239,7 @@ To apply it run:
241239 kubectl apply -f mysql-alert.yaml
242240
243241
244- ** Step 4: Adding a Robusta playbook **
242+ Step 4: Adding a Robusta playbook
245243----------------------------------------------------
246244
247245Now, we'd like to configure an automated action that will run each time this alert is fired.
@@ -269,12 +267,13 @@ To apply it, upgrade with helm:
269267 helm upgrade robusta robusta/robusta -f generated_values.yaml -set clusterName=YOUR_CLUSTER
270268
271269
272- ** Step 5: See It in Action **
270+ Step 5: See It in Action
273271----------------------------------------------------
274272
275273Let’s test the full automation pipeline by generating a log line that simulates a MySQL connection error.
276274
277- 1. **Deploy a demo pod **
275+ 1. Deploy a demo pod
276+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
278277
279278Use this manifest to deploy a demo pod that prints to the logs whatever is sent to its API (``postlog.yaml ``):
280279
@@ -326,7 +325,8 @@ Apply it to your cluster:
326325
327326 kubectl apply -f postlog.yaml
328327
329- 2. **Generate MySQL errors in the logs **
328+ 2. Generate MySQL errors in the logs
329+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
330330
331331Call the pod’s API to print some simulated MySQL errors.
332332
@@ -351,7 +351,8 @@ Then, after 60 seconds, with 10 log lines:
351351
352352 This will produce 10 log lines containing the error. Fluent Bit will match the log lines and emit metrics, which Prometheus will collect.
353353
354- 3. **Trigger the alert and observe the automation **
354+ 3. Trigger the alert and observe the automation
355+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
355356
356357Wait a few minutes (typically up to 5) for the alert to fire. This delay is due to the ``for `` condition in the alert and Prometheus' ``group_interval ``.
357358
0 commit comments