apps sc: sum ThanosReceiveTrafficBelowThreshold alert to adjust for multiple receive distributors

Ajarmar · Ajarmar · commit 8467894ae5f2 · 2025-07-31T10:03:58.000+02:00
diff --git a/helmfile.d/charts/prometheus-alerts/files/thanos.yaml b/helmfile.d/charts/prometheus-alerts/files/thanos.yaml
@@ -289,9 +289,9 @@ groups:
         to avg. 12-hr ingestion rate.
     expr: |
       (
-        avg_over_time(rate(http_requests_total{job=~".*thanos-receiver-receive.*", code=~"2..", handler="receive"}[5m])[1h:5m])
+        avg_over_time(sum by (tenant) (rate(http_requests_total{job=~".*thanos-receiver-receive.*", code=~"2..", handler="receive"}[5m]))[1h:5m])
       /
-        avg_over_time(rate(http_requests_total{job=~".*thanos-receiver-receive.*", code=~"2..", handler="receive"}[5m])[12h:5m])
+        avg_over_time(sum by (tenant) (rate(http_requests_total{job=~".*thanos-receiver-receive.*", code=~"2..", handler="receive"}[5m]))[12h:5m])
       ) * 100 < 50
     for: 1h
     labels:

Original file line number	Diff line number	Diff line change
`@@ -289,9 +289,9 @@ groups:`
`289`	`289`	`to avg. 12-hr ingestion rate.`
`290`	`290`	`expr: \|`
`291`	`291`	`(`
`292`		`- avg_over_time(rate(http_requests_total{job=~".thanos-receiver-receive.", code=~"2..", handler="receive"}[5m])[1h:5m])`
	`292`	`+ avg_over_time(sum by (tenant) (rate(http_requests_total{job=~".thanos-receiver-receive.", code=~"2..", handler="receive"}[5m]))[1h:5m])`
`293`	`293`	`/`
`294`		`- avg_over_time(rate(http_requests_total{job=~".thanos-receiver-receive.", code=~"2..", handler="receive"}[5m])[12h:5m])`
	`294`	`+ avg_over_time(sum by (tenant) (rate(http_requests_total{job=~".thanos-receiver-receive.", code=~"2..", handler="receive"}[5m]))[12h:5m])`
`295`	`295`	`) * 100 < 50`
`296`	`296`	`for: 1h`
`297`	`297`	`labels:`