Skip to content

Commit 267e759

Browse files
scotwellsscotwells
committed
feat(telemetry): alert on gateway controller reconcile-error ratio
Adds two new PrometheusRule alerts and a matching runbook for the case where the gateway controller is stuck retrying rejected API server writes. Prod context: since v0.23.4 shipped PR #217 (cert-listener withholding), controller_runtime_reconcile_total{controller="gateway",result="error"} has been ~1963 vs result="success" ~395 — an 83% failure rate with no alert firing. The regression: witholding all listeners on a gateway produces a downstream Gateway with zero listeners, which the Gateway-API CRD rejects as "Required value". The controller hot-loops silently. Alerts added to config/telemetry/alerts/gateways.yaml: - GatewayControllerReconcileErrorRatioHigh (warning, >20% for 15m) - GatewayControllerReconcileErrorRatioCritical (critical, >50% for 10m) Expression uses sum without(result) to aggregate across result label dimensions so the ratio is computed correctly and the alert carries only the controller label (not result="error"). Also adds promtool unit tests (test/prometheus-rules/gateways/) and a runbook (docs/runbooks/gateway-controller-health.md) covering meaning, impact, diagnosis, and remediation for both tiers. Closes #212 (alerting gap identified in that issue).
1 parent b1eded2 commit 267e759

4 files changed

Lines changed: 280 additions & 0 deletions

File tree

‎config/telemetry/alerts/gateways.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,51 @@ spec:
5050
summary: "Gateway {{ $labels.resource_name }} has been degraded for over 60 seconds"
5151
description: "Gateway {{ $labels.resource_name }} in namespace {{ $labels.resource_namespace }} has been in a degraded state for over 60 seconds without recovering, which exceeds the 60-second SLO threshold."
5252

53+
# Controller reconcile-error alerts detect when the gateway controller is
54+
# failing to write downstream Gateway objects. A high error ratio means updates
55+
# are being rejected by the API server (e.g. invalid objects), the controller
56+
# is hot-looping, and the stale configuration stays programmed at the edge.
57+
- name: nso-gateway-controller-health
58+
interval: 30s
59+
rules:
60+
# Fires when more than 20% of gateway controller reconcile attempts are
61+
# failing over a 10-minute window, sustained for 15 minutes. A common cause
62+
# is a downstream Gateway object failing CRD validation (e.g. all listeners
63+
# withheld leaves spec.listeners empty, which is rejected as Required value).
64+
# While this fires, controller changes are not reaching the edge.
65+
- alert: GatewayControllerReconcileErrorRatioHigh
66+
expr: |
67+
(
68+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway",result="error"}[10m]))
69+
/
70+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway"}[10m]))
71+
) > 0.2
72+
for: 15m
73+
labels:
74+
severity: warning
75+
annotations:
76+
summary: "Gateway controller reconcile error ratio is {{ $value | humanizePercentage }} (threshold 20%)"
77+
description: "More than 20% of gateway controller reconcile attempts are failing. The controller cannot write downstream Gateway objects — changes are not reaching the edge and the stale configuration stays programmed. Common causes: a downstream Gateway fails CRD validation (e.g. zero listeners after cert withholding empties the list), an API server admission error, or a permissions regression. Check controller logs for 'Reconciler error' entries on the 'gateway' controller."
78+
runbook_url: "https://github.com/datum-cloud/network-services-operator/blob/main/docs/runbooks/gateway-controller-health.md#gatewaycontrollerreconcileerrorratohigh"
79+
80+
# Fires at a critical threshold when more than 50% of reconciles are failing,
81+
# sustained for 10 minutes. At this rate the controller has essentially
82+
# stopped applying changes — treat as an active outage for gateway programming.
83+
- alert: GatewayControllerReconcileErrorRatioCritical
84+
expr: |
85+
(
86+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway",result="error"}[10m]))
87+
/
88+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway"}[10m]))
89+
) > 0.5
90+
for: 10m
91+
labels:
92+
severity: critical
93+
annotations:
94+
summary: "Gateway controller reconcile error ratio is {{ $value | humanizePercentage }} (threshold 50%) — edge programming is stalled"
95+
description: "More than 50% of gateway controller reconcile attempts are failing. The controller has effectively stopped programming downstream gateways — this is an active outage for any gateway changes (new listeners, cert updates, connector changes). Check controller logs for 'Reconciler error' entries. See GatewayControllerReconcileErrorRatioHigh for initial diagnosis."
96+
runbook_url: "https://github.com/datum-cloud/network-services-operator/blob/main/docs/runbooks/gateway-controller-health.md#gatewaycontrollerreconcileerrorratocritical"
97+
5398
# TLS certificate health alerts fire on nso_* metrics emitted directly by the
5499
# NSO operator and extension server. They are available in the same Prometheus
55100
# that loads this rule, alongside the envoy_gateway_* metrics above.

‎docs/runbooks/gateway-controller-health.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Runbook: Gateway controller reconcile-error alerts
2+
3+
These alerts fire when the gateway controller's reconcile error rate is
4+
persistently high. A reconcile error means the controller attempted to write or
5+
update a downstream Gateway object and was rejected by the API server. While
6+
errors persist, that gateway's configuration is frozen at its last successfully
7+
written state — listener changes, certificate updates, and connector changes do
8+
not reach the edge.
9+
10+
The most common trigger is PR #217's cert-withholding feature: when every
11+
listener on a gateway has an unusable TLS certificate, the controller withholds
12+
all of them, producing a downstream Gateway with zero listeners. The Gateway-API
13+
CRD rejects that as a Required value validation error, and the controller enters
14+
a hot-loop retrying the same rejected write.
15+
16+
Related: issue [#212](https://github.com/datum-cloud/network-services-operator/issues/212)
17+
and PR [#217](https://github.com/datum-cloud/network-services-operator/pull/217).
18+
19+
## Shared diagnosis
20+
21+
The controller exposes reconcile outcomes via the standard controller-runtime
22+
counter `controller_runtime_reconcile_total` with a `result` label
23+
(`error` / `success`).
24+
25+
Identify which gateways are failing by inspecting controller logs:
26+
27+
```sh
28+
kubectl -n <nso-ns> logs -l <controller-selector> | grep 'Reconciler error' | grep 'gateway'
29+
```
30+
31+
Each error log line names the namespace and name of the object that failed. The
32+
error message explains why the write was rejected.
33+
34+
Check the downstream Gateway object directly to confirm the current state:
35+
36+
```sh
37+
kubectl --context <downstream> -n <downstream-ns> get gateway <name> -o yaml
38+
```
39+
40+
## GatewayControllerReconcileErrorRatioHigh
41+
42+
**Meaning (warning).** More than 20% of gateway controller reconcile attempts
43+
are returning errors, sustained for 15 minutes. The controller is struggling to
44+
write downstream Gateway objects and some changes are not reaching the edge.
45+
46+
**Impact.** Affected gateways are not receiving configuration updates. New
47+
listeners, TLS cert rotation, and connector-status changes all depend on
48+
successful reconciles. The edge continues running whatever configuration was
49+
last successfully programmed.
50+
51+
**Diagnose.** Find the failing gateways in controller logs (see Shared
52+
diagnosis). The most common error messages are:
53+
54+
- `spec.listeners: Required value` — all listeners were withheld (cert
55+
withholding left none); the Gateway-API CRD requires at least one listener.
56+
If this is the cause, `GatewayListenerCertUnusable` should also be firing for
57+
the same gateway — check whether all listeners on that gateway have unusable
58+
certificates.
59+
- `Forbidden` / `Unauthorized` — permissions regression; check the controller's
60+
RBAC.
61+
- `conflict` / `ResourceVersion` — transient write conflicts; these resolve on
62+
their own and should not sustain a high error rate.
63+
64+
**Remediate.** Fix the root cause identified in the logs. For the
65+
all-listeners-withheld case, restore at least one usable TLS certificate for the
66+
gateway (or remove the broken Certificate references) — the listener returns
67+
automatically once the certificate is valid.
68+
69+
## GatewayControllerReconcileErrorRatioCritical
70+
71+
**Meaning (critical).** More than 50% of gateway controller reconcile attempts
72+
are failing, sustained for 10 minutes. The controller has effectively stopped
73+
programming downstream gateways.
74+
75+
**Impact.** Treat as an active outage for gateway configuration changes. No
76+
listener updates, TLS rotations, or connector state changes are being applied
77+
to any affected gateways. Customers may see stale routing, expired certificates
78+
left in place, or connectors showing incorrect availability — whatever was
79+
programmed before the errors began.
80+
81+
**Diagnose.** Follow the same steps as
82+
[GatewayControllerReconcileErrorRatioHigh](#gatewaycontrollerreconcileerrorratohigh).
83+
At this error rate the problem is systematic — check whether the issue affects a
84+
single gateway (one bad object) or many (a broader regression like an RBAC
85+
change or API server outage).
86+
87+
Check the API server error rate to rule out an upstream problem:
88+
89+
```sh
90+
kubectl get --raw /healthz
91+
```
92+
93+
**Remediate.** Fix the root cause as for the warning tier. If the error is
94+
a validation failure on one gateway, fixing that object's configuration will
95+
unblock the rest. If an API server or permissions change caused a broad failure,
96+
roll it back and verify the controller regains a healthy reconcile ratio before
97+
resolving the alert.

‎test/prometheus-rules/gateways/controller-health-rules.yaml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
groups:
2+
- name: nso-gateway-controller-health
3+
interval: 30s
4+
rules:
5+
# Fires when more than 20% of gateway controller reconcile attempts are
6+
# failing over a 10-minute window, sustained for 15 minutes. A common cause
7+
# is a downstream Gateway object failing CRD validation (e.g. all listeners
8+
# withheld leaves spec.listeners empty, which is rejected as Required value).
9+
# While this fires, controller changes are not reaching the edge.
10+
- alert: GatewayControllerReconcileErrorRatioHigh
11+
expr: |
12+
(
13+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway",result="error"}[10m]))
14+
/
15+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway"}[10m]))
16+
) > 0.2
17+
for: 15m
18+
labels:
19+
severity: warning
20+
annotations:
21+
summary: "Gateway controller reconcile error ratio is {{ $value | humanizePercentage }} (threshold 20%)"
22+
description: "More than 20% of gateway controller reconcile attempts are failing. The controller cannot write downstream Gateway objects — changes are not reaching the edge and the stale configuration stays programmed. Common causes: a downstream Gateway fails CRD validation (e.g. zero listeners after cert withholding empties the list), an API server admission error, or a permissions regression. Check controller logs for 'Reconciler error' entries on the 'gateway' controller."
23+
runbook_url: "https://github.com/datum-cloud/network-services-operator/blob/main/docs/runbooks/gateway-controller-health.md#gatewaycontrollerreconcileerrorratohigh"
24+
25+
# Fires at a critical threshold when more than 50% of reconciles are failing,
26+
# sustained for 10 minutes. At this rate the controller has essentially
27+
# stopped applying changes — treat as an active outage for gateway programming.
28+
- alert: GatewayControllerReconcileErrorRatioCritical
29+
expr: |
30+
(
31+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway",result="error"}[10m]))
32+
/
33+
sum without(result) (rate(controller_runtime_reconcile_total{controller="gateway"}[10m]))
34+
) > 0.5
35+
for: 10m
36+
labels:
37+
severity: critical
38+
annotations:
39+
summary: "Gateway controller reconcile error ratio is {{ $value | humanizePercentage }} (threshold 50%) — edge programming is stalled"
40+
description: "More than 50% of gateway controller reconcile attempts are failing. The controller has effectively stopped programming downstream gateways — this is an active outage for any gateway changes (new listeners, cert updates, connector changes). Check controller logs for 'Reconciler error' entries. See GatewayControllerReconcileErrorRatioHigh for initial diagnosis."
41+
runbook_url: "https://github.com/datum-cloud/network-services-operator/blob/main/docs/runbooks/gateway-controller-health.md#gatewaycontrollerreconcileerrorratocritical"

‎test/prometheus-rules/gateways/controller-health-tests.yaml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
rule_files:
2+
- controller-health-rules.yaml
3+
4+
evaluation_interval: 1m
5+
6+
tests:
7+
# GatewayControllerReconcileErrorRatioHigh — >20% error ratio sustained >15 minutes
8+
# Counter increments every minute: 9 errors + 1 success = 90% error rate.
9+
# The for: 15m means the alert fires at eval_time 16m (pending from 1m through 15m, firing at 16m).
10+
- interval: 1m
11+
input_series:
12+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="error"}'
13+
values: '0+9x20'
14+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="success"}'
15+
values: '0+1x20'
16+
alert_rule_test:
17+
- eval_time: 16m
18+
alertname: GatewayControllerReconcileErrorRatioHigh
19+
exp_alerts:
20+
- exp_labels:
21+
severity: warning
22+
controller: gateway
23+
exp_annotations:
24+
summary: "Gateway controller reconcile error ratio is 90% (threshold 20%)"
25+
description: "More than 20% of gateway controller reconcile attempts are failing. The controller cannot write downstream Gateway objects — changes are not reaching the edge and the stale configuration stays programmed. Common causes: a downstream Gateway fails CRD validation (e.g. zero listeners after cert withholding empties the list), an API server admission error, or a permissions regression. Check controller logs for 'Reconciler error' entries on the 'gateway' controller."
26+
runbook_url: "https://github.com/datum-cloud/network-services-operator/blob/main/docs/runbooks/gateway-controller-health.md#gatewaycontrollerreconcileerrorratohigh"
27+
28+
# GatewayControllerReconcileErrorRatioHigh — healthy ratio (10% errors), should NOT alert
29+
# 1 error + 9 success = 10% error rate, below the 20% threshold
30+
- interval: 1m
31+
input_series:
32+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="error"}'
33+
values: '0+1x20'
34+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="success"}'
35+
values: '0+9x20'
36+
alert_rule_test:
37+
- eval_time: 20m
38+
alertname: GatewayControllerReconcileErrorRatioHigh
39+
exp_alerts: []
40+
41+
# GatewayControllerReconcileErrorRatioHigh — ratio above threshold but under 15m duration, should NOT alert
42+
# 90% error rate but checked at 14m (for: 15m not yet satisfied)
43+
- interval: 1m
44+
input_series:
45+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="error"}'
46+
values: '0+9x20'
47+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="success"}'
48+
values: '0+1x20'
49+
alert_rule_test:
50+
- eval_time: 14m
51+
alertname: GatewayControllerReconcileErrorRatioHigh
52+
exp_alerts: []
53+
54+
# GatewayControllerReconcileErrorRatioCritical — >50% error ratio sustained >10 minutes
55+
# 8 errors + 2 success = 80% error rate. Fires at eval_time 11m.
56+
- interval: 1m
57+
input_series:
58+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="error"}'
59+
values: '0+8x20'
60+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="success"}'
61+
values: '0+2x20'
62+
alert_rule_test:
63+
- eval_time: 11m
64+
alertname: GatewayControllerReconcileErrorRatioCritical
65+
exp_alerts:
66+
- exp_labels:
67+
severity: critical
68+
controller: gateway
69+
exp_annotations:
70+
summary: "Gateway controller reconcile error ratio is 80% (threshold 50%) — edge programming is stalled"
71+
description: "More than 50% of gateway controller reconcile attempts are failing. The controller has effectively stopped programming downstream gateways — this is an active outage for any gateway changes (new listeners, cert updates, connector changes). Check controller logs for 'Reconciler error' entries. See GatewayControllerReconcileErrorRatioHigh for initial diagnosis."
72+
runbook_url: "https://github.com/datum-cloud/network-services-operator/blob/main/docs/runbooks/gateway-controller-health.md#gatewaycontrollerreconcileerrorratocritical"
73+
74+
# GatewayControllerReconcileErrorRatioCritical — 40% error rate (below 50%), should NOT alert
75+
# 4 errors + 6 success = 40% error rate
76+
- interval: 1m
77+
input_series:
78+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="error"}'
79+
values: '0+4x20'
80+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="success"}'
81+
values: '0+6x20'
82+
alert_rule_test:
83+
- eval_time: 20m
84+
alertname: GatewayControllerReconcileErrorRatioCritical
85+
exp_alerts: []
86+
87+
# GatewayControllerReconcileErrorRatioCritical — >50% ratio but under 10m duration, should NOT alert
88+
- interval: 1m
89+
input_series:
90+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="error"}'
91+
values: '0+8x20'
92+
- series: 'controller_runtime_reconcile_total{controller="gateway",result="success"}'
93+
values: '0+2x20'
94+
alert_rule_test:
95+
- eval_time: 9m
96+
alertname: GatewayControllerReconcileErrorRatioCritical
97+
exp_alerts: []

0 commit comments

Comments
 (0)