Skip to content

Commit f2d49dd

Browse files
scotwellsclaude
scotwells
and
claude
committed
feat(telemetry): add alerts for certificate gating and the data-plane backstop
Make the certificate gating actionable: alert when a customer listener is withheld because its certificate is unusable, when a managed certificate is expiring soon (so it can be fixed before it starts gating), when the extension server is actively dropping broken certificates, and — critically — when a listener cannot be protected at all and the edge will reject its update. Includes promtool unit tests for all four rules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsQQoiXVgC1VaiWye4eEz8
1 parent c77d947 commit f2d49dd

3 files changed

Lines changed: 286 additions & 0 deletions

File tree

‎config/telemetry/alerts/gateways.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,16 @@ spec:
2222
summary: "Gateway {{ $labels.resource_name }} is taking longer than 60 seconds to reach Ready status"
2323
description: "Gateway {{ $labels.resource_name }} in namespace {{ $labels.resource_namespace }} has been in creation state for {{ $value }} seconds without reaching Ready status (Accepted=True AND Programmed=True), which exceeds the 60-second SLO threshold."
2424

25+
- alert: EnvoyPatchPolicyProgrammingFailed
26+
expr: |
27+
envoy_gateway_envoypatchpolicy_status_condition{type="Programmed"} == 0
28+
for: 5m
29+
labels:
30+
severity: critical
31+
annotations:
32+
summary: "EnvoyPatchPolicy {{ $labels.name }} failed to program"
33+
description: "EnvoyPatchPolicy {{ $labels.name }} in namespace {{ $labels.namespace }} has been unable to apply its xDS patch for over 5 minutes (reason: {{ $labels.reason }}). Customer traffic on affected gateways may be impacted."
34+
2535
- alert: GatewayDegradedSLOViolation
2636
expr: |
2737
(
@@ -39,3 +49,71 @@ spec:
3949
annotations:
4050
summary: "Gateway {{ $labels.resource_name }} has been degraded for over 60 seconds"
4151
description: "Gateway {{ $labels.resource_name }} in namespace {{ $labels.resource_namespace }} has been in a degraded state for over 60 seconds without recovering, which exceeds the 60-second SLO threshold."
52+
53+
# TLS certificate health alerts fire on nso_* metrics emitted directly by the
54+
# NSO operator and extension server. They are available in the same Prometheus
55+
# that loads this rule, alongside the envoy_gateway_* metrics above.
56+
# These complement the infrastructure EnvoyListenerUpdateRejected alert (which
57+
# fires when Envoy rejects a bad LDS update). These alerts cover the earlier
58+
# prevention path: NSO withholds a listener or the extension server drops a
59+
# broken filter chain before Envoy has a chance to reject the update.
60+
- name: nso-tls-cert-health
61+
interval: 30s
62+
rules:
63+
# Fires when NSO has withheld a Gateway listener because its TLS certificate
64+
# is unusable. The customer's HTTPS hostname is dark until the cert recovers.
65+
# If EnvoyListenerUpdateRejected is also firing without this alert, NSO's
66+
# cert gating has regressed and a bad cert reached Envoy directly.
67+
- alert: GatewayListenerCertUnusable
68+
expr: |
69+
nso_gateway_listener_cert_withheld == 1
70+
for: 5m
71+
labels:
72+
severity: warning
73+
annotations:
74+
summary: "Gateway listener {{ $labels.namespace }}/{{ $labels.name }}/{{ $labels.listener }} has an unusable TLS certificate"
75+
description: "NSO has withheld listener {{ $labels.listener }} (hostname {{ $labels.hostname }}) on Gateway {{ $labels.name }} in namespace {{ $labels.namespace }} because its TLS certificate is unusable (reason: {{ $labels.reason }}). The customer cannot serve HTTPS on this hostname. Check the cert-manager Certificate and Secret in the downstream cluster."
76+
77+
# Fires when a managed TLS certificate is within 7 days of expiry while it
78+
# is still healthy. cert-manager renews automatically, but renewal fails if
79+
# the domain's DNS no longer points to Datum. Acting here avoids a future
80+
# GatewayListenerCertUnusable alert.
81+
- alert: GatewayListenerCertExpiringSoon
82+
expr: |
83+
(nso_gateway_listener_cert_expiry_time - time()) / 86400 < 7
84+
for: 1h
85+
labels:
86+
severity: warning
87+
annotations:
88+
summary: "TLS certificate for Gateway listener {{ $labels.namespace }}/{{ $labels.name }}/{{ $labels.listener }} expires in less than 7 days"
89+
description: "The cert-manager Certificate for listener {{ $labels.listener }} (hostname {{ $labels.hostname }}, secret {{ $labels.secret }}) on Gateway {{ $labels.name }} in namespace {{ $labels.namespace }} expires within 7 days. cert-manager should renew it automatically, but renewal fails if the domain's DNS no longer points to Datum. Verify the Certificate is Ready=True in the downstream cluster."
90+
91+
# Fires when the extension server is actively dropping broken certificates
92+
# from the configuration it sends to the edge. This is expected briefly
93+
# between a certificate failing and the controller withholding the listener.
94+
# If only this fires and GatewayListenerCertUnusable does not, the controller
95+
# may have missed the listener.
96+
- alert: TLSBackstopPruningChains
97+
expr: |
98+
nso_extension_tls_pruned_chains_active > 0
99+
for: 2m
100+
labels:
101+
severity: warning
102+
annotations:
103+
summary: "Extension server is dropping {{ $value }} broken certificate(s) to protect the edge listener"
104+
description: "The extension server is dropping {{ $value }} broken certificate(s) from the configuration it sends to the edge gateway. Check extension server logs for 'pruned invalid TLS chains' to find the affected hostnames. If GatewayListenerCertUnusable is also firing, both layers of protection are working as expected. If only this alert fires, the controller may have missed the listener."
105+
106+
# Fires when the extension server could not protect a listener because every
107+
# certificate on it is broken. It never removes a listener entirely, so the
108+
# edge will reject the update for that listener — EnvoyListenerUpdateRejected
109+
# (infra) confirms it. It means the controller did not withhold the listener
110+
# before it reached the edge.
111+
- alert: TLSBackstopListenerAllCertsBroken
112+
expr: |
113+
nso_extension_tls_listeners_left_intact_active > 0
114+
for: 2m
115+
labels:
116+
severity: critical
117+
annotations:
118+
summary: "{{ $value }} edge listener(s) have every TLS certificate broken and cannot be protected"
119+
description: "The extension server left {{ $value }} edge listener(s) untouched because every certificate on them is broken. It never removes a listener entirely, so the edge will reject the configuration update for those listeners (EnvoyListenerUpdateRejected confirms it). This means the controller did not withhold the listener before it reached the edge. Check extension server logs for 'listeners_left_intact' and why the controller did not withhold the listener."

‎test/prometheus-rules/gateways/cert-health-rules.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
groups:
2+
- name: nso-tls-cert-health
3+
interval: 30s
4+
rules:
5+
# Fires when NSO has withheld a Gateway listener because its TLS certificate
6+
# is unusable. The customer's HTTPS hostname is dark until the cert recovers.
7+
# If EnvoyListenerUpdateRejected is also firing without this alert, NSO's
8+
# cert gating has regressed and a bad cert reached Envoy directly.
9+
- alert: GatewayListenerCertUnusable
10+
expr: |
11+
nso_gateway_listener_cert_withheld == 1
12+
for: 5m
13+
labels:
14+
severity: warning
15+
annotations:
16+
summary: "Gateway listener {{ $labels.namespace }}/{{ $labels.name }}/{{ $labels.listener }} has an unusable TLS certificate"
17+
description: "NSO has withheld listener {{ $labels.listener }} (hostname {{ $labels.hostname }}) on Gateway {{ $labels.name }} in namespace {{ $labels.namespace }} because its TLS certificate is unusable (reason: {{ $labels.reason }}). The customer cannot serve HTTPS on this hostname. Check the cert-manager Certificate and Secret in the downstream cluster."
18+
19+
# Fires when a managed TLS certificate is within 7 days of expiry while it
20+
# is still healthy. cert-manager renews automatically, but renewal fails if
21+
# the domain's DNS no longer points to Datum. Acting here avoids a future
22+
# GatewayListenerCertUnusable alert.
23+
- alert: GatewayListenerCertExpiringSoon
24+
expr: |
25+
(nso_gateway_listener_cert_expiry_time - time()) / 86400 < 7
26+
for: 1h
27+
labels:
28+
severity: warning
29+
annotations:
30+
summary: "TLS certificate for Gateway listener {{ $labels.namespace }}/{{ $labels.name }}/{{ $labels.listener }} expires in less than 7 days"
31+
description: "The cert-manager Certificate for listener {{ $labels.listener }} (hostname {{ $labels.hostname }}, secret {{ $labels.secret }}) on Gateway {{ $labels.name }} in namespace {{ $labels.namespace }} expires within 7 days. cert-manager should renew it automatically, but renewal fails if the domain's DNS no longer points to Datum. Verify the Certificate is Ready=True in the downstream cluster."
32+
33+
# Fires when the extension server's data-plane backstop is actively dropping
34+
# broken TLS filter chains from PostTranslateModify responses. This is normal
35+
# during the window between cert failure and controller-side gating catching
36+
# up. If only this alert fires and GatewayListenerCertUnusable does not, the
37+
# controller may have missed the bad listener — investigate.
38+
- alert: TLSBackstopPruningChains
39+
expr: |
40+
nso_extension_tls_pruned_chains_active > 0
41+
for: 2m
42+
labels:
43+
severity: warning
44+
annotations:
45+
summary: "Extension server TLS backstop is actively pruning {{ $value }} broken filter chain(s)"
46+
description: "The extension server's TLS backstop is dropping {{ $value }} broken TLS filter chain(s) from PostTranslateModify responses. Check extension server logs for 'pruned invalid TLS chains' to identify affected hostnames. If GatewayListenerCertUnusable is also firing, the two-layer defense is working as expected. If only this alert fires, the controller-side cert gating may have missed the listener."
47+
48+
# Fires when the extension server's backstop could not save a listener because
49+
# every TLS filter chain on it is broken. The backstop never empties a listener,
50+
# so Envoy will NACK the full LDS update — see EnvoyListenerUpdateRejected in
51+
# the infra alerts for the confirming signal. This means NSO's controller-side
52+
# cert gating did not suppress the listener before it reached the data plane.
53+
- alert: TLSBackstopListenerAllCertsBroken
54+
expr: |
55+
nso_extension_tls_listeners_left_intact_active > 0
56+
for: 2m
57+
labels:
58+
severity: critical
59+
annotations:
60+
summary: "{{ $value }} Envoy listener(s) have all TLS certificates broken — backstop cannot protect them"
61+
description: "The extension server left {{ $value }} Envoy listener(s) completely intact because every TLS filter chain on them referenced a broken certificate. The backstop never empties a listener, so Envoy will reject the LDS update for those listeners (see EnvoyListenerUpdateRejected). This means NSO's controller-side cert gating did not suppress the listener before it reached the data plane. Check extension server logs for 'listeners_left_intact' and investigate why evaluateListenerCertHealth did not catch the listener."

‎test/prometheus-rules/gateways/cert-health-tests.yaml‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
rule_files:
2+
- cert-health-rules.yaml
3+
4+
evaluation_interval: 30s
5+
6+
tests:
7+
# GatewayListenerCertUnusable — listener withheld for more than 5 minutes
8+
- interval: 1m
9+
input_series:
10+
- series: 'nso_gateway_listener_cert_withheld{namespace="customer-ns", name="my-gateway", listener="https-custom", hostname="example.com", reason="InvalidCertificateRef"}'
11+
values: '1+0x5'
12+
alert_rule_test:
13+
- eval_time: 6m
14+
alertname: GatewayListenerCertUnusable
15+
exp_alerts:
16+
- exp_labels:
17+
severity: warning
18+
namespace: customer-ns
19+
name: my-gateway
20+
listener: https-custom
21+
hostname: example.com
22+
reason: InvalidCertificateRef
23+
exp_annotations:
24+
summary: "Gateway listener customer-ns/my-gateway/https-custom has an unusable TLS certificate"
25+
description: "NSO has withheld listener https-custom (hostname example.com) on Gateway my-gateway in namespace customer-ns because its TLS certificate is unusable (reason: InvalidCertificateRef). The customer cannot serve HTTPS on this hostname. Check the cert-manager Certificate and Secret in the downstream cluster."
26+
27+
# GatewayListenerCertUnusable — listener withheld for less than 5 minutes (should NOT alert)
28+
- interval: 1m
29+
input_series:
30+
- series: 'nso_gateway_listener_cert_withheld{namespace="customer-ns", name="my-gateway", listener="https-custom", hostname="example.com", reason="InvalidCertificateRef"}'
31+
values: '1+0x2'
32+
alert_rule_test:
33+
- eval_time: 3m
34+
alertname: GatewayListenerCertUnusable
35+
exp_alerts: []
36+
37+
# GatewayListenerCertUnusable — listener is healthy (withheld=0, should NOT alert)
38+
- interval: 1m
39+
input_series:
40+
- series: 'nso_gateway_listener_cert_withheld{namespace="customer-ns", name="healthy-gateway", listener="https-custom", hostname="ok.com", reason="InvalidCertificateRef"}'
41+
values: '0+0x5'
42+
alert_rule_test:
43+
- eval_time: 6m
44+
alertname: GatewayListenerCertUnusable
45+
exp_alerts: []
46+
47+
# GatewayListenerCertExpiringSoon — cert expires in 5 days (within 7-day threshold)
48+
# At eval_time 62m (3720s), expiry_time = 3720 + 5*86400 = 435720
49+
- interval: 1m
50+
input_series:
51+
- series: 'nso_gateway_listener_cert_expiry_time{namespace="customer-ns", name="my-gateway", listener="https-custom", hostname="example.com", secret="my-gateway-https-custom"}'
52+
values: '435720+0x62'
53+
alert_rule_test:
54+
- eval_time: 62m
55+
alertname: GatewayListenerCertExpiringSoon
56+
exp_alerts:
57+
- exp_labels:
58+
severity: warning
59+
namespace: customer-ns
60+
name: my-gateway
61+
listener: https-custom
62+
hostname: example.com
63+
secret: my-gateway-https-custom
64+
exp_annotations:
65+
summary: "TLS certificate for Gateway listener customer-ns/my-gateway/https-custom expires in less than 7 days"
66+
description: "The cert-manager Certificate for listener https-custom (hostname example.com, secret my-gateway-https-custom) on Gateway my-gateway in namespace customer-ns expires within 7 days. cert-manager should renew it automatically, but renewal fails if the domain's DNS no longer points to Datum. Verify the Certificate is Ready=True in the downstream cluster."
67+
68+
# GatewayListenerCertExpiringSoon — cert expires in 8 days (outside threshold, should NOT alert)
69+
# At eval_time 62m (3720s), expiry_time = 3720 + 8*86400 = 694920
70+
- interval: 1m
71+
input_series:
72+
- series: 'nso_gateway_listener_cert_expiry_time{namespace="customer-ns", name="my-gateway", listener="https-custom", hostname="example.com", secret="my-gateway-https-custom"}'
73+
values: '694920+0x62'
74+
alert_rule_test:
75+
- eval_time: 62m
76+
alertname: GatewayListenerCertExpiringSoon
77+
exp_alerts: []
78+
79+
# TLSBackstopPruningChains — backstop actively dropping chains for more than 2 minutes
80+
- interval: 1m
81+
input_series:
82+
- series: 'nso_extension_tls_pruned_chains_active'
83+
values: '3+0x2'
84+
alert_rule_test:
85+
- eval_time: 3m
86+
alertname: TLSBackstopPruningChains
87+
exp_alerts:
88+
- exp_labels:
89+
severity: warning
90+
exp_annotations:
91+
summary: "Extension server TLS backstop is actively pruning 3 broken filter chain(s)"
92+
description: "The extension server's TLS backstop is dropping 3 broken TLS filter chain(s) from PostTranslateModify responses. Check extension server logs for 'pruned invalid TLS chains' to identify affected hostnames. If GatewayListenerCertUnusable is also firing, the two-layer defense is working as expected. If only this alert fires, the controller-side cert gating may have missed the listener."
93+
94+
# TLSBackstopPruningChains — backstop just started, under 2 minutes (should NOT alert)
95+
- interval: 1m
96+
input_series:
97+
- series: 'nso_extension_tls_pruned_chains_active'
98+
values: '2+0x0'
99+
alert_rule_test:
100+
- eval_time: 1m
101+
alertname: TLSBackstopPruningChains
102+
exp_alerts: []
103+
104+
# TLSBackstopPruningChains — no chains pruned (should NOT alert)
105+
- interval: 1m
106+
input_series:
107+
- series: 'nso_extension_tls_pruned_chains_active'
108+
values: '0+0x2'
109+
alert_rule_test:
110+
- eval_time: 3m
111+
alertname: TLSBackstopPruningChains
112+
exp_alerts: []
113+
114+
# TLSBackstopListenerAllCertsBroken — all certs broken for more than 2 minutes (critical)
115+
- interval: 1m
116+
input_series:
117+
- series: 'nso_extension_tls_listeners_left_intact_active'
118+
values: '1+0x2'
119+
alert_rule_test:
120+
- eval_time: 3m
121+
alertname: TLSBackstopListenerAllCertsBroken
122+
exp_alerts:
123+
- exp_labels:
124+
severity: critical
125+
exp_annotations:
126+
summary: "1 Envoy listener(s) have all TLS certificates broken — backstop cannot protect them"
127+
description: "The extension server left 1 Envoy listener(s) completely intact because every TLS filter chain on them referenced a broken certificate. The backstop never empties a listener, so Envoy will reject the LDS update for those listeners (see EnvoyListenerUpdateRejected). This means NSO's controller-side cert gating did not suppress the listener before it reached the data plane. Check extension server logs for 'listeners_left_intact' and investigate why evaluateListenerCertHealth did not catch the listener."
128+
129+
# TLSBackstopListenerAllCertsBroken — condition just started, under 2 minutes (should NOT alert)
130+
- interval: 1m
131+
input_series:
132+
- series: 'nso_extension_tls_listeners_left_intact_active'
133+
values: '2+0x0'
134+
alert_rule_test:
135+
- eval_time: 1m
136+
alertname: TLSBackstopListenerAllCertsBroken
137+
exp_alerts: []
138+
139+
# TLSBackstopListenerAllCertsBroken — no intact listeners (should NOT alert)
140+
- interval: 1m
141+
input_series:
142+
- series: 'nso_extension_tls_listeners_left_intact_active'
143+
values: '0+0x2'
144+
alert_rule_test:
145+
- eval_time: 3m
146+
alertname: TLSBackstopListenerAllCertsBroken
147+
exp_alerts: []

0 commit comments

Comments
 (0)