Skip to content

Commit 54cee8f

Browse files
mastermanas805claude
mastermanas805
and
claude
committed
infra: NR alert + Prom rules + YAML diff doc for /readyz (NOT auto-applied)
Land the alerting side of the deep readiness rollout: 1. newrelic/alerts/readyz-component-failed.json — NR NRQL alert that pages when any /readyz check on any service stays at status=failed (gauge value=0) for >5 min, with a WARNING tier at degraded (0.5) for >10 min. This is the alert that would have caught the Brevo silent-rejection bug from 2026-05-20 weeks earlier. 2. prometheus/alert-rules.yml — ReadyzCheckFailed (critical, 5m at 0) + ReadyzCheckDegraded (warning, 10m at 0.5) rules for clusters where the Prometheus → NR adapter isn't installed. 3. READYZ-WIRING-2026-05-20.md — the k8s manifest diff for app.yaml + worker/deployment.yaml + provisioner/deployment.yaml that flips readinessProbe from /healthz to /readyz, with periodSeconds bumped from 10s→15s (matches the per-check 10s cache TTL) and timeoutSeconds=5 made explicit. NOT applied — operator runs kubectl apply manually after curl-verifying /readyz on one port-forwarded pod per service. /healthz stays the livenessProbe target (untouched) — a Brevo outage MUST NOT SIGKILL every pod. Rollout order in the doc: api → worker → provisioner, verify-live each step per CLAUDE.md rule 23. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 03d6cee commit 54cee8f

3 files changed

Lines changed: 278 additions & 0 deletions

File tree

‎READYZ-WIRING-2026-05-20.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# /readyz wiring — k8s manifest diff (NOT auto-applied)
2+
3+
## Context
4+
5+
Today's `/healthz` is a shallow liveness probe — DB ping + commit_id only. The
6+
Brevo silent-rejection bug from 2026-05-20 (and analogous upstream blips) is
7+
invisible to it. Three repo PRs land `/readyz` with component-by-component
8+
checks (platform_db, customer_db, provisioner_grpc, brevo, razorpay, redis,
9+
do_spaces, river) per service.
10+
11+
This file is the k8s manifest diff. **Do not auto-apply.** Apply manually
12+
after the new images deploy + the existing `/healthz` is verified untouched.
13+
14+
## Why manual
15+
16+
`/healthz` stays wired to `livenessProbe`. The risk is that a misconfigured
17+
`/readyz` returns 503 every probe → kubelet pulls every pod → outage. Verify
18+
on one pod first (`kubectl port-forward` + `curl /readyz`) before flipping
19+
the readinessProbe.
20+
21+
## Diff 1 — `infra/k8s/app.yaml` (api)
22+
23+
```diff
24+
readinessProbe:
25+
httpGet:
26+
- path: /healthz
27+
+ path: /readyz
28+
port: 8080
29+
- initialDelaySeconds: 5
30+
- periodSeconds: 10
31+
- failureThreshold: 3
32+
+ initialDelaySeconds: 10
33+
+ periodSeconds: 15
34+
+ timeoutSeconds: 5
35+
+ failureThreshold: 3
36+
livenessProbe:
37+
httpGet:
38+
path: /livez # process-only check; see startupProbe note
39+
port: 8080
40+
initialDelaySeconds: 15
41+
periodSeconds: 30
42+
failureThreshold: 6
43+
```
44+
45+
**What changes:** readinessProbe now points at `/readyz` (deep check),
46+
`livenessProbe` stays at `/livez` (shallow). `periodSeconds` bumps from 10s
47+
to 15s so the per-check cache (10s TTL) saves an upstream call per period.
48+
`timeoutSeconds: 5` is added explicitly so a slow Brevo probe can't stall
49+
the probe (the runner's internal `OverallTimeout: 3s` is well within this).
50+
51+
**Why livenessProbe stays at `/livez`:** a Brevo outage must NOT restart the
52+
api pod. `/livez` is the pure process-up signal — wired correctly today.
53+
54+
## Diff 2 — `infra/k8s/worker/deployment.yaml`
55+
56+
```diff
57+
readinessProbe:
58+
httpGet:
59+
- path: /healthz
60+
+ path: /readyz
61+
port: 8091
62+
- initialDelaySeconds: 10
63+
- periodSeconds: 10
64+
- failureThreshold: 3
65+
+ initialDelaySeconds: 10
66+
+ periodSeconds: 15
67+
+ timeoutSeconds: 5
68+
+ failureThreshold: 3
69+
livenessProbe:
70+
httpGet:
71+
path: /healthz # worker serves only /healthz (no /livez)
72+
port: 8091
73+
initialDelaySeconds: 15
74+
periodSeconds: 30
75+
failureThreshold: 6
76+
```
77+
78+
**Note:** the worker has no Service endpoint to be pulled from, so a 503
79+
on `/readyz` is purely an observability signal today (readiness fires the
80+
NR alert and updates the Prometheus gauge; nothing routes around the
81+
pod). If a future PR adds a worker Service for in-cluster lookups, the
82+
readiness gate is already wired.
83+
84+
## Diff 3 — `infra/k8s/provisioner/deployment.yaml`
85+
86+
```diff
87+
readinessProbe:
88+
httpGet:
89+
- path: /healthz
90+
+ path: /readyz
91+
port: 8092
92+
- initialDelaySeconds: 5
93+
- periodSeconds: 10
94+
- failureThreshold: 3
95+
+ initialDelaySeconds: 10
96+
+ periodSeconds: 15
97+
+ timeoutSeconds: 5
98+
+ failureThreshold: 3
99+
livenessProbe:
100+
httpGet:
101+
path: /healthz # provisioner serves only /healthz (no /livez)
102+
port: 8092
103+
initialDelaySeconds: 15
104+
periodSeconds: 30
105+
failureThreshold: 6
106+
```
107+
108+
**Provisioner extra:** the provisioner's gRPC server now registers the
109+
standard `grpc.health.v1.Health` service so the api's `/readyz` check
110+
`provisioner_grpc` works (it calls `Check(service="")`). No manifest
111+
change is required for that — it's a code change inside the provisioner
112+
binary.
113+
114+
## Rollout order (per CLAUDE.md rule 23 — verify-live each step)
115+
116+
1. **api** (largest blast radius). Build + push + rollout. Verify on
117+
one pod via `kubectl port-forward -n instant svc/instant-api 8080:8080`
118+
then `curl http://localhost:8080/readyz | jq .` Expect 200 + overall=ok
119+
(or degraded if Brevo/Razorpay configuration is in fact broken).
120+
Then apply Diff 1 and watch `kubectl get pods -n instant -w` for any
121+
Ready=False flapping.
122+
123+
2. **worker.** Same pattern with port-forward to `svc/instant-worker`
124+
(which doesn't exist as a Service today — `kubectl port-forward
125+
pod/<worker-pod> 8091:8091` instead). Apply Diff 2.
126+
127+
3. **provisioner.** `kubectl port-forward -n instant-infra
128+
svc/instant-provisioner 8092:8092` then `curl
129+
http://localhost:8092/readyz | jq .` Apply Diff 3.
130+
131+
## Verification commands
132+
133+
```bash
134+
# 1) api — expect ok + 200, or degraded + 200 if Brevo etc. are configured wrong
135+
curl https://api.instanode.dev/readyz | jq .
136+
137+
# 2) worker — in-cluster only
138+
kubectl port-forward -n instant-infra pod/<worker-pod> 8091 &
139+
curl http://localhost:8091/readyz | jq .
140+
141+
# 3) provisioner — in-cluster only
142+
kubectl port-forward -n instant-infra svc/instant-provisioner 8092 &
143+
curl http://localhost:8092/readyz | jq .
144+
145+
# 4) verify /healthz untouched on api (must still be the shallow shape)
146+
curl https://api.instanode.dev/healthz | jq .
147+
# expected: ok, service, commit_id, build_time, version, migration_*
148+
```
149+
150+
## Rollback
151+
152+
```bash
153+
# Revert one service:
154+
kubectl set image -n instant deploy/instant-api app=ghcr.io/instanode-dev/api:<prev-sha>
155+
156+
# Or revert manifest:
157+
git revert <commit>
158+
kubectl apply -f infra/k8s/app.yaml
159+
```
160+
161+
A pod that flips Ready=False on `/readyz` but Ready=True on the OLD
162+
`/healthz` is benign — kubelet just stops sending it traffic. Revert
163+
the readinessProbe path back to `/healthz` to restore the pre-change
164+
behavior without re-deploying the binary.
165+
166+
## Why readiness, not liveness
167+
168+
`/readyz` checks upstream HTTP APIs (Brevo, Razorpay, DO Spaces). If any of
169+
those is flapping, a livenessProbe on `/readyz` would SIGKILL every api pod
170+
and a Brevo outage becomes an api outage. The readinessProbe just pulls the
171+
pod from the Service endpoint list — no restart, just no new traffic.
172+
173+
The criticality matrix per service (which checks return 503 vs which
174+
return 200+degraded) is hard-coded in handlers/readyz.go for each repo —
175+
NOT env-tunable, because a misconfigured matrix is worse than no /readyz.
176+
177+
## Field names + wire shape
178+
179+
```json
180+
{
181+
"overall": "ok" | "degraded" | "failed",
182+
"service": "instant-api",
183+
"commit_id": "<sha>",
184+
"checks": [
185+
{
186+
"name": "platform_db",
187+
"status": "ok" | "degraded" | "failed",
188+
"latency_ms": 3,
189+
"last_check_at": "2026-05-20T08:01:23Z",
190+
"last_error": ""
191+
},
192+
...
193+
]
194+
}
195+
```
196+
197+
Sorted by check name alphabetically so jq filters are stable across
198+
probes.

‎newrelic/alerts/readyz-component-failed.json‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{
2+
"name": "instant-* — /readyz component failed > 5 min (Brevo / Razorpay / DO Spaces / DB / River)",
3+
"type": "NRQL",
4+
"description": "Page when ANY /readyz component on ANY service stays at status=failed for >5 minutes. The metric readyz_check_status is set per-probe by api / worker / provisioner /readyz handlers (1=ok, 0.5=degraded, 0=failed); a sustained 0 means the upstream is broken in a way the operator must investigate. This is the exact alert that would have caught the Brevo silent-rejection bug from 2026-05-20 weeks earlier — Brevo's /v3/account would have returned 401 on every probe, the brevo check would have flipped to 0.5 (degraded) immediately, and a sustained-degraded alert below would have paged the on-call. CRITICAL on 5 minutes at 0; WARNING on 5 minutes at 0.5 (degraded — upstream reachable but creds/policy off). Labels: service=instant-{api,worker,provisioner}, check={platform_db,customer_db,provisioner_grpc,brevo,razorpay,redis,do_spaces,river,backend_*}. Source: Prometheus gauge readyz_check_status forwarded to NR via OTLP or the Prometheus → NR adapter — same path as the existing instant_circuit_breaker_state gauge. Runbook: open the failing pod's /readyz body for last_error, cross-check against the upstream status page, rotate keys if 401, file an incident if 5xx.",
5+
"enabled": true,
6+
"nrql": {
7+
"query": "SELECT latest(readyz_check_status) FROM Metric WHERE metricName = 'readyz_check_status' FACET service, check"
8+
},
9+
"terms": [
10+
{
11+
"priority": "CRITICAL",
12+
"operator": "EQUALS",
13+
"threshold": 0,
14+
"thresholdDuration": 300,
15+
"thresholdOccurrences": "ALL"
16+
},
17+
{
18+
"priority": "WARNING",
19+
"operator": "EQUALS",
20+
"threshold": 0.5,
21+
"thresholdDuration": 600,
22+
"thresholdOccurrences": "ALL"
23+
}
24+
],
25+
"signal": {
26+
"aggregationWindow": 60,
27+
"aggregationMethod": "EVENT_FLOW",
28+
"aggregationDelay": 60,
29+
"fillOption": "LAST_VALUE"
30+
},
31+
"expiration": {
32+
"expirationDuration": 600,
33+
"openViolationOnExpiration": false,
34+
"closeViolationsOnExpiration": true
35+
},
36+
"violationTimeLimitSeconds": 86400
37+
}

‎prometheus/alert-rules.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,3 +150,46 @@ groups:
150150
New /storage/new provisions cannot complete — a customer's first-byte
151151
flow is blocked. Check the object-store backend (DO Spaces) credentials
152152
and connectivity, and api storage.go logs.
153+
154+
# ── /readyz component health ───────────────────────────────────────────────
155+
#
156+
# Fire when ANY /readyz component on ANY service stays at status=failed
157+
# (gauge value = 0) for >5 minutes. This is the alert that would have
158+
# caught the Brevo silent-rejection bug from 2026-05-20 weeks earlier:
159+
# Brevo's /v3/account would have returned 401 → brevo check would have
160+
# flipped to 0.5 (degraded) on every probe → the WARNING below would
161+
# have paged the on-call within 5 minutes.
162+
#
163+
# Critical components (platform_db, provisioner_grpc, river) pull a pod
164+
# out of rotation when failed, so a sustained 0 here means EVERY pod in
165+
# the deployment is in trouble — fire CRITICAL.
166+
#
167+
# Non-critical components (brevo, razorpay, do_spaces) keep the pod in
168+
# rotation but mean an upstream is silently rejecting traffic. Fire
169+
# WARNING — operator investigation needed but not a wake-up.
170+
- alert: ReadyzCheckFailed
171+
expr: max(readyz_check_status) by (service, check) == 0
172+
for: 5m
173+
labels:
174+
severity: critical
175+
annotations:
176+
summary: "/readyz check {{ $labels.check }} on {{ $labels.service }} failed for 5m"
177+
description: >
178+
/readyz reports {{ $labels.check }} failed on {{ $labels.service }}
179+
for 5 consecutive minutes. Curl the pod's /readyz body for last_error,
180+
cross-check the upstream status page, rotate keys if 401, file an
181+
incident if 5xx. Runbook: infra/READYZ-WIRING-2026-05-20.md.
182+
183+
- alert: ReadyzCheckDegraded
184+
expr: max(readyz_check_status) by (service, check) == 0.5
185+
for: 10m
186+
labels:
187+
severity: warning
188+
annotations:
189+
summary: "/readyz check {{ $labels.check }} on {{ $labels.service }} degraded for 10m"
190+
description: >
191+
/readyz reports {{ $labels.check }} degraded on {{ $labels.service }}
192+
(upstream reachable but credentials/policy off — e.g. expired API
193+
key, 401 from Brevo/Razorpay) for 10 consecutive minutes. Pod stays
194+
in rotation but the upstream call path is broken. Curl the pod's
195+
/readyz body for last_error and rotate the relevant secret.

0 commit comments

Comments
 (0)