Skip to content

Commit b95ee3f

Browse files
mastermanas805claude
mastermanas805
and
claude
committed
infra(brevo): BREVO_WEBHOOK_SECRET + NR alert/dashboard for delivery ratio
Wires the receiver-side machinery for the api's POST /webhooks/brevo/:secret endpoint that closes the '201 ≠ delivered' gap (api PR 3341f55). * k8s/secrets.yaml — template entry for BREVO_WEBHOOK_SECRET with full rationale + operator step (paste secret into Brevo dashboard's webhook URL). * k8s/app.yaml — wire the secret into the api Deployment as BREVO_WEBHOOK_SECRET env var (optional: true so a fresh cluster boots even before the operator rotates the value). * newrelic/alerts/email-delivery-ratio-low.json — CRITICAL alert when brevo_webhook_events_total{event='delivered'} / total < 95% for 1h. This is the alert that would have caught the 2026-05-20 incident weeks earlier. * newrelic/dashboards/email-delivery.json — live delivery ratio + per-event-class breakdown + send-vs-deliver log overlay. OPERATOR ACTIONS (BLOCKING for production): 1. openssl rand -hex 32 → set BREVO_WEBHOOK_SECRET in instant-secrets: kubectl patch secret instant-secrets -n instant --type merge \ -p "{\"data\":{\"BREVO_WEBHOOK_SECRET\":\"$(printf '%s' $SECRET | base64)\"}}" 2. kubectl rollout restart deploy/instant-api -n instant 3. In Brevo dashboard → Transactional → Settings → Webhook URL: paste https://api.instanode.dev/webhooks/brevo/<SECRET> 4. Enable every event class (delivered, soft_bounce, hard_bounce, blocked, complaint, deferred, unsubscribed, error). Ensure 'single-event-per-webhook-call' is selected, NOT batched. 5. Apply the NR alert + dashboard via newrelic/apply.sh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 54cee8f commit b95ee3f

4 files changed

Lines changed: 164 additions & 0 deletions

File tree

‎k8s/app.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,17 @@ spec:
9595
secretKeyRef:
9696
name: instant-secrets
9797
key: RAZORPAY_WEBHOOK_SECRET
98+
# BREVO_WEBHOOK_SECRET — URL-token for the transactional-
99+
# delivery receiver at POST /webhooks/brevo/:secret +
100+
# HMAC key for the legacy /api/v1/email/webhook/brevo path.
101+
# See api/internal/handlers/brevo_webhook.go for the full
102+
# rationale ("201 ≠ delivered" gap closure).
103+
- name: BREVO_WEBHOOK_SECRET
104+
valueFrom:
105+
secretKeyRef:
106+
name: instant-secrets
107+
key: BREVO_WEBHOOK_SECRET
108+
optional: true
98109
- name: RAZORPAY_PLAN_ID_HOBBY
99110
valueFrom:
100111
secretKeyRef:

‎k8s/secrets.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,21 @@ stringData:
2929
RAZORPAY_PLAN_ID_PRO: "CHANGE_ME"
3030
RAZORPAY_PLAN_ID_TEAM: "CHANGE_ME"
3131
RESEND_API_KEY: "CHANGE_ME"
32+
# Brevo (transactional email) — shared secret consumed by TWO endpoints:
33+
# 1. The HMAC-signed bounce/unsubscribe path at
34+
# /api/v1/email/webhook/brevo (HMAC-SHA256 over raw body, header
35+
# X-Sib-Signature / X-Mailin-Custom). Used by Brevo's optional
36+
# webhook-signing toggle.
37+
# 2. The transactional-delivery receiver at /webhooks/brevo/:secret
38+
# (URL-token compare in api/internal/handlers/brevo_webhook.go).
39+
# Closes the "201 ≠ delivered" gap — Brevo POSTs delivered /
40+
# hard_bounce / etc. events here and the api updates the
41+
# forwarder_sent ledger row keyed by provider_id.
42+
# Generate with: openssl rand -hex 32 (must be 32+ chars).
43+
# Operator step after rotate: paste the new value into Brevo dashboard
44+
# → Transactional → Settings → Webhook URL as the final path segment,
45+
# e.g. https://api.instanode.dev/webhooks/brevo/<SECRET>.
46+
BREVO_WEBHOOK_SECRET: "CHANGE_ME"
3247
GITHUB_CLIENT_SECRET: "CHANGE_ME"
3348
GOOGLE_CLIENT_SECRET: "CHANGE_ME"
3449
R2_API_TOKEN: "CHANGE_ME"

‎newrelic/alerts/email-delivery-ratio-low.json‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{
2+
"name": "Email delivery ratio < 95% (1h)",
3+
"type": "NRQL",
4+
"description": "The receiver-side counter brevo_webhook_events_total{event=\"delivered\"} / brevo_webhook_events_total over a 1h window. Closes the '201 ≠ delivered' gap — Brevo's transactional API returns 201 the instant it accepts the POST, but the actual SMTP relay happens async. This alert fires when the proportion of POSTs that Brevo's webhook later confirmed as 'delivered' drops below 95%. Common causes: sender domain dropped from validated list (the 2026-05-20 launch incident), DKIM record stale, IP on a blocklist, or a broad bounce/rejected event class spiking. The receiver path is api/internal/handlers/brevo_webhook.go; the metric is registered in api/internal/metrics/metrics.go. Cardinality is bounded (closed set of event labels).",
5+
"enabled": true,
6+
"nrql": {
7+
"query": "SELECT percentage(sum(brevo_webhook_events_total), WHERE event = 'delivered') FROM Metric WHERE event IN ('delivered', 'bounced_hard', 'bounced_soft', 'rejected', 'complaint', 'deferred', 'unsubscribed', 'error')"
8+
},
9+
"terms": [
10+
{
11+
"priority": "CRITICAL",
12+
"operator": "BELOW",
13+
"threshold": 95.0,
14+
"thresholdDuration": 3600,
15+
"thresholdOccurrences": "ALL"
16+
},
17+
{
18+
"priority": "WARNING",
19+
"operator": "BELOW",
20+
"threshold": 98.0,
21+
"thresholdDuration": 3600,
22+
"thresholdOccurrences": "ALL"
23+
}
24+
],
25+
"signal": {
26+
"aggregationWindow": 300,
27+
"aggregationMethod": "EVENT_FLOW",
28+
"aggregationDelay": 120,
29+
"fillOption": "NONE"
30+
},
31+
"expiration": {
32+
"expirationDuration": 7200,
33+
"openViolationOnExpiration": false,
34+
"closeViolationsOnExpiration": true
35+
},
36+
"violationTimeLimitSeconds": 86400
37+
}

‎newrelic/dashboards/email-delivery.json‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
{
2+
"name": "Email delivery (Brevo)",
3+
"description": "Closes the '201 ≠ delivered' gap. Brevo's transactional API returns 201 the instant it accepts the POST; the actual SMTP relay happens async. This dashboard surfaces the RECEIVER-side ledger updates (api /webhooks/brevo/:secret) so the team sees what Brevo's relay actually did, not what the API said it would do. Source: brevo_webhook_events_total Prometheus counter + forwarder_sent rows via NRQL log queries.",
4+
"permissions": "PUBLIC_READ_WRITE",
5+
"pages": [
6+
{
7+
"name": "Delivery",
8+
"description": "Live delivery ratio + per-event-class breakdown",
9+
"widgets": [
10+
{
11+
"title": "Delivery ratio (delivered / all-terminal events)",
12+
"layout": { "column": 1, "row": 1, "width": 4, "height": 3 },
13+
"visualization": { "id": "viz.billboard" },
14+
"rawConfiguration": {
15+
"nrqlQueries": [
16+
{
17+
"accountIds": [0],
18+
"query": "SELECT percentage(sum(brevo_webhook_events_total), WHERE event = 'delivered') AS 'delivered %' FROM Metric WHERE event IN ('delivered', 'bounced_hard', 'bounced_soft', 'rejected', 'complaint', 'deferred', 'unsubscribed', 'error') SINCE 1 hour ago"
19+
}
20+
],
21+
"thresholds": [
22+
{ "value": 95, "alertSeverity": "CRITICAL" },
23+
{ "value": 98, "alertSeverity": "WARNING" }
24+
],
25+
"platformOptions": { "ignoreTimeRange": false }
26+
}
27+
},
28+
{
29+
"title": "Events received by class (1h)",
30+
"layout": { "column": 5, "row": 1, "width": 4, "height": 3 },
31+
"visualization": { "id": "viz.pie" },
32+
"rawConfiguration": {
33+
"nrqlQueries": [
34+
{
35+
"accountIds": [0],
36+
"query": "SELECT sum(brevo_webhook_events_total) FROM Metric FACET event SINCE 1 hour ago"
37+
}
38+
],
39+
"platformOptions": { "ignoreTimeRange": false }
40+
}
41+
},
42+
{
43+
"title": "Delivery ratio over time",
44+
"layout": { "column": 9, "row": 1, "width": 4, "height": 3 },
45+
"visualization": { "id": "viz.line" },
46+
"rawConfiguration": {
47+
"nrqlQueries": [
48+
{
49+
"accountIds": [0],
50+
"query": "SELECT percentage(sum(brevo_webhook_events_total), WHERE event = 'delivered') AS 'delivered %' FROM Metric WHERE event IN ('delivered', 'bounced_hard', 'bounced_soft', 'rejected', 'complaint', 'deferred', 'unsubscribed', 'error') TIMESERIES AUTO"
51+
}
52+
],
53+
"platformOptions": { "ignoreTimeRange": false }
54+
}
55+
},
56+
{
57+
"title": "Per-class event rate over time",
58+
"layout": { "column": 1, "row": 4, "width": 12, "height": 3 },
59+
"visualization": { "id": "viz.line" },
60+
"rawConfiguration": {
61+
"nrqlQueries": [
62+
{
63+
"accountIds": [0],
64+
"query": "SELECT rate(sum(brevo_webhook_events_total), 5 minutes) FROM Metric WHERE event IN ('delivered', 'bounced_hard', 'bounced_soft', 'rejected', 'complaint', 'deferred', 'unsubscribed', 'error', 'unhandled') FACET event TIMESERIES AUTO"
65+
}
66+
],
67+
"platformOptions": { "ignoreTimeRange": false }
68+
}
69+
},
70+
{
71+
"title": "Receiver-side rejection paths (auth + payload)",
72+
"layout": { "column": 1, "row": 7, "width": 6, "height": 3 },
73+
"visualization": { "id": "viz.line" },
74+
"rawConfiguration": {
75+
"nrqlQueries": [
76+
{
77+
"accountIds": [0],
78+
"query": "SELECT rate(sum(brevo_webhook_events_total), 5 minutes) FROM Metric WHERE event IN ('unauthorized', 'invalid_payload', 'oversized', 'missing_message_id') FACET event TIMESERIES AUTO"
79+
}
80+
],
81+
"platformOptions": { "ignoreTimeRange": false }
82+
}
83+
},
84+
{
85+
"title": "Send-vs-deliver (worker INFO logs vs receiver INFO logs)",
86+
"layout": { "column": 7, "row": 7, "width": 6, "height": 3 },
87+
"visualization": { "id": "viz.line" },
88+
"rawConfiguration": {
89+
"nrqlQueries": [
90+
{
91+
"accountIds": [0],
92+
"query": "SELECT count(*) FROM Log WHERE message IN ('email.brevo.event_sent', 'webhook.brevo.delivered') FACET message TIMESERIES AUTO"
93+
}
94+
],
95+
"platformOptions": { "ignoreTimeRange": false }
96+
}
97+
}
98+
]
99+
}
100+
]
101+
}

0 commit comments

Comments
 (0)