|
| 1 | +# Deliverability Guide |
| 2 | + |
| 3 | +How to handle bounces, complaints, and address suppression in the Constructive notification system. |
| 4 | + |
| 5 | +For SQL-level internals (generator source, index details), see the `constructive-db-notifications` skill in constructive-db. |
| 6 | + |
| 7 | +## Setting Up Provider Webhooks |
| 8 | + |
| 9 | +### Amazon SES |
| 10 | + |
| 11 | +1. **Create an SNS topic** for bounce/complaint notifications: |
| 12 | + ```bash |
| 13 | + aws sns create-topic --name ses-notifications |
| 14 | + ``` |
| 15 | + |
| 16 | +2. **Subscribe your webhook endpoint** to the topic: |
| 17 | + ```bash |
| 18 | + aws sns subscribe \ |
| 19 | + --topic-arn arn:aws:sns:us-east-1:123456789:ses-notifications \ |
| 20 | + --protocol https \ |
| 21 | + --notification-endpoint https://api.yourapp.com/webhooks/ses |
| 22 | + ``` |
| 23 | + |
| 24 | +3. **Configure SES to publish to the topic:** |
| 25 | + ```bash |
| 26 | + aws ses set-identity-notification-topic \ |
| 27 | + --identity your-domain.com \ |
| 28 | + --notification-type Bounce \ |
| 29 | + --sns-topic arn:aws:sns:us-east-1:123456789:ses-notifications |
| 30 | + |
| 31 | + aws ses set-identity-notification-topic \ |
| 32 | + --identity your-domain.com \ |
| 33 | + --notification-type Complaint \ |
| 34 | + --sns-topic arn:aws:sns:us-east-1:123456789:ses-notifications |
| 35 | + ``` |
| 36 | + |
| 37 | +4. **Webhook handler pseudocode:** |
| 38 | + ```ts |
| 39 | + async function handleSESWebhook(payload) { |
| 40 | + const message = JSON.parse(payload.Message); |
| 41 | + const messageId = message.mail.messageId; |
| 42 | + |
| 43 | + // 1. Correlate via provider_message_id |
| 44 | + const logRow = await db.query( |
| 45 | + `SELECT * FROM notification_delivery_log WHERE provider_message_id = $1`, |
| 46 | + [messageId] |
| 47 | + ); |
| 48 | + |
| 49 | + // 2. Update delivery log status |
| 50 | + if (message.notificationType === 'Bounce') { |
| 51 | + await db.query( |
| 52 | + `UPDATE notification_delivery_log SET status = 'bounced', response_payload = $1 WHERE provider_message_id = $2`, |
| 53 | + [message, messageId] |
| 54 | + ); |
| 55 | + |
| 56 | + // 3. Suppress hard bounces |
| 57 | + if (message.bounce.bounceType === 'Permanent') { |
| 58 | + for (const recipient of message.bounce.bouncedRecipients) { |
| 59 | + await db.query( |
| 60 | + `INSERT INTO notification_suppressions (address, channel_type, reason, source, provider, metadata) |
| 61 | + VALUES ($1, 'email', 'hard_bounce', 'ses_webhook', 'ses', $2) |
| 62 | + ON CONFLICT (address, channel_type) DO NOTHING`, |
| 63 | + [recipient.emailAddress, message] |
| 64 | + ); |
| 65 | + } |
| 66 | + } |
| 67 | + } |
| 68 | + |
| 69 | + if (message.notificationType === 'Complaint') { |
| 70 | + await db.query( |
| 71 | + `UPDATE notification_delivery_log SET status = 'complained', response_payload = $1 WHERE provider_message_id = $2`, |
| 72 | + [message, messageId] |
| 73 | + ); |
| 74 | + |
| 75 | + for (const recipient of message.complaint.complainedRecipients) { |
| 76 | + await db.query( |
| 77 | + `INSERT INTO notification_suppressions (address, channel_type, reason, source, provider, metadata) |
| 78 | + VALUES ($1, 'email', 'complained', 'ses_webhook', 'ses', $2) |
| 79 | + ON CONFLICT (address, channel_type) DO NOTHING`, |
| 80 | + [recipient.emailAddress, message] |
| 81 | + ); |
| 82 | + } |
| 83 | + } |
| 84 | + } |
| 85 | + ``` |
| 86 | + |
| 87 | +### SendGrid |
| 88 | + |
| 89 | +1. **Configure Event Webhook** in SendGrid dashboard → Settings → Mail Settings → Event Webhook. |
| 90 | +2. Set the POST URL to `https://api.yourapp.com/webhooks/sendgrid`. |
| 91 | +3. Enable: `bounce`, `dropped`, `spamreport`, `delivered`. |
| 92 | + |
| 93 | +4. **Webhook handler pseudocode:** |
| 94 | + ```ts |
| 95 | + async function handleSendGridWebhook(events) { |
| 96 | + for (const event of events) { |
| 97 | + const messageId = event.sg_message_id?.split('.')[0]; // strip filter suffix |
| 98 | + |
| 99 | + if (event.event === 'bounce') { |
| 100 | + await db.query( |
| 101 | + `UPDATE notification_delivery_log SET status = 'bounced', response_payload = $1 WHERE provider_message_id = $2`, |
| 102 | + [event, messageId] |
| 103 | + ); |
| 104 | + await db.query( |
| 105 | + `INSERT INTO notification_suppressions (address, channel_type, reason, source, provider, metadata) |
| 106 | + VALUES ($1, 'email', 'hard_bounce', 'sendgrid_webhook', 'sendgrid', $2) |
| 107 | + ON CONFLICT (address, channel_type) DO NOTHING`, |
| 108 | + [event.email, event] |
| 109 | + ); |
| 110 | + } |
| 111 | + |
| 112 | + if (event.event === 'spamreport') { |
| 113 | + await db.query( |
| 114 | + `UPDATE notification_delivery_log SET status = 'complained', response_payload = $1 WHERE provider_message_id = $2`, |
| 115 | + [event, messageId] |
| 116 | + ); |
| 117 | + await db.query( |
| 118 | + `INSERT INTO notification_suppressions (address, channel_type, reason, source, provider, metadata) |
| 119 | + VALUES ($1, 'email', 'complained', 'sendgrid_webhook', 'sendgrid', $2) |
| 120 | + ON CONFLICT (address, channel_type) DO NOTHING`, |
| 121 | + [event.email, event] |
| 122 | + ); |
| 123 | + } |
| 124 | + } |
| 125 | + } |
| 126 | + ``` |
| 127 | + |
| 128 | +### Twilio (SMS) |
| 129 | + |
| 130 | +Configure `StatusCallback` URL when sending: |
| 131 | + |
| 132 | +```ts |
| 133 | +const message = await client.messages.create({ |
| 134 | + to: '+15551234567', |
| 135 | + from: '+15559876543', |
| 136 | + body: 'Your verification code is 123456', |
| 137 | + statusCallback: 'https://api.yourapp.com/webhooks/twilio' |
| 138 | +}); |
| 139 | + |
| 140 | +// Store message.sid as provider_message_id in delivery_log |
| 141 | +``` |
| 142 | + |
| 143 | +Handle `undelivered` and `failed` statuses to suppress undeliverable phone numbers. |
| 144 | + |
| 145 | +## Suppression List Management |
| 146 | + |
| 147 | +### Checking before send |
| 148 | + |
| 149 | +The delivery worker runs this before every send: |
| 150 | + |
| 151 | +```sql |
| 152 | +SELECT EXISTS ( |
| 153 | + SELECT 1 FROM notification_suppressions |
| 154 | + WHERE address = $1 AND channel_type = $2 |
| 155 | +) AS is_suppressed; |
| 156 | +``` |
| 157 | + |
| 158 | +### Viewing recent suppressions |
| 159 | + |
| 160 | +```sql |
| 161 | +SELECT address, channel_type, reason, source, provider, created_at |
| 162 | +FROM notification_suppressions |
| 163 | +ORDER BY created_at DESC |
| 164 | +LIMIT 50; |
| 165 | +``` |
| 166 | + |
| 167 | +### Removing a suppression |
| 168 | + |
| 169 | +When a user re-verifies their email or an admin decides to retry: |
| 170 | + |
| 171 | +```sql |
| 172 | +DELETE FROM notification_suppressions |
| 173 | +WHERE address = 'dan@example.com' AND channel_type = 'email'; |
| 174 | +``` |
| 175 | + |
| 176 | +### Bulk export for audit |
| 177 | + |
| 178 | +```sql |
| 179 | +COPY ( |
| 180 | + SELECT address, channel_type, reason, source, provider, created_at |
| 181 | + FROM notification_suppressions |
| 182 | + ORDER BY created_at |
| 183 | +) TO '/tmp/suppressions.csv' WITH CSV HEADER; |
| 184 | +``` |
| 185 | + |
| 186 | +## Channel Auto-Deactivation |
| 187 | + |
| 188 | +Each `notification_channels` row tracks consecutive failures: |
| 189 | + |
| 190 | +``` |
| 191 | +failed_count: 0 → 1 → 2 → 3 → is_active = false |
| 192 | +``` |
| 193 | + |
| 194 | +The delivery worker should: |
| 195 | +1. On failure: increment `failed_count`, set `last_error` and `last_error_at` |
| 196 | +2. On success: reset `failed_count = 0`, set `last_used_at = now()` |
| 197 | +3. When `failed_count` exceeds threshold (e.g., 3): set `is_active = false` |
| 198 | + |
| 199 | +This is **per-endpoint** deactivation (expired push tokens, revoked webhooks). Address-level suppression (bounced emails) is handled separately by `notification_suppressions`. |
| 200 | + |
| 201 | +## Monitoring & Troubleshooting |
| 202 | + |
| 203 | +### Delivery success rate (last 24h) |
| 204 | + |
| 205 | +```sql |
| 206 | +SELECT channel_type, status, COUNT(*) AS cnt, |
| 207 | + ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (PARTITION BY channel_type), 1) AS pct |
| 208 | +FROM notification_delivery_log |
| 209 | +WHERE created_at > now() - interval '24 hours' |
| 210 | +GROUP BY channel_type, status |
| 211 | +ORDER BY channel_type, cnt DESC; |
| 212 | +``` |
| 213 | + |
| 214 | +### Complaint rate (should be < 0.1%) |
| 215 | + |
| 216 | +```sql |
| 217 | +SELECT |
| 218 | + COUNT(*) FILTER (WHERE status = 'complained') AS complaints, |
| 219 | + COUNT(*) FILTER (WHERE status IN ('delivered', 'sent')) AS total_sent, |
| 220 | + ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'complained') |
| 221 | + / NULLIF(COUNT(*) FILTER (WHERE status IN ('delivered', 'sent')), 0), 3) AS complaint_rate_pct |
| 222 | +FROM notification_delivery_log |
| 223 | +WHERE channel_type = 'email' |
| 224 | + AND created_at > now() - interval '7 days'; |
| 225 | +``` |
| 226 | + |
| 227 | +### Channels approaching deactivation |
| 228 | + |
| 229 | +```sql |
| 230 | +SELECT id, owner_id, channel_type, endpoint, failed_count, last_error, last_error_at |
| 231 | +FROM notification_channels |
| 232 | +WHERE failed_count > 0 AND is_active = true |
| 233 | +ORDER BY failed_count DESC; |
| 234 | +``` |
| 235 | + |
| 236 | +### Debug a specific delivery |
| 237 | + |
| 238 | +```sql |
| 239 | +SELECT status, provider, provider_message_id, error, response_payload, attempt_count, sent_at |
| 240 | +FROM notification_delivery_log |
| 241 | +WHERE notification_id = $1 |
| 242 | +ORDER BY created_at; |
| 243 | +``` |
0 commit comments