Skip to content

Commit 65976dc

Browse files
MartinPaulEveMartinPaulEve
committed
docs(reply-by-email): document SparkPost reply-by-email and dev limitation
Add docs/reply-by-email.md covering the inbound (SparkPost) reply-by-email flow, the per-network configuration, SparkPost API setup, the error-1602 domain namespace conflict, and the fact that the non-public dev environment cannot receive the inbound relay webhook. Link it from the docs index.
1 parent 13a2a8e commit 65976dc

2 files changed

Lines changed: 316 additions & 1 deletion

File tree

‎docs/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,5 @@ The `docs` directory contains developer-oriented public documentation specific t
1212
- [Scripts reference](scripts.md)
1313
- [Working with Git subtrees](subtrees.md)
1414
- [Working with user content](user-content.md)
15-
- [CILogon `resync-all` WP-CLI command](cilogon-resync.md)
15+
- [CILogon `resync-all` WP-CLI command](cilogon-resync.md)
16+
- [Reply By Email (BuddyPress group discussions)](reply-by-email.md)

‎docs/reply-by-email.md‎

Lines changed: 314 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
# Reply By Email (BuddyPress group discussions)
2+
3+
Knowledge Commons lets members reply to BuddyPress group activity and
4+
discussion notifications directly from their email client. This is provided by
5+
the bundled **BP Reply By Email** (RBE) plugin in
6+
`plugins/bp-reply-by-email/`, using [SparkPost](https://www.sparkpost.com/) as
7+
the inbound-email provider.
8+
9+
This document explains how the feature works end to end, how SparkPost is
10+
configured, what has to be set up for a given network/domain, and how to
11+
diagnose the common failure mode where "nothing fires."
12+
13+
> ## ⚠️ The dev environment cannot test reply-by-email
14+
>
15+
> Reply-by-email depends on SparkPost making an **inbound HTTP POST into the
16+
> site** (the relay webhook — hop 3 below). **The dev environment is not
17+
> externally accessible**, so SparkPost has no route to reach it and the webhook
18+
> can never be delivered. This is a hard limitation of the architecture, not a
19+
> configuration bug:
20+
>
21+
> - You can configure everything correctly — inbound domain, MX records, relay
22+
> webhook, RBE settings — and replies will **still** never post on dev, because
23+
> the final hop physically cannot complete.
24+
> - Therefore **do not expect to verify reply-by-email end to end on dev.**
25+
> The feature can only be exercised on an environment SparkPost can reach over
26+
> the public internet (staging or production).
27+
> - Dev is still useful for verifying the *outbound* half (that notifications go
28+
> out with correctly-encoded reply-to addresses) and the SparkPost-side
29+
> configuration, but the round trip cannot be completed there.
30+
31+
## Operating mode: Inbound, not IMAP
32+
33+
RBE supports two modes:
34+
35+
- **IMAP** — the site connects to an IMAP mailbox and polls it for replies.
36+
- **Inbound** — replies are routed through a third-party inbound-email provider
37+
that parses the message and POSTs it to the site as a webhook.
38+
39+
**Knowledge Commons uses Inbound mode with SparkPost.** The only RBE
40+
configuration that lives in this repository is the SparkPost webhook token,
41+
defined in `site/config/application.php`:
42+
43+
```php
44+
Config::define( 'BP_RBE_SPARKPOST_WEBHOOK_TOKEN', getenv( 'BP_RBE_SPARKPOST_WEBHOOK_TOKEN' ) );
45+
Config::define( 'BP_RBE_DEBUG', getenv( 'BP_RBE_DEBUG' ) );
46+
```
47+
48+
IMAP-mode settings (mail server, port, keep-alive, etc.) on the admin page are
49+
**not** used on KC and can be ignored.
50+
51+
## How it works, end to end
52+
53+
Reply-by-email is a three-hop chain. A break in any hop means replies silently
54+
fail to post.
55+
56+
### 1. Outbound — the encoded reply-to address
57+
58+
When BuddyPress sends a group notification, RBE rewrites the reply-to address to
59+
an *encoded address at the network's inbound domain*. See
60+
`bp_rbe_inject_qs_in_email()` in `includes/bp-rbe-functions.php`:
61+
62+
```
63+
THEQUERYSTRING@<inbound-domain> // e.g. a1b2c3...@post.hcommons-dev.org
64+
```
65+
66+
`THEQUERYSTRING` is an encrypted token (`bp_rbe_encode()`) that encodes the
67+
replying user, the target group, and the item being replied to. It is signed
68+
with the **Key** set in the RBE admin settings — a reply encoded under one key
69+
cannot be decoded under another.
70+
71+
### 2. Inbound mail → SparkPost
72+
73+
The member replies; their mail client sends to `…@<inbound-domain>`. For that
74+
mail to reach SparkPost rather than bouncing:
75+
76+
- The inbound domain's **MX records must point at SparkPost's inbound relay
77+
hosts** (`rx1.sparkpostmail.com`, `rx2.sparkpostmail.com`).
78+
- The domain must be registered as an **Inbound Domain** in SparkPost. Unlike
79+
*sending* domains, inbound domains have **no DKIM/ownership challenge to
80+
"verify"** — an inbound domain is simply created via the API, and it becomes
81+
functional once its MX records resolve to SparkPost and a relay webhook is
82+
bound to it.
83+
84+
### 3. SparkPost → the site (relay webhook)
85+
86+
SparkPost parses the inbound message and POSTs it as JSON to a **Relay Webhook**
87+
URL on the site. On the WordPress side, RBE catches the request on `wp_loaded`
88+
at priority `0` (`includes/bp-rbe-hooks.php`):
89+
90+
```php
91+
add_action( 'wp_loaded', 'bp_rbe_inbound_catch_callback', 0 );
92+
```
93+
94+
The callback runs on *every* request, but the SparkPost parser
95+
(`includes/classes/bp-reply-by-email-inbound-provider-sparkpost.php`) only acts
96+
when **both** of the following are true:
97+
98+
1. `Content-Type: application/json`, and
99+
2. the request carries the `X-MessageSystems-Webhook-Token` header.
100+
101+
It then compares that header against `BP_RBE_SPARKPOST_WEBHOOK_TOKEN`. **If the
102+
token does not match, the request is logged and `die()`d before anything is
103+
parsed.** On success it looks up the WordPress user by the message's `from`
104+
address, decodes the querystring to resolve the group/item, and posts the reply.
105+
106+
Because the catch runs on `wp_loaded`, the relay webhook can target essentially
107+
any URL on the site (the site home URL is fine) — there is no dedicated endpoint
108+
path.
109+
110+
> This third hop is exactly why **dev cannot test the feature**: SparkPost must
111+
> open a connection *to* the site from the public internet, and dev is not
112+
> publicly reachable. See the warning at the top of this document.
113+
114+
## What must be configured for a domain/network
115+
116+
Reply-by-email is **not** self-provisioning. Nothing in the plugin registers a
117+
domain automatically. Each network/domain needs all of the following.
118+
119+
> **Multi-network caveat.** The RBE admin page is *not* network-aware (see the
120+
> `@todo` in `includes/bp-rbe-admin.php` and the `bp_is_root_blog()` guard in
121+
> `setup_admin()`). Settings are stored per root blog via
122+
> `bp_get_option( 'bp-rbe' )`. KC runs several networks (hcommons, hastac,
123+
> hcommons-dev, …), each with its own root blog, so **each network needs its own
124+
> RBE settings filled in** — they are not inherited from production.
125+
126+
### DNS (for the reply domain)
127+
128+
- MX records pointing at SparkPost's inbound relay hosts:
129+
130+
```
131+
<reply-domain>. MX 10 rx1.sparkpostmail.com.
132+
<reply-domain>. MX 10 rx2.sparkpostmail.com.
133+
```
134+
135+
### SparkPost
136+
137+
- Register the reply domain as an **Inbound Domain**.
138+
- Create a **Relay Webhook** that matches that domain and POSTs to the site's
139+
URL, carrying an **auth token**.
140+
141+
See [Configuring SparkPost](#configuring-sparkpost) below for the exact API
142+
calls.
143+
144+
### WordPress / environment (on the target network's root blog)
145+
146+
- Define `BP_RBE_SPARKPOST_WEBHOOK_TOKEN` (read via `getenv()` in
147+
`application.php`) and make it **identical** to the token configured on the
148+
SparkPost relay webhook.
149+
- In the BP Reply By Email admin settings:
150+
- **Mode** = Inbound
151+
- **Provider** = SparkPost
152+
- **Inbound Domain** = the reply domain (must exactly match the SparkPost
153+
inbound domain and the MX records)
154+
- Note the **Key** — replies are only decodable under the key they were
155+
encoded with.
156+
157+
## Configuring SparkPost
158+
159+
Inbound domains and relay webhooks are managed through the **SparkPost API**, not
160+
the web dashboard. All calls below assume `$SPARKPOST_API_KEY` is exported and an
161+
API key with the *Inbound Domains* and *Relay Webhooks* permissions. EU accounts
162+
use `https://api.eu.sparkpost.com` instead of `https://api.sparkpost.com`.
163+
164+
### Naming convention and the dev reply domain
165+
166+
Production/staging networks use a `reply.<network>` label (e.g.
167+
`reply.hcommons-staging.org`). For **dev**, the `reply.hcommons-dev.org` label is
168+
**unavailable** — it collides with another SparkPost resource of the same name
169+
(see [Domain namespace conflicts](#domain-namespace-conflicts-error-1602)
170+
below). The dev reply domain is therefore **`post.hcommons-dev.org`**.
171+
172+
### Shared auth token
173+
174+
All KC relay webhooks share a single `auth_token`. Whatever value the target
175+
environment has in `BP_RBE_SPARKPOST_WEBHOOK_TOKEN` must be the value used in its
176+
relay webhook — they have to match exactly or the parser rejects the POST.
177+
178+
### Create the inbound domain
179+
180+
```bash
181+
# confirm the label is free (expect a 404)
182+
curl -s https://api.sparkpost.com/api/v1/inbound-domains/post.hcommons-dev.org \
183+
-H "Authorization: $SPARKPOST_API_KEY" | jq
184+
185+
# create it
186+
curl -sX POST https://api.sparkpost.com/api/v1/inbound-domains \
187+
-H "Authorization: $SPARKPOST_API_KEY" \
188+
-H "Content-Type: application/json" \
189+
-d '{"domain": "post.hcommons-dev.org"}'
190+
```
191+
192+
The inbound domain can be created before DNS is in place; SparkPost will not
193+
relay mail until the MX records resolve, but the registration itself succeeds.
194+
195+
### Create the relay webhook
196+
197+
```bash
198+
curl -sX POST https://api.sparkpost.com/api/v1/relay-webhooks \
199+
-H "Authorization: $SPARKPOST_API_KEY" \
200+
-H "Content-Type: application/json" \
201+
-d '{
202+
"name": "HCommons Dev Replies Webhook",
203+
"target": "https://hcommons-dev.org",
204+
"auth_token": "<value of BP_RBE_SPARKPOST_WEBHOOK_TOKEN in target env>",
205+
"match": { "protocol": "SMTP", "domain": "post.hcommons-dev.org" }
206+
}'
207+
```
208+
209+
### Listing / inspecting existing configuration
210+
211+
```bash
212+
# inbound domains (account-global)
213+
curl -s https://api.sparkpost.com/api/v1/inbound-domains \
214+
-H "Authorization: $SPARKPOST_API_KEY" | jq
215+
216+
# relay webhooks (master scope; pass X-MSYS-SUBACCOUNT: <id> for a subaccount)
217+
curl -s https://api.sparkpost.com/api/v1/relay-webhooks \
218+
-H "Authorization: $SPARKPOST_API_KEY" | jq
219+
220+
# sending domains — useful when diagnosing namespace conflicts
221+
curl -s https://api.sparkpost.com/api/v1/sending-domains/<domain> \
222+
-H "Authorization: $SPARKPOST_API_KEY" | jq
223+
```
224+
225+
## Troubleshooting "nothing fires"
226+
227+
> Remember the overriding constraint: **on dev, "nothing fires" is expected** —
228+
> SparkPost cannot reach a non-public host, so the relay webhook never arrives.
229+
> The steps below apply to publicly-reachable environments (staging/production).
230+
231+
Walk the three hops in order; each has a distinct symptom.
232+
233+
### 1. Turn on debug logging
234+
235+
`BP_RBE_DEBUG` is already wired in `application.php`. With it enabled, a
236+
successful webhook hit logs:
237+
238+
```
239+
- SparkPost webhook received -
240+
- Webhook parsing completed -
241+
```
242+
243+
A token mismatch logs `SparkPost token verification failed.` and then stops.
244+
**No log line at all** on a reply attempt means SparkPost never reached the site
245+
— the problem is upstream (hop 2, the relay webhook, or — on dev — the host not
246+
being publicly reachable), not in WordPress.
247+
248+
### 2. Trace each hop
249+
250+
- **Hop 2 (mail → SparkPost):** Check SparkPost's inbound/event logs. Did the
251+
reply arrive at SparkPost at all? If not, the MX/inbound-domain registration is
252+
the issue even if the MX record *exists* — confirm the domain is present as an
253+
**Inbound Domain** in SparkPost and that MX resolves to the relay hosts.
254+
- **Hop 3 (SparkPost → site):** Check SparkPost's **relay webhook delivery log**.
255+
Is a relay webhook configured for this domain, pointed at the correct site URL,
256+
and is the site **publicly reachable**? A missing/misaddressed webhook — or an
257+
unreachable host — stops everything here. Confirm the webhook's `auth_token`
258+
matches `BP_RBE_SPARKPOST_WEBHOOK_TOKEN` in that environment; a mismatch shows
259+
as `SparkPost token verification failed.` in the site log.
260+
- **WordPress settings:** Confirm the network's root blog has Mode = Inbound,
261+
Provider = SparkPost, and **Inbound Domain** set to the reply domain. If
262+
outbound notifications carry a malformed or empty reply-to address, this
263+
setting is missing — replies would be undeliverable regardless of the relay
264+
configuration.
265+
266+
### Domain namespace conflicts (error 1602)
267+
268+
When creating an inbound domain, SparkPost may reject it with:
269+
270+
```json
271+
{ "errors": [ { "message": "resource conflict",
272+
"description": "An inbound domain with similar attributes already exists.",
273+
"code": "1602" } ] }
274+
```
275+
276+
This means the label is already claimed **somewhere in the account by a resource
277+
of any type** — an inbound domain, a CNAME/MX-verified sending or bounce domain,
278+
a tracking domain, or a relay-webhook match. Crucially, the conflicting resource
279+
may be **invisible to your `GET /inbound-domains` listing**: that list is
280+
master-scoped, and the clash can be with a *different resource type* that shares
281+
the name.
282+
283+
A domain (and its subdomains) can only be claimed once across an account and its
284+
subaccounts, so a `GET` for the exact inbound domain can return 404 while the
285+
create still conflicts. To diagnose, check each resource type for the label:
286+
287+
```bash
288+
for kind in inbound-domains sending-domains tracking-domains; do
289+
echo "== $kind =="
290+
curl -s https://api.sparkpost.com/api/v1/$kind/<label> \
291+
-H "Authorization: $SPARKPOST_API_KEY" | jq '.results // .errors'
292+
done
293+
# and check whether any relay webhook already matches the label:
294+
curl -s https://api.sparkpost.com/api/v1/relay-webhooks \
295+
-H "Authorization: $SPARKPOST_API_KEY" | jq '.results[].match'
296+
```
297+
298+
**Resolution:** either remove the conflicting resource (only if it is unused —
299+
e.g. a stray tracking domain on a dev host), or **choose a different reply
300+
label**. The latter is why dev uses `post.hcommons-dev.org` rather than
301+
`reply.hcommons-dev.org`: the `reply.` label was already claimed and the
302+
low-risk fix was to pick a fresh, unclaimed label. Any label works as long as
303+
its MX points at SparkPost, an inbound domain + relay webhook exist for it, and
304+
RBE's **Inbound Domain** setting matches it exactly.
305+
306+
## Key files
307+
308+
| Path | Purpose |
309+
| --- | --- |
310+
| `plugins/bp-reply-by-email/includes/bp-rbe-admin.php` | Admin settings page (mode, provider, inbound domain, key). |
311+
| `plugins/bp-reply-by-email/includes/bp-rbe-hooks.php` | Registers `bp_rbe_inbound_catch_callback` on `wp_loaded`. |
312+
| `plugins/bp-reply-by-email/includes/bp-rbe-functions.php` | Address encoding (`bp_rbe_inject_qs_in_email`, `bp_rbe_encode`) and webhook dispatch. |
313+
| `plugins/bp-reply-by-email/includes/classes/bp-reply-by-email-inbound-provider-sparkpost.php` | SparkPost webhook parser and token verification. |
314+
| `site/config/application.php` | Defines `BP_RBE_SPARKPOST_WEBHOOK_TOKEN` and `BP_RBE_DEBUG`. |

0 commit comments

Comments
 (0)