hyperpolymath
diff --git a/‎docs/decisions/0011-webhooks-inbound-mcp-notifications.md‎
Lines changed: 198 additions & 0 deletions b/‎docs/decisions/0011-webhooks-inbound-mcp-notifications.md‎
Lines changed: 198 additions & 0 deletions
@@ -0,0 +1,198 @@
+<!-- SPDX-License-Identifier: MPL-2.0 -->
+<!-- Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> -->
+
+# 11. Webhooks inbound + MCP notifications — closing the agent feedback loop
+
+Date: 2026-05-20
+
+## Status
+
+Proposed (RFC — implementation tracked in epic #87 item 5)
+
+## Context
+
+BoJ today is **strictly pull-based**: an MCP client calls a tool, BoJ dispatches to a cartridge, returns a response. There is no path for *external events to surface to the connected LLM*. Concretely:
+
+- A GitHub Actions workflow fails → no notification surfaces; the agent must poll
+- Cloudflare detects a DDoS → no notification; the agent must poll
+- Sentry fires an alert → no notification; the agent must poll
+- A new PR opens on a watched repo → no notification; the agent must poll
+- A long-running sandbox job (ADR-0009) completes → no notification; the agent must poll
+
+This forces agents into either (a) polling loops, which burn LLM context and quota, or (b) blindness to events between tool calls. Either way, BoJ ends up being a query interface, not an operator. The "agent feedback loop" that's the differentiator for multi-agent systems doesn't close.
+
+MCP supports server-pushed notifications via `notifications/*` methods. BoJ declares the *capability* (`tools/listChanged`, `prompts/listChanged`) but the bridge never *emits* a notification. The infrastructure is there; the wiring isn't.
+
+External webhook sources (GitHub, GitLab, Cloudflare, Sentry, Stripe-style services) are the canonical mechanism for "something happened, here it is, on your URL of choice." BoJ can sit between webhook sources and MCP clients to bridge the gap.
+
+## Decision
+
+Add an **inbound webhook listener** to the BoJ REST backend (port 7700 — the existing Cowboy listener per ADR-0004) and a **MCP notification fan-out** layer that surfaces matching webhook events as `notifications/*` messages to connected MCP clients.
+
+### Architectural shape
+
+```
+External source ──HTTPS──▶ BoJ REST /webhooks/{provider}/{token}
+                                │
+                                ▼
+                        verify signature (HMAC / OIDC)
+                                │
+                                ▼
+                        match against subscriptions
+                                │
+                                ▼
+                   ┌────────────┼─────────────┐
+                   ▼            ▼             ▼
+              MCP client A  MCP client B  audit log
+                (notifications/event)
+```
+
+### Inbound endpoint
+
+The REST backend grows two new endpoints (no change to the bridge's stdio surface):
+
+- `POST /webhooks/{provider}/{token}` — receives external webhook payloads
+- `GET /webhooks/subscriptions` — operator-side, lists active subscriptions
+
+Providers in v1: `github`, `gitlab`, `cloudflare`, `sentry`, `stripe`, `generic`. `generic` accepts any JSON payload and applies generic shape matching; provider-specific endpoints know the source's signature scheme and event-type taxonomy.
+
+The `{token}` segment is an opaque, per-subscription secret. Loss of the token URL = subscription compromise (can spoof events). Tokens are generated by `boj_webhook_subscribe` (see below) and stored in an encrypted form on disk; rotated by `boj_webhook_rotate`.
+
+### Subscription model
+
+Subscriptions are first-class. Stored in a new `webhook-subscriptions.a2ml` file under `~/.boj/webhooks/`. Schema:
+
+```a2ml
+- id: <uuid>
+  provider: github
+  token: <opaque>           # the path segment for the inbound URL
+  filter:                   # only emit notification if event matches
+    event_types: [pull_request.opened, push, workflow_run.completed]
+    refs: ["refs/heads/main"]
+    repos: ["hyperpolymath/boj-server"]
+  fan_out:                  # which MCP clients receive the notification
+    - client_kind: claude   # all Claude sessions
+    - peer_token: <coord_token>  # one specific coord-registered peer
+    - "*"                   # broadcast to all connected MCP clients (default)
+  audit: true               # log every received event regardless of fan-out
+```
+
+### Bridge-level tools (subscription management)
+
+- `boj_webhook_subscribe` — create a subscription; returns `(url, token, subscription_id)`
+- `boj_webhook_list` — list active subscriptions
+- `boj_webhook_unsubscribe` — delete a subscription
+- `boj_webhook_rotate` — rotate the token for a subscription (e.g. after a leak)
+- `boj_webhook_replay` — replay the last N events to the calling MCP client (useful when reconnecting)
+
+These wire to the same `policies/boj-default.ncl` (ADR-0007). `boj_webhook_subscribe` is tier-2 (small_write — creates a resource); `boj_webhook_unsubscribe` is tier-2; `boj_webhook_rotate` is tier-3 (impacts auth).
+
+### MCP notification format
+
+When a webhook event matches a subscription:
+
+```jsonrpc
+{
+  "jsonrpc": "2.0",
+  "method": "notifications/event",
+  "params": {
+    "subscription_id": "<uuid>",
+    "provider": "github",
+    "event_type": "pull_request.opened",
+    "received_at": "2026-05-20T12:34:56Z",
+    "source_signature_verified": true,
+    "payload": { ... raw provider payload ... },
+    "extracted": {
+      // provider-specific normalised fields for LLM convenience
+      "pr_number": 42,
+      "repo": "hyperpolymath/boj-server",
+      "title": "Fix the thing",
+      "author": "alice"
+    }
+  }
+}
+```
+
+The bridge emits this to **every connected MCP client whose fan_out matches the subscription's selector**. Clients that don't want notifications can ignore them; clients that want them can react.
+
+### Replay buffer
+
+Bridge keeps the last 100 events (per subscription, bounded ring buffer) so a reconnecting client can request `boj_webhook_replay` and catch up. Bounded so memory stays predictable.
+
+### Signature verification
+
+Each provider has a documented signature scheme:
+
+| Provider | Scheme |
+|---|---|
+| GitHub | HMAC-SHA256 with subscription's secret on `X-Hub-Signature-256` |
+| GitLab | HMAC-SHA256 on `X-Gitlab-Token` |
+| Cloudflare | Webhook Signature spec (JSON Web Signature) |
+| Sentry | HMAC-SHA256 on `Sentry-Hook-Signature` |
+| Stripe | `Stripe-Signature` with timestamp-windowed HMAC |
+| `generic` | HMAC-SHA256 on `X-Boj-Signature`; constant-time compare |
+
+If signature fails, the event is rejected, logged, and **never reaches the notification path**. `source_signature_verified: true` always means "we verified it"; we never accept unverified events as notifications.
+
+## Consequences
+
+### Positive
+
+- **Closes the agent feedback loop** — agents react to events instead of polling. Major architectural unlock.
+- **MCP-native, not parallel infrastructure** — events come through the same MCP stdio surface clients already speak. No second protocol.
+- **Composable with existing cartridges** — \`github-api-mcp\` already speaks GitHub's REST; this RFC adds the *push* side. Same auth, same provider.
+- **Replay supports reconnection** — long-running MCP clients (Claude Code, Cursor, etc.) that lose stdio connection don't lose events; they catch up on reconnect.
+- **Multi-client fan-out matches the multi-agent story** — one event can wake the right peer on the BoJ coord bus, leaving others uninvolved.
+- **Audit by default** — every received event is logged regardless of fan-out, so post-incident forensics is straightforward.
+- **Signature verification is bedrock** — no path for unverified events to reach a client; eliminates whole classes of webhook-spoofing attack.
+
+### Negative
+
+- **Stdio + push is awkward** — MCP stdio is a duplex stream, but most MCP clients don't expect server-initiated messages. Mitigation: per the MCP spec, `notifications/*` is part of the protocol; clients that implement the spec correctly will handle it. Clients that don't will simply ignore the notification (no fatal failure).
+- **Public ingress surface** — the inbound webhook endpoint must be reachable from the public internet for external providers to call it. Adds firewall configuration burden. Mitigation: document Cloudflare Tunnel / Tailscale-funnel / ngrok patterns; do not require operators to expose port 7700 directly.
+- **Subscription state on disk** — `webhook-subscriptions.a2ml` is sensitive (tokens grant impersonation). Must be `chmod 0600` and lives under \`~/.boj/webhooks/\`. Documented; enforced at write time.
+- **Event ordering** — webhooks from external sources may arrive out of order or be redelivered. Mitigation: include the provider's native idempotency key (delivery ID) in the notification; the LLM (or downstream tooling) deduplicates.
+- **Latency budget** — webhook → signature verify → match → fan-out adds milliseconds. Mitigation: signature verification + matching is in-process; fan-out is async; the external provider doesn't wait for fan-out completion.
+
+## Non-goals
+
+- **Not building a generic event bus** — this is webhook-in, MCP-notification-out. Not for arbitrary inter-cartridge pub-sub (that's coord-mcp's job).
+- **Not implementing webhook *outbound*** — BoJ doesn't *send* webhooks to external systems. It receives them. Outbound notifications are a separate problem.
+- **Not retaining events beyond the replay window** — 100 events × ~10 active subscriptions = bounded memory. Long-term retention is the user's external observability stack (Sentry, etc.).
+- **Not bridging providers** — a GitHub webhook doesn't get re-emitted as a Slack message. That's the LLM's job once it receives the notification; BoJ is the delivery layer, not the policy layer.
+- **Not requiring TLS termination in BoJ** — the REST backend speaks HTTPS but operators are encouraged to terminate at Cloudflare/nginx/Tailscale-funnel. BoJ accepts HTTP locally; the inbound URL is HTTPS by virtue of the operator's chosen ingress.
+
+## Open questions
+
+1. **Subscription persistence across restarts** — \`~/.boj/webhooks/webhook-subscriptions.a2ml\` is the persistent store, but what about the replay buffer? Recommend: replay buffer is in-memory only; subscriptions persist. Replay catches reconnection within a single BoJ process lifetime, not across restarts.
+
+2. **MCP client identification** — when fanning out, "all Claude sessions" requires the bridge to know each client's identity. Today MCP's \`initialize\` carries \`clientInfo\` — sufficient for fan-out matching? Recommend yes, with fallback to peer_token for fine-grained selection.
+
+3. **Provider extensibility** — adding a sixth provider (e.g. Linear webhooks) means new code. Should provider definitions be configuration-driven (Nickel contract per provider) rather than code? Recommend: signature schemes stay code (cryptography); event-shape normalisation can move to Nickel.
+
+4. **Backpressure on slow clients** — if one MCP client is slow to consume notifications, do others wait? Recommend per-client send queues with bounded size; slow client gets events dropped (with a `notifications/missed` summary on reconnect).
+
+5. **Authorization model on subscription creation** — should creating a webhook subscription require master role (ADR-0007 tier-3)? Recommend yes — webhook URLs grant impersonation capability, so creation is master-gated by default; can be overridden in policy.
+
+6. **Reusing the existing REST backend vs separate listener** — the existing REST backend handles cartridge dispatch. Should webhooks share that listener or run on a separate port? Recommend share — ADR-0004 already chose single-listener; honour that. Path-routed (`/webhooks/...`) is unambiguous.
+
+## Implementation sketch
+
+When this RFC is accepted:
+
+1. New module `elixir/lib/boj_rest/webhooks/` with one file per provider's signature scheme.
+2. Persistent store at `~/.boj/webhooks/webhook-subscriptions.a2ml` (chmod 0600); Idris2-verified read/write.
+3. Bridge-level tools `boj_webhook_subscribe` / `list` / `unsubscribe` / `rotate` / `replay` added to \`mcp-bridge/lib/tools.js\` (5 tools).
+4. Bridge notification emitter — new `mcp-bridge/lib/notifications.js` that maintains the per-client send queue + bounded backpressure.
+5. Default policy entries (ADR-0007): `boj_webhook_subscribe` tier-3, others tier-2.
+6. Provider documentation: a new `docs/cartridges/WEBHOOK-PROVIDERS.md` showing how to register a webhook URL with each supported provider.
+7. Tests: per-provider signature verification fixtures + end-to-end replay test.
+
+## Linked
+
+- ADR-0004 (unified gateway) — single listener; webhooks ride the same Cowboy endpoint.
+- ADR-0007 (policy DSL) — webhook subscriptions are tier-gated artefacts.
+- ADR-0008 (cartridge marketplace) — marketplace updates can flow to MCP clients via webhook notifications (see ADR-0008 open question 4).
+- ADR-0009 (sandbox cartridge) — long-running sandbox completion can emit notifications via this mechanism.
+- ADR-0010 (federation) — federated quarantine review can emit notifications to the master's MCP client.
+- Epic #87 item 5 (this).