[integrations] Add Readwise highlight capture (backfill + webhook + schema)

[mlava]

Contribution Type

• Recipe (/recipes/readwise-import)
• Schema (/schemas/readwise-books)
• Integration (/integrations/readwise-capture)

What does this do?

Adds end-to-end Readwise highlight capture in three cooperating pieces:

1. schemas/readwise-books — cache table for book-level metadata (title, author, category, source, cover, highlight count) keyed by Readwise's user_book_id. Plus get_book_highlights and increment_book_highlight_count RPCs.

2. recipes/readwise-import — Python backfill that pages through /api/v2/export/, upserts each book, and inserts highlights as thoughts with source_type='readwise'. Idempotent, filter-aware (by book, source, category, date range, or --list-books discovery), batch-embeds, and recovers from Supabase statement timeouts by splitting the insert batch.

3. integrations/readwise-capture — Supabase Edge Function subscribing to readwise.highlight.created. Covers every source Readwise aggregates (Kindle, Reader, Twitter, Hypothesis, Instapaper, Apple Books, podcasts, Roam, physical books via OCR, etc.). Uses the cache as a write-through lookup.

Shipping together because the integration and recipe both depend on the schema.

Requirements

• Services: Readwise account (Free tier works — webhook creation is available on all plans)
• Tools: Supabase CLI, Python 3.10+
• APIs: Readwise /api/v2/export/ and /api/v2/books/, OpenRouter embeddings (existing OB dependency)

Testing

Full end-to-end test on my own Open Brain:

• Schema: applied cleanly; idempotent.
• Backfill: imported 11,173 highlights / 656 books in under 4 min. Filter paths (source, category, date-range, book-id, list-books) all verified. Mid-run statement_timeout exercised and recovered via batch split.
• Webhook: deployed, registered, verified live capture from a fresh Roam→Readwise sync. Book-metadata lookup and cache upsert both verified end-to-end (including cache-miss path after DELETE FROM readwise_books WHERE book_id = X).

Follow-ups (not in this PR)

1. Add metadata.book_source to highlight rows — parallel to book_title/book_author/book_category — so source filtering in search_thoughts doesn't need a join. Requires a small UPDATE migration.
2. Optional: top up ~1% export-vs-highlights gap via /api/v2/highlights/ with is_discard=false.

Checklist

• I've read CONTRIBUTING.md
• Each contribution has a README with prerequisites, steps, and expected outcome
• Each metadata.json has all required fields
• Tested on my own Open Brain instance
• No credentials or secrets included

[alanshurafa]

Thanks for the contribution. This is a substantial, well-documented capture integration — a webhook function, a backfill recipe, and a readwise-books schema, all clearly written.

Reviewed at a structural level. The webhook verifies READWISE_WEBHOOK_SECRET before processing, which is the key thing for a public ingestion endpoint, and the readwise-books schema is additive (a new table, no change to thoughts). The cost framing in the README is honest.

One thing for the maintainer to confirm: the README notes the empty-body check short-circuits before the secret check, so Readwise's Test Webhook (which sends an empty body) passes without the secret. That should be a pure no-op responder — worth a quick confirm that an empty-body request cannot do anything beyond returning 200. Recommend maintainer review.

— Alan (community reviewer; non-binding)

[mlava]

Thanks Alan — good thing to flag, and yes, the empty-body path is a pure no-op by design.

The handler short-circuits in two places before the secret check, and neither does anything beyond returning 200:

// GET (infra health checks / probes)
if (req.method === "GET") {
  return new Response("readwise-capture is live", { status: 200 });
}

const bodyText = await req.text();
// Readwise's "Test Webhook" button sends an empty body
if (!bodyText) {
  return new Response("ok (empty body)", { status: 200 });
}

That's the entire empty-body branch — no DB read or write, no embedding call, no Readwise API fetch, no insert. Everything with a side effect (dedup lookup, resolveBook, embedding, the thoughts insert, the increment_book_highlight_count RPC) sits after the body.secret !== READWISE_WEBHOOK_SECRET check, so an unauthenticated caller — empty body or otherwise — can't trigger any work or write any data. The short-circuit exists purely so Readwise's setup-time "Test Webhook" (and plain health probes) can get a 200 without us pretending to process missing data.

So: empty body → static 200, nothing else. Happy to add a code comment making the "no-op" intent explicit if that's clearer for future readers.

[alanshurafa]

Thanks for walking through it — that resolves the flag. Empty body and GET both short-circuit to a static 200 with no DB, embedding, or fetch call, and everything with a side effect sits behind the secret check. Exactly what I wanted to confirm.

A one-line comment on the if (!bodyText) branch, marking it a deliberate no-op for Readwise's Test Webhook, would help the next reader, though it's minor — fine either way. Recommending maintainer review.

— Alan

[mlava]

Added the inline no-op comment in e920507 — thanks for the review, Alan.

[justfinethanku]

Maintainer review: Alan's triage resolved the empty-body webhook question and recommended maintainer review. I updated the branch, added the repo community badge/creator credit convention across the Readwise integration/recipe/schema READMEs, made one markdownlint-only blockquote fix, ran markdownlint locally, and GitHub label/lint checks are passing.