feat: semantic search via weighted FTS, body backfill, pgvector and MCP server

[salmonumbrella]

Summary

Overhauls search end to end: results ranked by relevance instead of date-only, bodies searchable across the whole mailbox, an optional embeddings pipeline adding a hybrid semantic mode, and a token-scoped MCP endpoint exposing search to agents. Everything costly or privacy-sensitive is off by default; stock Postgres still boots clean with semantic disabled.

Changes

• Weighted lexical ranking — subject > sender > body (setweight A–D + ts_rank_cd), prefix matching as you type, served by a GIN index over a trigger-maintained search_fts column. Backfill runs as a resumable background drainer (fast-DDL migrations only, no boot-blocking rewrite); not-yet-backfilled rows fall back to the previous query path. Filter-only queries stay date-ordered; quoted phrases supported.
• Body materialization — a provider-gated background IMAP drainer fills body_text (apple/yahoo/generic on; Gmail/PurelyMail/Microsoft off given their throttle history), with circuit breakers, poison-message forward progress, and background_jobs progress rows.
• Optional semantic search — explicit opt-in, against any OpenAI-compatible /v1/embeddings endpoint (key encrypted at rest, host-validated, masked on read). Fingerprinted generations (config change ⇒ clean rebuild), a crash-safe fill worker, per-dimension HNSW partial indexes, re-embedding of late-arriving bodies. Query side is hybrid BM25∪ANN reciprocal-rank fusion behind an in-input Semantic toggle — lexical stays the default, with silent fallback while the index builds. Settings UI with explicit privacy copy and live Test/Build progress, in all 7 locales.
• MCP server — bearer-tokened Streamable-HTTP /mcp endpoint (SHA-256-hashed tokens minted in Profile, Origin-validated, rate-limited) exposing 12 tools including semantic_search_messages and search_in_message. Every call is scoped to the token owner's accounts; deletion is staged-only.

Evals on a real 18k-message mailbox: paraphrase Recall@20 goes 0% (current search) → 76% (hybrid); keyword MRR 0.43, the best of all modes tested.

Deployment notes

• compose moves postgres:16-alpine → pgvector/pgvector:pg16 (same PG16 major; the data dir mounts as-is). Run REINDEX DATABASE once after the first boot: the musl → glibc collation change can misorder existing text btree indexes (the app logs a loud warning when it detects this).
• Migrations 0035–0041 are fast-DDL, crash-idempotent, and boot-safe. The legacy search_vector column stays until a follow-up removes it once the search_fts backfill completes on real installs.
• Rebased on top of fix: cap body tsvector at 600k chars so an oversized email can't crash search #281 — that cap lives on in search/lexicalRepo.js (same constant, equivalent tests).
• One new backend dependency: @modelcontextprotocol/sdk.

Testing

• Backend: 1118 passed / 43 skipped (pgvector integration suites, gated on a live DB), lint clean (--max-warnings 0)
• Frontend: 1447/1447, lint clean, production build clean
• Exercised end-to-end against a real 18k-message mailbox: search flows, embeddings build (~27 msg/s), hybrid mode, MCP round-trips including the Origin and rate-limit guards

---

Contributor License Agreement

By submitting this pull request I confirm that:

• I have read and agree to the Contributor License Agreement.
• My contribution is my own original work (or I have identified any
  third-party material and confirmed it is compatible with the CLA).
• I have the right to submit this contribution under the terms of the CLA.

🤖 Generated with Claude Code

[salmonumbrella]

@maathimself this one is interesting. look forward to your thoughts.

[maathimself]

This is great, the security, scoping, and provider gating are all done right.

My one ask before merging: could you split it? Right now four independent features land in one 14.7k-line squash commit, and they have very different risk profiles:

1. Weighted FTS ranking + searchable bodies (high value, no infra change)
2. Semantic embeddings (pulls in the pgvector image switch, the collation/REINDEX step, and external email-text egress)
3. MCP server (new authed endpoint + new dependency)

The keyword-search improvements benefit everyone with near-zero risk, but they're currently coupled to the database image change and a big new surface most users won't enable. Split into separate PRs, (1) can land almost immediately, and (2)/(3) can be reviewed and opted into on their own. It also makes each far easier to roll back or bisect.

[salmonumbrella]

Done — split as requested:

1. feat(search): weighted FTS ranking + searchable message bodies #294 — weighted FTS ranking + searchable bodies (no infra change, no new deps; ready for review)
2. feat(search): optional semantic embeddings with hybrid ranking (pgvector) #295 — semantic embeddings (pgvector image switch, collation/REINDEX step, opt-in egress; draft, stacked on feat(search): weighted FTS ranking + searchable message bodies #294)
3. feat(mcp): bearer-tokened MCP server exposing search to agents #296 — MCP server (new authed endpoint + @modelcontextprotocol/sdk; draft, stacked on feat(search): optional semantic embeddings with hybrid ranking (pgvector) #295)

The split is a re-partition of this PR's reviewed tree, not a rewrite: with all three applied, the result is byte-identical to this PR's head (3773150), modulo a rebase onto current main (one test-file overlap with the IMAP connect-backoff change, resolved by keeping both test groups). Each split PR passes the full backend/frontend suites, lint, and production build independently; the drafts will be marked ready and rebased as their predecessors merge.

Happy to close this one in favor of the stack whenever you prefer.