feat(search): msgvault-grade search — weighted FTS, body backfill, pgvector hybrid semantic search, MCP server

Port msgvault's proven search architecture into Mailflow, end to end:

- Weighted lexical ranking: subject > sender > body (10:4:1 setweight A-D +
  ts_rank_cd), prefix matching as you type, NULL-safe ordering during
  backfill, served by a GIN index on a new trigger-maintained search_fts
  column filled by a resumable drainer (fast-DDL migrations only — no
  boot-blocking table rewrite). Free-text ranks by relevance; filter-only
  queries stay date-ordered. Stopword-safe term matching; quoted phrases.
- Body materialization: provider-gated background IMAP drainer fills
  body_text (apple/yahoo/generic on; Gmail/PurelyMail/Microsoft off per
  their throttle history) with circuit breakers, snippet-indexer
  sequencing, poison-message forward progress, and background_jobs
  progress rows.
- Embeddings (msgvault port): any OpenAI-compatible /v1/embeddings
  endpoint via AI settings (encrypted key, host-validated, masked reads);
  fingerprinted generations (config change => clean rebuild, generations
  never mix); crash-safe scan-and-fill worker with CAS stamps, 24h
  backstop, poison downshift, single-flight lock, activation-on-coverage,
  and config-fingerprint guards; HNSW per-dimension partial indexes; a
  content-change trigger re-embeds late-arriving bodies. Admin settings UI
  with explicit opt-in privacy copy, Test/Build buttons with live
  progress, and local-embeddings guidance — all in 7 locales. Stock
  Postgres still boots clean with semantic disabled.
- Hybrid search: single-query BM25∪ANN reciprocal-rank fusion behind an
  explicit in-input Semantic toggle (lexical stays the default; silent
  lexical fallback while the index builds). Subject boost gated on a live
  BM25 leg; folder scope, negation, and pagination applied to all modes;
  hnsw.ef_search sized for the fused candidate pool. Eval-verified on a
  real 18k-message mailbox (paraphrase Recall@20: lexical 0% -> hybrid
  76%; keyword MRR 0.43, best of all modes).
- MCP: bearer-tokened Streamable-HTTP /mcp endpoint (SHA-256-hashed
  tokens minted in Profile, Origin-validated, per-token rate-limited)
  exposing 12 msgvault-parity tools incl. search_in_message (keyword +
  vector) and semantic_search_messages; every call scoped to the token
  owner's accounts; staged-only soft deletion with a no-enforceable-filter
  guard.

Deployment: compose ships pgvector/pgvector:pg16 (same PG16 major — the
data dir mounts as-is). Run REINDEX DATABASE after the first boot on the
new image: the alpine(musl)->Debian(glibc) collation change can misorder
existing text btree indexes (the app logs a loud warning when it detects
this). Migrations 0035-0041 are fast-DDL, crash-idempotent, boot-safe.
The legacy search_vector column survives until a follow-up removes it
after the search_fts backfill completes on real installs.

Verified: backend 1111 unit tests + gated pgvector integration suites;
frontend 1402 tests, lint/build clean; deployed and exercised against a
real 18k-message mailbox (search, embeddings build ~27 msg/s, hybrid,
MCP round-trips + Origin/rate-limit guards); independent per-phase review
gates plus a full-diff Codex audit and three msgvault-parity/correctness
audits — all findings fixed pre-merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: salmonumbrella

.gitignore

@@ -31,3 +31,5 @@ Thumbs.db
 
 .superpowers/
 output/
+
+# Search-eval query cache — paraphrases of real mail; the harness regenerates it

README.md

@@ -186,6 +186,14 @@ docker compose up -d
 
 To pin to a specific version instead of `latest`, add `MAILFLOW_VERSION=1.9.0` to your `.env`.
 
+> **Upgrading an existing install across the Postgres image change** (`postgres:16-alpine` → `pgvector/pgvector:pg16`): the new image uses a different C library (musl → glibc), which changes text collation order. Reindex once after the switch or text indexes can silently return wrong results — the backend also prints this warning at startup when it detects the mismatch:
+>
+> ```bash
+> docker compose exec postgres psql -U mailflow -d mailflow \
+>   -c 'REINDEX DATABASE "mailflow";' \
+>   -c 'ALTER DATABASE "mailflow" REFRESH COLLATION VERSION;'
+> ```
+
 ---
 
 ## Option B — Build from source

backend/migrations/0035_search_fts.sql

@@ -0,0 +1,55 @@
+-- Weighted full-text search column (search_fts) + version stamp, kept fresh by
+-- a BEFORE trigger. Fast, metadata-only DDL only (README D1): the nullable
+-- columns force no table rewrite on PG16, and the function/trigger are instant.
+-- Pre-existing rows are populated by the resumable drainer (ftsBackfill.js);
+-- the GIN index is built CONCURRENTLY in the separate no-transaction migration
+-- 0037 (a $$-quoted plpgsql body cannot survive the no-transaction ; splitter,
+-- so the trigger and the CONCURRENTLY index MUST live in different files).
+--
+-- The setweight(...) expression MUST stay identical to
+-- lexicalRepo.searchFtsExpr('NEW'); fts_version = 1 matches lexicalRepo.FTS_VERSION.
+-- Idempotent (IF NOT EXISTS / CREATE OR REPLACE / DROP IF EXISTS) because a
+-- crash before the schema_migrations INSERT retries this migration.
+
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS search_fts tsvector;
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS fts_version int;
+
+CREATE OR REPLACE FUNCTION messages_search_fts_refresh() RETURNS trigger AS $$
+BEGIN
+  -- Skip recompute on an UPDATE that changes none of the indexed source
+  -- columns (read/star flag flips, snippet-only writes), so the trigger does
+  -- not tax hot sync UPSERTs; this also lets the backfill's explicit SET win.
+  IF TG_OP = 'UPDATE'
+     AND NEW.subject      IS NOT DISTINCT FROM OLD.subject
+     AND NEW.from_name    IS NOT DISTINCT FROM OLD.from_name
+     AND NEW.from_email   IS NOT DISTINCT FROM OLD.from_email
+     AND NEW.to_addresses IS NOT DISTINCT FROM OLD.to_addresses
+     AND NEW.cc_addresses IS NOT DISTINCT FROM OLD.cc_addresses
+     AND NEW.body_text    IS NOT DISTINCT FROM OLD.body_text
+  THEN
+     RETURN NEW;
+  END IF;
+
+  BEGIN
+    NEW.search_fts := setweight(to_tsvector('english', coalesce(NEW.subject,'')), 'A') ||
+    setweight(to_tsvector('english', coalesce(NEW.from_name,'') || ' ' || coalesce(NEW.from_email,'')), 'B') ||
+    setweight(to_tsvector('english', coalesce(NEW.to_addresses::text,'') || ' ' || coalesce(NEW.cc_addresses::text,'')), 'C') ||
+    setweight(to_tsvector('english', LEFT(coalesce(NEW.body_text,''), 600000)), 'D');
+    NEW.fts_version := 1;
+  EXCEPTION WHEN program_limit_exceeded THEN
+    -- Even with the 600k LEFT cap, a pathologically dense/multibyte body can
+    -- exceed Postgres's ~1MB tsvector limit (SQLSTATE 54000). Never fail the
+    -- row write: leave search_fts NULL so the message still persists and stays
+    -- findable via the ILIKE fallback; the backfill's row-by-row skip stamps it.
+    NEW.search_fts := NULL;
+    NEW.fts_version := NULL;
+  END;
+
+  RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_messages_search_fts ON messages;
+CREATE TRIGGER trg_messages_search_fts
+  BEFORE INSERT OR UPDATE ON messages
+  FOR EACH ROW EXECUTE FUNCTION messages_search_fts_refresh();

backend/migrations/0036_background_jobs.sql

@@ -0,0 +1,19 @@
+-- Generic progress substrate for observable background drainers (first consumer:
+-- the FTS backfill). Plain table, fast DDL. One row per (kind, account); global
+-- jobs use a NULL account_id, COALESCE'd to '' in the unique index so the upsert
+-- has a single conflict target for both cases.
+
+CREATE TABLE IF NOT EXISTS background_jobs (
+  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  kind        VARCHAR(64) NOT NULL,
+  account_id  UUID REFERENCES email_accounts(id) ON DELETE CASCADE,
+  state       VARCHAR(20) NOT NULL DEFAULT 'idle',
+  processed   BIGINT NOT NULL DEFAULT 0,
+  total       BIGINT NOT NULL DEFAULT 0,
+  last_error  TEXT,
+  started_at  TIMESTAMPTZ,
+  updated_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS ux_background_jobs_kind_account
+  ON background_jobs (kind, COALESCE(account_id::text, ''));

backend/migrations/0037_search_fts_index.sql

@@ -0,0 +1,24 @@
+-- no-transaction
+--
+-- GIN index that serves `search_fts @@ tsquery`, plus a partial btree that lets
+-- the backfill drainer (and any "needs backfill" probe) find not-yet-stamped
+-- rows without a full seq scan; it self-prunes as fts_version = 1 fills in.
+-- CONCURRENTLY must run outside a transaction, so these live apart from 0035's
+-- trigger. Idempotent via IF NOT EXISTS (retried if a crash precedes the
+-- schema_migrations INSERT).
+--
+-- DROP ... IF EXISTS before each CREATE: a cancelled or crashed CREATE INDEX
+-- CONCURRENTLY leaves an INVALID index under the target name. On retry, plain
+-- IF NOT EXISTS sees that name and silently skips the create, recording the
+-- migration as done while the scan stays unindexed forever. This file only re-runs
+-- after such a failure — in which case the index is either absent (drop is a no-op)
+-- or invalid (drop removes the dead stub) — so dropping first is safe and cheap.
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_messages_search_fts;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_search_fts
+  ON messages USING GIN (search_fts);
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_messages_fts_stale_v1;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_fts_stale_v1
+  ON messages (date DESC)
+  WHERE fts_version IS DISTINCT FROM 1;

backend/migrations/0038_embed_watermark.sql

@@ -0,0 +1,44 @@
+-- Vector-substrate CAS columns + last_modified trigger (slice 04).
+-- Extension-independent, fast DDL only. NO vector-typed DDL here — the
+-- embeddings/index_generations/embed_watermark/embed_runs tables and the HNSW
+-- index are created by ensureVectorSchema() at startup (README invariant).
+-- Transactional migration (NOT -- no-transaction): the no-transaction runner
+-- splits on ';' and would break the dollar-quoted function below.
+
+-- last_modified: content-change CAS token the embed worker compares to detect a
+-- late-arriving body invalidating a stale subject-only embedding. NOT NULL DEFAULT
+-- now() is metadata-only on PG16 (now() is STABLE → one stored missing-value, no
+-- table rewrite).
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS last_modified TIMESTAMPTZ NOT NULL DEFAULT now();
+
+-- embed_gen: the index generation this row is embedded under. NULL = needs embedding.
+-- Plain BIGINT soft-stamp (no FK): a generation can be retired/deleted while stamps linger.
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS embed_gen BIGINT;
+
+-- Bump last_modified AND clear embed_gen when an embedding-input column changes. The
+-- WHEN clause filters at the C level so unchanged-content UPDATEs (the hot re-sync path)
+-- and stamp-only UPDATEs (the worker setting embed_gen) skip the function entirely.
+-- Clearing embed_gen is what re-surfaces a late-arriving body: a row embedded
+-- subject-only and stamped, whose body later lands (phase-2 drainer / on-open fetch),
+-- has its stamp cleared here so the NULL-only embed scan re-finds it and the idempotent
+-- upsert replaces the stale chunks. The CAS only guards the read→stamp window; this
+-- trigger covers the post-stamp case (Mailflow's late-arriving bodies — msgvault's rows
+-- are immutable after ingest, so it never needed this). Together with createGeneration's
+-- stamp-reset (which handles generation rebuilds: unchanged content, new fingerprint),
+-- the invariant is exact: embed_gen IS NULL ⟺ the row needs embedding.
+CREATE OR REPLACE FUNCTION messages_bump_last_modified() RETURNS trigger AS $$
+BEGIN
+  NEW.last_modified := now();
+  NEW.embed_gen := NULL;
+  RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_messages_last_modified ON messages;
+CREATE TRIGGER trg_messages_last_modified
+  BEFORE UPDATE ON messages
+  FOR EACH ROW
+  WHEN (NEW.subject IS DISTINCT FROM OLD.subject
+        OR NEW.body_text IS DISTINCT FROM OLD.body_text
+        OR NEW.body_html IS DISTINCT FROM OLD.body_html)
+  EXECUTE FUNCTION messages_bump_last_modified();

backend/migrations/0039_embed_pending_index.sql

@@ -0,0 +1,17 @@
+-- no-transaction
+-- Partial index over the embed-scan's steady-state hot predicate. Once a generation
+-- reaches full coverage, the only live rows still needing work are newly-arrived
+-- messages with embed_gen IS NULL, so the 60s scheduler scan (scanForEmbedding) should
+-- touch O(pending), not O(mailbox). Built CONCURRENTLY (hence -- no-transaction) so it
+-- never blocks boot on a large messages table; extension-independent and cheap to
+-- maintain (only the sparse NULL set is indexed). Idempotent (IF NOT EXISTS) because a
+-- crash before the schema_migrations INSERT retries the migration.
+--
+-- DROP ... IF EXISTS before the CREATE: a cancelled or crashed CREATE INDEX
+-- CONCURRENTLY leaves an INVALID index under this name, and plain IF NOT EXISTS would
+-- then silently skip the create on retry — recording the migration as done while the
+-- embed scan stays unindexed forever. This file only re-runs after such a failure, so
+-- the index is either absent (drop is a no-op) or invalid (drop clears the dead stub).
+DROP INDEX CONCURRENTLY IF EXISTS idx_messages_embed_pending;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_embed_pending
+  ON messages (id) WHERE embed_gen IS NULL;

backend/migrations/0040_api_tokens.sql

@@ -0,0 +1,12 @@
+-- MCP API tokens. We store only a SHA-256 hash of the token; the plaintext is
+-- shown to the operator exactly once at mint time and is never recoverable.
+CREATE TABLE IF NOT EXISTS api_tokens (
+  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  user_id      UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+  token_hash   TEXT NOT NULL UNIQUE,
+  name         TEXT NOT NULL,
+  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  last_used_at TIMESTAMPTZ
+);
+
+CREATE INDEX IF NOT EXISTS idx_api_tokens_user_id ON api_tokens(user_id);

backend/migrations/0041_mcp_deletion_batches.sql

@@ -0,0 +1,21 @@
+-- Staged (not executed) MCP deletions. stage_deletion records a batch; a separate
+-- session-authenticated execute step flips messages.is_deleted (soft delete). No
+-- tool ever hard-deletes. Renumbered to 0041 (0040 is api_tokens) per the README
+-- migration-numbering rule.
+CREATE TABLE IF NOT EXISTS mcp_deletion_batches (
+  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  user_id       UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+  description   TEXT NOT NULL DEFAULT '',
+  status        TEXT NOT NULL DEFAULT 'staged',
+  message_count INTEGER NOT NULL DEFAULT 0,
+  created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  executed_at   TIMESTAMPTZ
+);
+
+CREATE TABLE IF NOT EXISTS mcp_deletion_batch_messages (
+  batch_id   UUID NOT NULL REFERENCES mcp_deletion_batches(id) ON DELETE CASCADE,
+  message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
+  PRIMARY KEY (batch_id, message_id)
+);
+
+CREATE INDEX IF NOT EXISTS idx_mcp_deletion_batches_user_id ON mcp_deletion_batches(user_id);