add external backed saves

Author: disnet

CLAUDE.md

@@ -18,7 +18,7 @@ Each package has its own CLAUDE.md with detailed guidance.
 
 A reading app that helps you make sense of what you read. It brings everything you follow — RSS, newsletters, social posts, YouTube, the Atmosphere — into one calm place, gives you a focused surface to read and annotate it, and helps it become understanding through highlights, margin notes, a linkblog, and integrations with sensemaking tools (Semble, Margin).
 
-It's built on the AT Protocol (Bluesky): subscriptions, saved articles, and shares live in the user's Personal Data Server (PDS), so a reader's reading life is portable and outlives any one app. That ownership is the **foundation, not the headline** — lead with reading, not protocol jargon. Read state lives server-side in D1, not the PDS. Features offline support (PWA + IndexedDB), real-time updates, and calm social sharing via linkblogs.
+It's built on the AT Protocol (Bluesky): subscriptions, saved articles, and shares can sync to the user's Personal Data Server (PDS), so a reader's reading life is portable and outlives any one app. That ownership is the **foundation, not the headline** — lead with reading, not protocol jargon. Read state lives server-side in D1, not the PDS — and so, canonically, do saves (the PDS `app.skyreader.feed.saved` record is an opt-in export, see the Copy & Voice accuracy note below). Features offline support (PWA + IndexedDB), real-time updates, and calm social sharing via linkblogs.
 
 ## Design Context
 
@@ -53,8 +53,14 @@ When writing user-facing copy about data ownership, portability, or AT Protocol,
   user's **atproto PDS**, making it backed up, portable to any Atmospheric app, and
   **publicly visible**. Always surface the public-visibility tradeoff.
 - `PDS` is fine as a concrete noun once the framing is established (e.g. "stored on your PDS"), but
-  don't _lead_ with it — lead with the Atmosphere/portability idea. Saves and shares always live on
-  the PDS; only the subscription/feed list is the opt-in choice.
+  don't _lead_ with it — lead with the Atmosphere/portability idea. **Accuracy note:** the canonical
+  store for saves is **D1**, not the PDS. Writing the `app.skyreader.feed.saved` record to the PDS is
+  an opt-in _export_ gated on the same `pds_sync_enabled` flag as subscriptions, and today it only
+  fires for `feed`/`url` saves — `document` and `share` saves never write a PDS save record at all
+  (see `backend/src/routes/saved.ts`). So don't claim in copy that "your saves always live on your
+  PDS"; the honest line is that saves live in Skyreader and become PDS-portable when Atmospheric sync
+  is on. (See [External-backed saves](docs/plans/EXTERNAL_BACKED_SAVES_PLAN.md), which would make a
+  Saved list _be_ a foreign collection.)
 - Keep the voice **calm, terse, reading-first** (per PRODUCT.md). Prefer one short clause over a
   paragraph; let the user act, don't oversell.
 

backend/migrations/0057_external_backed_saves.sql

@@ -0,0 +1,62 @@
+-- External-backed saves: back the Saved list with a Semble/Margin collection.
+--
+-- See docs/plans/EXTERNAL_BACKED_SAVES_PLAN.md. The read path is TWO STORES, not
+-- one (this is the safety property): an ENRICHMENT store (saved_articles) holding
+-- reading work, which a poll NEVER deletes from, and a MEMBERSHIP SNAPSHOT
+-- (backed_collection_members) replaced WHOLESALE from each provably-complete poll.
+-- The Saved set is membership ⋈ enrichment, joined on url_normalized. A botched
+-- poll can only stale the displayed membership (recovered next good poll); it can
+-- never strip a body or a highlight.
+--
+-- Provider-agnostic by design: external_* are foreign at-uri HANDLES, so they hold
+-- a Semble card + collectionLink, OR a Margin community.lexicon.bookmarks.bookmark
+-- + collectionItem (which Phase 0 proved can live in a DIFFERENT repo than the
+-- collection owner — a plain TEXT at-uri stores either).
+
+-- (1) Enrichment store gains the cross-app JOIN KEY. normalizeArticleUrl(url) lives
+--     in feed-proxy today; Phase 1 ports a backend copy for the backfill + write
+--     paths. Existing rows stay NULL until backfilled; SQLite treats NULLs as
+--     distinct in a UNIQUE index, so the index does not collide on un-backfilled rows.
+ALTER TABLE saved_articles ADD COLUMN url_normalized TEXT;
+CREATE UNIQUE INDEX idx_saved_articles_dedup ON saved_articles(user_did, url_normalized);
+
+-- (2) Membership snapshot: the latest PROVABLY-COMPLETE listRecords snapshot of a
+--     backed collection, replaced WHOLESALE per good poll (DELETE the collection's
+--     rows, bulk-INSERT the snapshot — one atomic D1 batch; never row-diffed).
+--     The external_* fields are foreign HANDLES (what to read/delete), not identity.
+--     Reading work never lives here.
+CREATE TABLE backed_collection_members (
+  user_did            TEXT NOT NULL,
+  external_collection TEXT NOT NULL,  -- backing collection at-uri
+  url_normalized      TEXT NOT NULL,  -- join key into saved_articles
+  url                 TEXT NOT NULL,  -- resolved web URL (raw)
+  external_provider   TEXT NOT NULL,  -- 'semble' | 'margin'
+  external_item_uri   TEXT NOT NULL,  -- card / bookmark / note at-uri (may be cross-repo)
+  external_link_uri   TEXT NOT NULL,  -- collectionLink / collectionItem (deleted on unsave)
+  metadata            TEXT,           -- JSON: title/author/etc + canonical at:// (Semble only)
+  PRIMARY KEY (user_did, external_collection, url_normalized)
+);
+CREATE INDEX idx_backed_members_collection ON backed_collection_members(user_did, external_collection);
+
+-- (3) Short-lived unsave tombstones: suppress a just-unsaved URL until a snapshot
+--     confirms the foreign membership is actually gone, so a fire-and-forget
+--     membership delete that hasn't propagated can't be resurrected by the next
+--     wholesale-replace poll. Cleared when a complete poll no longer lists the URL;
+--     a TTL backstop surfaces a permanently-stuck delete. See Phase 4.
+CREATE TABLE backed_unsave_tombstones (
+  user_did            TEXT NOT NULL,
+  external_collection TEXT NOT NULL,
+  url_normalized      TEXT NOT NULL,
+  created_at          INTEGER NOT NULL,
+  PRIMARY KEY (user_did, external_collection, url_normalized)
+);
+
+-- (4) Backing configuration, one per user. 'skyreader' (or NULL) = today's behavior
+--     (app.skyreader.feed.saved export). Otherwise 'semble:<collectionUri>' /
+--     'margin:<collectionUri>'. One backing per ACCOUNT, applied to all saves.
+ALTER TABLE user_settings ADD COLUMN backing TEXT;
+
+-- (5) Last time we polled+replaced the membership snapshot for this user's backing.
+--     One backing per account => one timestamp. Lets the Saved-list-open poll
+--     minute-gate itself instead of hitting the PDS on every GET (epoch ms).
+ALTER TABLE user_settings ADD COLUMN last_backing_poll INTEGER;

backend/src/index.ts

@@ -57,6 +57,7 @@ import {
   handleUpdateSaved,
   handleDeleteSaved,
   handleDeleteSavedByGuid,
+  handleSetBacking,
 } from './routes/saved';
 import { handleExtract } from './routes/extract';
 import { handleGetSettings, handleUpdateSettings } from './routes/settings';
@@ -380,7 +381,7 @@ export default {
         case url.pathname === '/api/saved':
           if (!session) return unauthorizedResponse(headers);
           if (request.method === 'GET') {
-            response = await handleGetSaved(request, env);
+            response = await handleGetSaved(request, env, ctx);
           } else if (request.method === 'POST') {
             response = await handleCreateSaved(request, env, ctx);
           } else {
@@ -394,6 +395,10 @@ export default {
           if (!session) return unauthorizedResponse(headers);
           response = await handleDeleteSavedByGuid(request, env, ctx);
           break;
+        case url.pathname === '/api/saved/backing':
+          if (!session) return unauthorizedResponse(headers);
+          response = await handleSetBacking(request, env);
+          break;
         case url.pathname.startsWith('/api/saved/'):
           if (!session) return unauthorizedResponse(headers);
           if (request.method === 'PATCH') {

backend/src/routes/integrations.ts

@@ -6,7 +6,7 @@ import { SEMBLE_SCOPES, MARGIN_SCOPES } from './auth';
 /**
  * Check if the session has the required scopes for a specific integration
  */
-function hasIntegrationScopes(session: Session, integration: 'semble' | 'margin'): boolean {
+export function hasIntegrationScopes(session: Session, integration: 'semble' | 'margin'): boolean {
   if (!session.grantedScopes) return false;
   const granted = new Set(session.grantedScopes.split(' '));
   const required = integration === 'semble' ? SEMBLE_SCOPES : MARGIN_SCOPES;