Merge pull request #542 from atomantic/feat/creative-ingredients-catalog

Creative Ingredients Catalog: scraps, ingredients, pgvector sync, bible backfill

Author: atomantic

.changelog/NEXT.md

@@ -12,6 +12,8 @@
 
 ## Added
 
+- **[creative-ingredients-catalog-upgrade-path] Catalog deployment + upgrade story.** The bible→catalog backfill is now peer-aware: when a universe arrives via federated sync carrying an `ingredientId` that doesn't exist in the receiving install's catalog, the backfill creates the local catalog row WITH that explicit id instead of minting a new one — so the same logical character keeps the same catalog id across every install in the federation, even when the catalog rows weren't in the original sync payload. A new admin endpoint `POST /api/catalog/migration/rerun` (with `{ force: true }` to ignore the applied-marker) lets users recover from a stuck backfill or fix data on a peer. Catalog DDL ships in both `server/scripts/init-db.sql` (fresh installs via docker-entrypoint-initdb.d or `setup-db.js`/`db.sh setup-native`) AND `server/lib/db.js:ensureSchema()` (idempotent runtime upgrades — runs on every server boot), so existing installs pick up the catalog tables on the first restart post-pull without needing to re-run `init-db.sql`.
+- **[creative-ingredients-catalog] New "Catalog" feature under Create.** A central, Postgres-backed store of typed creative ingredients — characters, places, objects, ideas, scenes, concepts. Paste any creative scrap (a one-line story spark, a scene snippet, a rough short-story draft) into `/catalog/ingest` and the server extracts candidate characters / places / objects via LLM, streams progress as a stage checklist, and shows a review screen where you check the ones to commit. The Catalog page (`/catalog`) lists every ingredient with type-chip filters, debounced search, and a one-click "+ New" inline form for the lighter idea/scene/concept types. Each ingredient has a detail page with source-scrap provenance and an "Appears in" panel that back-links to every universe / series / issue / work referencing it. Federation routes (`GET/POST /api/catalog/sync`) are wired and version-gated via the existing `PORTOS_SCHEMA_VERSIONS` contract (`catalog: 1`); the auto-orchestrator hookup ships in a follow-up. A boot-time migration backfilled 970 ingredients across 10 existing universes (94 characters, 140 places, 736 objects) so the catalog opens populated. **Vector embeddings are provider-agnostic** — a new Embeddings settings tab lets you pick Ollama or LM Studio + a 768-dim model (e.g. `nomic-embed-text`); embeddings auto-apply at ingest when configured and a `/api/catalog/embeddings/backfill` admin endpoint fills missing rows. Per-record sanitizer carries an `ingredientId` field through the round-trip so embedded universe canon stays linked to its catalog row.
 - **Voice coding agent can target a managed app.** When you dispatch a coding task by voice, you can now name a managed app ("fix the failing test in BookLoom") and the agent runs against that app's workspace instead of PortOS itself. The app name is fuzzy-matched, so "book loom", "BookLoom", and "bookloom" all resolve to the same app; if no app matches what you said, the agent refuses with a short list of valid names rather than silently running against PortOS.
 - **[voice-code-agent-status-query] Ask voice how a dispatched coding task is going.** A new voice query — "how's that coding task going?", "status of the agent", "is the agent still working?" — reports each in-flight voice-dispatched coding task with its current phase, target app, elapsed time, and a snippet of the task description. Until now you could only learn the outcome from the completion announcement; this is the mid-task check-in. Also available as "Coding agent status" in the command palette.
 

PLAN.md

@@ -4,6 +4,23 @@ For project goals, see [GOALS.md](./GOALS.md). For completed work, see [.changel
 
 ## Next Up
 
+### Creative Ingredients Catalog — follow-up slices
+
+The catalog backend, schema, federation routes, ingest extraction service, and Catalog/Ingest/Detail UI all landed in one PR (see `docs/plans/2026-05-29-creative-ingredients-catalog.md` for the design record). Three Phase-7-onward slices remain:
+
+- [ ] [catalog-universe-builder-picker] **Add "Pick from Catalog" button to `client/src/pages/UniverseBuilder.jsx` character/place/object panels (~3736 LOC file).** When a catalog ingredient is picked, copy its `payload` into a new embedded canon entry with `ingredientId` stamped, AND post `linkCatalogIngredient(id, { refKind: 'universe', refId, role: 'canon-character'|'canon-place'|'canon-object' })`. Picker component lives at `client/src/components/IngredientPicker.jsx` (already built). Deferred from the initial PR because of the file size — needs targeted exploration of the canon-panel structure first (CanonTab / BibleTab around `UniverseBuilder.jsx:1835` + `:2945`).
+- [ ] [catalog-pipeline-series-cast] **Add a "Cast (from Catalog)" section to `client/src/pages/PipelineSeries.jsx` that surfaces `listCatalogIngredientsForRef('series', seriesId)` and lets the user attach/detach via `IngredientPicker`.** Series records carry no embedded `characters[]` array, so this is purely additive — backed entirely by `catalog_ingredient_refs`. Same pattern applies to `client/src/pages/PipelineIssue.jsx` (refKind `'issue'`) and `client/src/pages/WritersRoom.jsx` work bibles (refKind `'work'`).
+- [ ] [catalog-federation-orchestrator] **Wire the catalog into `server/services/syncOrchestrator.js` (or `server/services/sharing/peerSync.js`, whichever owns memory sync orchestration) so the existing `/api/catalog/sync` + `/api/catalog/sync/apply` routes auto-replicate between peers.** Today the routes exist and are version-gated via `PORTOS_SCHEMA_VERSIONS.catalog = 1`, but no scheduler pulls from peers. Mirror how memory sync is registered (it's the only existing Postgres-backed sync category). Outbound `portosMeta.schemaVersions.catalog` is already stamped server-side. **Universe/series sync today does NOT block on catalog sync** — universes carry their full embedded canon payloads, so a peer-pushed universe still arrives complete; the receiving peer's boot-time backfill walks the freshly-arrived universe and (now, post-`[catalog-universe-builder-picker]`'s peer-reconciliation fix) creates local catalog rows with the original IDs preserved. Direct catalog sync only matters for **orphan ingredients** (idea/scene/concept rows not attached to a universe) and for **post-`[catalog-universe-builder-picker]`** universe-ingredient picker links (the `catalog_ingredient_refs` rows themselves) so the "Appears in" panel populates across peers.
+- [ ] [catalog-ddl-drift-test] **Add a server test that diffs the catalog DDL between `server/scripts/init-db.sql` and the catalog block in `server/lib/db.js:ensureSchema()`.** Both files now duplicate the four table definitions + indexes + trigger functions; a future PR that updates one without the other will leave fresh installs and upgrading installs with different schemas. Lift the catalog DDL into a shared constant array consumed by both, or write a smoke test that parses both files and compares the relevant statements. Same risk exists for the `memories`/`memory_links` DDL but has been tolerated since the memory system landed; catalog inherits it. Surfaced when the user asked about Postgres schema upgrade management (2026-05-29).
+- [ ] [catalog-bundled-universe-push] **Bundle catalog ingredients into universe peer-push payloads.** When peer A pushes a universe to peer B, the push should include the `catalog_ingredients` rows referenced by that universe's embedded canon plus the `catalog_ingredient_refs` rows linking universe→ingredient. Today the embedded canon payloads still replicate (so universes work end-to-end across peers), but the `tags`, `embedding`, `payload.summary`-style enrichments that live ONLY in the catalog row do not — they get re-derived from the embedded entry on the receiver's first backfill, which is a strictly-lossy view. Mirror the `bundleAdditionalKinds` pattern in `server/services/sharing/peerSync.js`. Schema-version gating already covers the new bundle: `RECORD_KIND_SCHEMA_CATEGORIES.universe` would need `['universes', 'catalog']` appended OR the bundling can rely on `cat-ingredient` arriving as its own already-gated kind in the same push. Pairs with `[catalog-federation-orchestrator]`.
+- [ ] [catalog-extraction-idea-scene-concept] **Extend `server/services/catalogExtraction.js` to also extract idea / scene / concept ingredients from a scrap.** Current extractor reuses `extractBible` for character/place/object only (the three storyBible-shaped types). For the other three, ship a single LLM JSON-mode call with a custom prompt; surface the new stages on `catalog:extract:progress`. Until this lands, ideas/scenes/concepts are created manually via the Catalog list's "+ New" button.
+- [ ] [catalog-fts-character-fields] **`catalog_ingredients.search_tsv` (GENERATED ALWAYS) indexes `payload->>description|notes|background|summary` but NOT `physicalDescription` or `personality`** — so a search for "tall" or "introverted" misses bible-backfilled characters whose narrative text lives in those keys. Fix needs a schema migration: DROP COLUMN search_tsv + ADD COLUMN with the expanded expression (Postgres can't ALTER a STORED generated expression), update both `server/lib/db.js` and `server/scripts/init-db.sql`, and bump `PORTOS_SCHEMA_VERSIONS.catalog`. Surfaced by gemini review pass on the catalog PR (2026-05-29).
+- [ ] [catalog-commit-transactional] **`POST /api/catalog/scraps/:id/commit` does N `createIngredient` + N `linkIngredientToSource` writes in a sequential loop with no transaction.** A mid-loop failure (DB timeout, validation throw) leaves a partial commit — some ingredients persisted, some not, and some ingredients without source-link rows. Wrap the loop in `withTransaction` and pass the client down to `createIngredient` / `linkIngredientToSource`. Requires the catalogDB write helpers to accept an optional `client` (currently they call the pool-level `query()` directly). Surfaced by gemini review pass (2026-05-29).
+- [ ] [catalog-scrap-embedding-or-search] **Scraps embed on every create (`POST /api/catalog/scraps`), but the embedding column is never read** — no semantic-search route, no "find similar scraps" UI. Either remove the LLM round-trip on create (and keep the column for future use) OR ship the search endpoint that justifies the cost. Currently every scrap creation burns a 768-dim embedding call for zero search benefit. Surfaced by gemini review pass (2026-05-29).
+- [ ] [catalog-ref-deletion-tombstones] **`catalog_ingredient_refs` uses hard DELETE on unlink — the row vanishes with no `sync_sequence` bump and no tombstone, so peers that already pulled the ref never learn it was removed and their "Appears in" panels stay stale.** Fix needs: add `deleted boolean DEFAULT false` + `deleted_at timestamptz` to `catalog_ingredient_refs` (DDL in both `server/lib/db.js` and `server/scripts/init-db.sql`), switch `unlinkIngredientFromRef` in `server/services/catalogDB.js:430` to soft-delete with `sync_sequence` bump, teach `applyRemoteChanges` in `server/services/catalogSync.js` to apply ref-delete envelopes, and bump `PORTOS_SCHEMA_VERSIONS.catalog`. Pairs with `[catalog-federation-orchestrator]` — also re-surfaced by the codex review pass on the catalog PR (2026-05-29).
+- [ ] [catalog-tests] **Tests for the catalog stack.** `server/services/catalogDB.test.js` (CRUD round-trip — bootstrap follows `server/services/memoryDB.test.js`), `catalogExtraction.test.js` (mock aiToolkit, assert Zod conformance), `catalogSync.test.js` (envelope round-trip + LWW), `server/lib/catalogValidation.test.js` (Zod boundary tests), `server/scripts/migrateBibleToCatalog.test.js` (idempotency), `client/src/pages/Catalog.test.jsx`, `client/src/components/IngredientPicker.test.jsx`.
+- [ ] [catalog-writers-room-version-refs] **Phase 9 (deferred at design time): capture catalog ingredient ids per Writers Room draft version.** Extend the draft save path in `server/services/writersRoom/local.js` to accept `referencedIngredientIds[]`. Add `scanProseForIngredientRefs(text, { universeId?, seriesId?, workId? })` to `catalogExtraction.js` (substring match scoped to refs linked to the target). UI: render referenced ingredients on each version-history chip in `client/src/pages/WritersRoom.jsx`.
+
 _Batch-cleared 2026-05-25: 23 Next Up items shipped together via parallel sub-agents (env-file helper, formatBytes migration, lazy voice UI text, voice long-term-memory routing, voice tool expansion + `ui_describe_visually`, three phosphene generate_ltx2.py hardenings, the peer-sync snapshot-coverage P1 refactor + ephemeralize-then-delete, per-record tombstone-ack clamp, MediaLightbox→MediaImage, global vitest peer mock, videoHistory sync category, reverse-sub UI broadcast, palette Health disambiguation, insertXxxWithId resurrection side-effects, importer tombstone detection, TUI finish-rejection note, mortalLoom errno widen, VideoGen stale-model toast, Universes sidebar grandchildren). See `git log` + `.changelog/NEXT.md`._
 
 - [ ] [mediacard-use-mediaimage-for-syncing-assets] **`MediaCard.jsx` grid thumbnails still use a raw `<img src={previewUrl}>`.** Same peer-sync placeholder/live-swap gap that `[peer-sync-medialightbox-use-mediaimage-for-syncing-assets]` fixed for the lightbox — `client/src/components/media/MediaCard.jsx` (~line 42) doesn't get the "Syncing" placeholder or the `peerSync:asset-arrived` atomic swap. Swap the raw `<img>` for `MediaImage`. Surfaced by that item's cross-check during the batch-clear (2026-05-25); was outside its stated scope.

client/src/App.jsx

@@ -43,6 +43,9 @@ const MediaModels = lazyWithReload(() => import('./pages/MediaModels'));
 const Loras = lazyWithReload(() => import('./pages/Loras'));
 const UniverseBuilder = lazyWithReload(() => import('./pages/UniverseBuilder'));
 const Universes = lazyWithReload(() => import('./pages/Universes'));
+const Catalog = lazyWithReload(() => import('./pages/Catalog'));
+const CatalogIngest = lazyWithReload(() => import('./pages/CatalogIngest'));
+const CatalogIngredient = lazyWithReload(() => import('./pages/CatalogIngredient'));
 const VideoTimeline = lazyWithReload(() => import('./pages/VideoTimeline'));
 const VideoTimelineEditor = lazyWithReload(() => import('./pages/VideoTimelineEditor'));
 const CreativeDirector = lazyWithReload(() => import('./pages/CreativeDirector'));
@@ -251,6 +254,9 @@ export default function App() {
               at `/universes/:universeId`; `new` is the create-mode sentinel
               (UniverseBuilder treats it as no-id → blank draft). Universe ids are
               UUIDs, so `new` can never collide with a real record. */}
+          <Route path="catalog" element={<Catalog />} />
+          <Route path="catalog/ingest" element={<CatalogIngest />} />
+          <Route path="catalog/:type/:id" element={<CatalogIngredient />} />
           <Route path="universes" element={<Universes />} />
           <Route path="universes/new" element={<UniverseBuilder />} />
           <Route path="universes/:universeId" element={<UniverseBuilder />} />