Move loop catalog to database (#61)

Author: mberman84

.github/workflows/ci.yml

@@ -33,30 +33,11 @@ jobs:
       - name: Install Worker dependencies
         run: npm ci --prefix worker
 
-      - name: Verify generated artifacts
-        run: |
-          node scripts/build-skill-catalog.mjs
-          node scripts/build-loop-pages.mjs
-          node scripts/build-social-images.mjs
-          if [[ -n "$(git status --porcelain --untracked-files=all)" ]]; then
-            git status --short
-            git diff
-            exit 1
-          fi
-
       - name: Validate site and skill sources
         run: |
-          node --check scripts/build-skill-catalog.mjs
-          node --check scripts/build-loop-pages.mjs
-          node --check scripts/build-social-images.mjs
-          node --check scripts/audit-seo-geo.mjs
-          node --check scripts/loop-data.mjs
-          node --check scripts/validate-loop-data.mjs
           node --check site/script.js
-          node scripts/audit-seo-geo.mjs
           node scripts/check.mjs
           python3 -m json.tool site/.herenow/data.json >/dev/null
-          python3 -m json.tool site/catalog.json >/dev/null
           python3 -m json.tool scripts/seo-geo-query-benchmark.json >/dev/null
           git diff --check
 

.gitignore

@@ -2,6 +2,7 @@
 /.herenow/
 site/.herenow/state.json
 node_modules/
+.wrangler/
 __pycache__/
 *.py[cod]
 /.playwright-mcp/

AGENTS.md

@@ -2,50 +2,49 @@
 
 ## Adding or editing loops
 
-- Treat `scripts/loop-data.mjs` as the canonical SEO/GEO content catalog for
-  every public loop.
-- Keep the matching searchable row in `site/index.html` aligned with the
-  catalog entry, including title, prompt, attribution, link, and visible
-  count.
-- Every loop must have a stable slug, unique search title and description,
-  contributor attribution, published and modified dates, practical context,
-  verification criteria, and related-loop links.
-- After changing the catalog or homepage rows, run
-  `node scripts/build-skill-catalog.mjs` and
-  `node scripts/build-loop-pages.mjs`, capture the versioned page screenshots,
-  and then run `node scripts/build-social-images.mjs`. Commit the skill catalog,
-  public Markdown/JSON catalogs, screenshots, generated detail pages,
-  `site/sitemap.xml`, and `site/feed.xml`.
-- Capture the homepage and every loop page in the light theme at 1200 x 630
-  using the versioned filenames in `site/assets/social/`. Before recapturing
-  published artwork, bump `site.socialImageVersion` in
-  `scripts/loop-data.mjs`; the social-image builder refuses to replace a path
-  already present in `HEAD`.
-- Preserve older versioned social cards so links that already use them keep
-  their artwork. Remove an old card only as an explicit cleanup.
-- Run the full repository checks before committing:
+- The production catalog database is the source of truth for public loops.
+  The current Git tree holds application code and the content-free site shell.
+  Do not commit published loop records, bootstrap data, generated loop pages,
+  catalogs, feeds, sitemaps, or offline catalog fallbacks. Legacy public
+  records remain in pre-migration Git history intentionally; do not rewrite
+  shared history as part of routine catalog work.
+- Publish a reviewed loop from a JSON file outside the repository with:
+
+  ```bash
+  LOOP_PUBLISH_TOKEN=... \
+    npm --prefix worker run loop:publish -- /path/to/loop.json
+  ```
+
+  Use `worker/examples/loop.json` as the record template. The command validates
+  the complete record before writing it, and the Worker records every revision.
+- Every loop must have a stable slug, unique number, search title and
+  description, contributor attribution, published and modified dates,
+  practical context, verification criteria, category, keywords, and valid
+  related-loop slugs.
+- Do not hand-edit the homepage, detail pages, catalogs, feed, sitemap, or skill
+  content when publishing a database record. The Worker renders those public
+  surfaces from the same record. New loops use the shared social card unless a
+  reviewed HTTPS `socialImageUrl` is supplied.
+- Keep bootstrap and backup exports outside the repository with owner-only
+  permissions. The one-time bootstrap command requires an explicit private
+  file path; routine recovery exports use `npm --prefix worker run loops:export`.
+  Restore an export only into a fresh empty catalog with
+  `npm --prefix worker run loops:restore`; never overwrite a live catalog.
+- Changes to the site shell, Worker, schema, or renderers still go through
+  GitHub. Run the full repository checks before committing those code changes:
 
   ```bash
-  node scripts/build-skill-catalog.mjs
-  node scripts/build-loop-pages.mjs
-  node scripts/build-social-images.mjs
-  node --check scripts/audit-seo-geo.mjs
-  node --check scripts/build-social-images.mjs
   node --check site/script.js
-  node --check scripts/build-loop-pages.mjs
-  node --check scripts/loop-data.mjs
-  node --check scripts/validate-loop-data.mjs
-  node scripts/audit-seo-geo.mjs
   node scripts/check.mjs
   npm --prefix worker run check
   python3 -m json.tool site/.herenow/data.json >/dev/null
   python3 -m json.tool scripts/seo-geo-query-benchmark.json >/dev/null
   git diff --check
   ```
 
-- Do not add a loop if the checks report drift between the homepage, source
-  catalog, live catalogs, installable skill fallback, generated pages,
-  structured data, sitemap, or feed.
+- Do not publish a loop unless its public homepage row, detail page,
+  `catalog.json`, `catalog.md`, sitemap, and feed all read back from production
+  with the expected slug and modified date.
 
 ## Protected forms
 
@@ -85,6 +84,7 @@ npm exec -- wrangler secret put TURNSTILE_SECRET_KEY
 npm exec -- wrangler secret put TURNSTILE_HOSTNAMES
 npm exec -- wrangler secret put HERENOW_API_KEY
 npm exec -- wrangler secret put HERENOW_SITE_SLUG
+npm exec -- wrangler secret put LOOP_PUBLISH_TOKEN
 npm run deploy
 ```
 
@@ -123,10 +123,18 @@ curl -sS "https://here.now/api/v1/publishes/{slug}/data/weekly_signups?limit=50"
   active deployment, then fetch and fast-forward again before selecting the
   deployment revision.
 - Hold the lock through here.now finalize and production verification.
-- Deploy and verify the form Worker before publishing a site revision that
-  changes Site Data form collections to owner-only.
-- Verify both `https://signals.forwardfuture.ai/loop-library/` and the backing
-  here.now Site before reporting success.
+- Deploy and verify the Worker before publishing a site revision that changes
+  Site Data form collections, catalog storage, or database-backed rendering.
+- For the initial database cutover, deploy the Worker, import the reviewed
+  private bootstrap bundle, verify all canonical database surfaces, and only
+  then deploy the content-free here.now shell. Never publish the empty shell
+  before the database catalog is active.
+- The exact Worker routes at `signals.forwardfuture.ai/loop-library` and
+  `signals.forwardfuture.ai/loop-library/*` render database content and pass
+  site-shell assets through to the explicit `PUBLIC_ORIGIN_URL` here.now
+  hostname. Update that variable if the backing Site changes. Verify the
+  canonical URL for database content and the backing here.now Site for the
+  static shell before reporting success.
 - After a production content deployment, submit
   `https://signals.forwardfuture.ai/loop-library/sitemap.xml` in Google Search
   Console and Bing Webmaster Tools. Verify that the custom domain's root

README.md

@@ -4,7 +4,7 @@ Loop Library has two separate but related parts in this repository:
 
 | Part | What it is | Where it lives |
 | --- | --- | --- |
-| **Loop Library website** | The public catalog where people and agents can browse published loops, read them, and copy their prompts. No installation is required. | [Live website](https://signals.forwardfuture.ai/loop-library/) · source in [`site/`](site/) and [`scripts/loop-data.mjs`](scripts/loop-data.mjs) |
+| **Loop Library website** | The public catalog where people and agents can browse published loops, read them, and copy their prompts. No installation is required. | [Live website](https://signals.forwardfuture.ai/loop-library/) · shell in [`site/`](site/), database and rendering in [`worker/`](worker/) |
 | **Loop Library skill** | An optional installable guide that helps an AI agent find, audit, repair, adapt, or design loops through conversation. It uses the website's live catalog when recommending published loops. | source in [`skills/loop-library/`](skills/loop-library/) |
 
 The website is the library; the skill is a companion way to work with it. You
@@ -185,6 +185,63 @@ available under the [MIT License](LICENSE).
 <details>
 <summary>Notes for maintainers</summary>
 
+### Publish a loop
+
+Public loops are stored in the catalog database attached to the Cloudflare
+Worker. Publishing a reviewed loop does not require a GitHub commit or a static
+site deployment.
+
+Copy `worker/examples/loop.json` somewhere outside the repository, fill in the
+record, and run:
+
+```bash
+LOOP_PUBLISH_TOKEN=... \
+  npm --prefix worker run loop:publish -- /path/to/loop.json
+```
+
+The command validates the record and publishes the homepage row, detail page,
+JSON/Markdown/plain-text catalogs, feed, and sitemap from the same database
+write. Use `--draft` to save a non-public record or `--archive` to remove a
+record from public responses without deleting its revision history.
+
+The first database-backed release needs one import from the private migration
+bundle. Loop records and bootstrap data are intentionally not committed to
+GitHub:
+
+```bash
+LOOP_PUBLISH_TOKEN=... \
+  npm --prefix worker run loops:import -- /private/path/bootstrap.json
+```
+
+Set a long random `LOOP_PUBLISH_TOKEN` as a Worker secret. The catalog uses a
+SQLite-backed Durable Object and keeps an append-only revision for every
+publish. The reviewed bootstrap digest is enforced before the database can be
+activated.
+
+Create a private backup of the current database with:
+
+```bash
+LOOP_PUBLISH_TOKEN=... \
+  npm --prefix worker run loops:export -- /private/path/catalog-backup.ndjson
+```
+
+Restore that snapshot only into a fresh, empty catalog database:
+
+```bash
+LOOP_PUBLISH_TOKEN=... \
+  npm --prefix worker run loops:restore -- /private/path/catalog-backup.ndjson
+```
+
+Bootstrap and backup files must be owner-only (`chmod 600`). Exports include
+drafts, archived records, and complete revision history; keep them outside the
+repository.
+
+The current Git tree contains the site shell and rendering code, but no
+published loop records, generated loop pages, catalogs, feed, sitemap, or
+offline catalog fallback. The legacy catalog and source-attribution metadata
+were already public and intentionally remain in pre-migration Git history;
+this migration does not rewrite repository history or disrupt existing clones.
+
 ### Preview locally
 
 ```bash
@@ -197,16 +254,7 @@ Then open `http://localhost:4173`.
 
 ```bash
 npm ci --prefix worker
-node scripts/build-skill-catalog.mjs
-node scripts/build-loop-pages.mjs
-node scripts/build-social-images.mjs
-node --check scripts/audit-seo-geo.mjs
-node --check scripts/build-social-images.mjs
 node --check site/script.js
-node --check scripts/build-loop-pages.mjs
-node --check scripts/loop-data.mjs
-node --check scripts/validate-loop-data.mjs
-node scripts/audit-seo-geo.mjs
 node scripts/check.mjs
 npm --prefix worker run check
 python3 -m json.tool site/.herenow/data.json >/dev/null
@@ -215,7 +263,7 @@ git diff --check
 ```
 
 Read [AGENTS.md](AGENTS.md) before editing loops or publishing the site. It
-contains the source-of-truth rules for generated files, form security, and
-clean-main deployments.
+contains the source-of-truth rules for database publishing, generated
+responses, form security, and clean-main deployments.
 
 </details>