Skip to content

[recipes] Brain backup and export#193

Open
alanshurafa wants to merge 1 commit into
NateBJones-Projects:mainfrom
alanshurafa:contrib/alanshurafa/brain-backup
Open

[recipes] Brain backup and export#193
alanshurafa wants to merge 1 commit into
NateBJones-Projects:mainfrom
alanshurafa:contrib/alanshurafa/brain-backup

Conversation

@alanshurafa
Copy link
Copy Markdown
Collaborator

Summary

Node.js recipe that exports every Open Brain Supabase table to dated local JSON files with streaming pagination.

  • Paginates through PostgREST 1,000 rows at a time; streams rows to disk so large tables never load fully into memory
  • Dated filenames (<table>-YYYY-MM-DD.json) for simple daily retention
  • Prints a row-count and file-size summary per table
  • thoughts is required; optional companion tables (entities, edges, thought_entities, ingestion_jobs, ingestion_items) are backed up only if present. Stock Open Brain installs skip them cleanly instead of erroring.
  • No dependencies beyond Node 18+; config via .env.local

Why

Open Brain users accumulate thousands of thoughts over time but Supabase-side backups require plan-tier access or manual dumps. A zero-dependency Node recipe gives anyone on the free tier a scriptable backup path that runs from cron / Task Scheduler.

The optional-table handling lands specifically so this recipe works against every Open Brain install, not just ones that have also installed the entity-extraction or smart-ingest schemas. Missing companion tables return a PostgREST 404, which the script treats as "skip with a log line" rather than an error.

Part 11 of 12 in the OB1 Alpha Milestone consolidation.

Test plan

  • Run against a stock Open Brain project (only thoughts table) — verify thoughts backs up, optional tables log skipped (table not present)
  • Run against a project that has the optional tables — verify all exports succeed
  • Drop thoughts temporarily → verify the script throws Required table "thoughts" not found
  • Kill mid-run on a large thoughts export → verify no partial/corrupt JSON left behind (next run overwrites cleanly)
  • Verify dated filename rotates correctly across days
  • node --check backup-brain.mjs passes
  • metadata.json passes the gate schema

@github-actions github-actions Bot added the recipe Contribution: step-by-step recipe label Apr 18, 2026
@alanshurafa alanshurafa added area: recipes Review area: recipes review: needs-refresh Branch is stale, conflicted, or needs rebase before review alan-reviewed Reviewed by Alan Shurafa in Community Reviewer role labels May 20, 2026
@alanshurafa
Copy link
Copy Markdown
Collaborator Author

Conflicts with main. Before I rebase, should this stay as-is, be reshaped, or be dropped? I can refresh it quickly if you want to keep it.

Exports Supabase tables to JSON. Hardened from code review: pagination
that terminates correctly, PGRST205-only "table missing" detection,
atomic writes via tmp+rename, an AbortController fetch timeout, and
BOM-tolerant env parsing.
@alanshurafa alanshurafa force-pushed the contrib/alanshurafa/brain-backup branch from 5d67ad3 to 119ac66 Compare May 20, 2026 16:34
@alanshurafa
Copy link
Copy Markdown
Collaborator Author

Rebased onto main — no conflicts now. This updates the existing recipes/brain-backup/ rather than adding it.

For review context: main's copy of this recipe predates its code review. This PR is the reviewed version — terminating pagination (the loop on main checks a stale page-size value), PGRST205-only "table missing" handling, atomic writes via tmp+rename, an AbortController fetch timeout, and BOM-tolerant env parsing. An upgrade over what's on main.

@alanshurafa alanshurafa added review: ready-for-maintainer Community reviewer recommends maintainer review and removed review: needs-refresh Branch is stale, conflicted, or needs rebase before review labels May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

alan-reviewed Reviewed by Alan Shurafa in Community Reviewer role area: recipes Review area: recipes recipe Contribution: step-by-step recipe review: ready-for-maintainer Community reviewer recommends maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant