Rework rollback / recovery story around forward-only migrations

[larryro]

Context

PR #1539 locked `tale deploy` to the running CLI's version and intentionally did not lean on `tale rollback` as a recovery path. Reason: Tale's migrations are forward-only — new versions can write data the old binary cannot read, so swapping the binary back without reversing data leaves the system in a half-broken state. Keeping a command that looks like it recovers, but actually corrupts, is worse than not having it.

This issue tracks the proper redesign.

Proposed scope

1. Audit and likely remove `tale rollback`

• Today `tale rollback --version X` accepts any version with no validation (verified in tools/cli/src/commands/rollback.ts) and re-implements its own deploy path.
• Remove the command and its `setPreviousVersion` state-tracking, OR gate it behind a strict precondition (e.g. only allowed when no migrations have run since the previous version).

2. Backup-based recovery as the official supported path

• Automatic volume snapshot in `tale deploy` before `runPendingMigrations` (the only step that mutates data). Snapshot the `STATEFUL_SERVICES` volumes via `docker run --rm -v :/data -v :/backup alpine tar czf /backup/-.tar.gz /data`.
• Rotation policy (keep last N snapshots / N days) to bound disk usage.
• Companion `tale restore ` command — listing, restoring, integrity-checking snapshots.
• Failure behavior: snapshot failure aborts deploy by default; `--skip-backup` to override explicitly.

3. Docs

• Update docs/production-deployment.md to make the recovery story explicit: "recovery = restore snapshot + re-deploy", not "roll back the binary".

Open questions

• Per-project backup directory location and ownership (host path vs. dedicated docker volume).
• How to handle very large stateful volumes — incremental snapshots? Hooks for app-level dumps (Convex export, pg_dump) instead of raw volume tar?
• Should snapshot creation be on `tale deploy` only, or also on `tale start` when migrations run?
• Should existing migrations (`namespace-volumes`, `split-convex`) be retroactively wrapped in the snapshot flow on first run after this lands?

Out of scope (for this issue)

• The CLI/platform version-lock and upgrade availability check — already shipped in fix(cli): pre-flight asset check on upgrade and lock deploy to CLI version #1539.

[yannickmonney]

Design input ahead of implementation (this is needs discussion, so capturing a concrete recommendation rather than landing code blind).

Recommendation: split into three shippable PRs, decide the open questions per-PR

PR 1 — Neutralize tale rollback (low risk, do first)

The current command accepts any version with no validation and re-runs its own deploy path, which is exactly the "looks like recovery, actually corrupts" trap described. Rather than delete outright (some operators may script against it), gate it: refuse unless getPreviousVersion() is set and no migrations have run since that version (compare against the applied-migrations ledger). When refused, print the snapshot-restore instructions instead. This removes the footgun immediately without waiting on the backup system. Drop setPreviousVersion only once PR 2 lands, so we don't strip state the gate relies on.

PR 2 — Pre-migration volume snapshots in tale deploy

• Snapshot STATEFUL_SERVICES volumes immediately before runPendingMigrations (the only data-mutating step), via the alpine tar pattern in the issue.
• Failure = abort deploy by default; --skip-backup to override (logged loudly).
• Rotation: keep last N (default 5) and N days (default 14), whichever is more generous, to bound disk.
• Only snapshot when runPendingMigrations actually has pending work — no-op deploys shouldn't churn disk.

PR 3 — tale restore <snapshot-id> + docs

• tale restore lists/inspects/restores snapshots, verifies the tar integrity (size + checksum sidecar written at snapshot time) before extracting, and refuses to restore into a running stack (stop services first).
• Rewrite docs/production-deployment.md: recovery = restore snapshot + re-deploy the matching version, never "roll back the binary."

Answers to the open questions

• Backup location/ownership: dedicated docker volume (tale-backups), not a host path — keeps it inside the same lifecycle/permission model as the data it protects and avoids host-path UID mismatches across platforms. Allow override via config for operators who want off-box storage.
• Large volumes: start with raw tar (simple, correct). Add app-level hooks (Convex export, pg_dump) as an opt-in backup.strategy later — don't block PR 2 on it. Incremental snapshots are a v2 concern.
• tale start too? Yes — any code path that calls runPendingMigrations must snapshot first. Factor the snapshot into the migration runner's entry, not into deploy specifically, so start and deploy both get it for free.
• Retroactive wrap of existing migrations (namespace-volumes, split-convex): no. Those have already run on existing installs; wrapping them retroactively does nothing for systems past them and complicates the runner. New snapshot flow applies to migrations that run after this lands.

Happy to take PR 1 first since it's self-contained and removes the active footgun. Holding for a 👍 on the split + the tale-backups volume decision before writing code.

[yannickmonney]

Confirming the maintainer's 3-PR split is the right shape, with one prerequisite that has to be settled first.

Blocker to resolve before PR 1: there is no applied-migrations ledger and no runPendingMigrations step in deploy.ts today. PR 1's "refuse rollback when migrations ran since the target version" gate presumes machinery that doesn't exist yet — so either build a lightweight ledger now (record applied migration ids + the version that applied them) or identify the existing signal to gate on.

The split:

1. Guard rail. Gate tale rollback (tools/cli/src/lib/actions/rollback.ts) to refuse when migrations have run since the target version — depends on the ledger decision above.
2. Backup/restore. tale backup / snapshot + tale restore for STATEFUL_SERVICES (db, proxy, convex, sandbox, sandbox-egress — compose/types.ts). Open decision: the tale-backups volume design (where snapshots live, retention, how restore swaps them in).
3. Docs. Rewrite operate/upgrades.md + a new backups-and-restore.md (en/de/fr).

Prerequisite #1539 is merged, so the deploy-path plumbing it added is available. The two open decisions — (a) build the migration ledger now? and (b) tale-backups volume design — gate PRs 1 and 2 respectively; once those are made this is straightforward CLI work (validated against a real tale start/tale deploy cycle).