From 8b0c02af74484875a9b9fd072a0d0abfd948a6a2 Mon Sep 17 00:00:00 2001 From: Andrei Date: Sun, 19 Jul 2026 19:10:35 +0100 Subject: [PATCH] docs: park the file-level backup design drafts (do not merge) Design work from a pass that was not taken forward: hosted v1 uses a control-plane-side provider snapshot instead, which needs no OS-side code. Parked rather than dropped because the design is the destination once the provider snapshot cap or whole-disk granularity starts to bite, and because the pass surfaced several findings that are true regardless of which backup mechanism ships -- two doc/code drifts, the unparsed data_volumes field that also blocks the specced pre-update snapshot, and the brain-state identity split that makes a naive restore fail silently. docs/drafts/ is deliberately not one of the three doc homes; this branch is not meant to merge. See docs/drafts/backup-file-level/README.md. --- docs/drafts/backup-file-level/BACKUP.md | 219 ++++++++++++++++++ .../ENVIRONMENT-box-identity-section.md | 36 +++ docs/drafts/backup-file-level/README.md | 45 ++++ .../drafts/backup-file-level/issue-1-specs.md | 80 +++++++ .../backup-file-level/issue-2-box-identity.md | 37 +++ .../backup-file-level/issue-3-backup-agent.md | 88 +++++++ 6 files changed, 505 insertions(+) create mode 100644 docs/drafts/backup-file-level/BACKUP.md create mode 100644 docs/drafts/backup-file-level/ENVIRONMENT-box-identity-section.md create mode 100644 docs/drafts/backup-file-level/README.md create mode 100644 docs/drafts/backup-file-level/issue-1-specs.md create mode 100644 docs/drafts/backup-file-level/issue-2-box-identity.md create mode 100644 docs/drafts/backup-file-level/issue-3-backup-agent.md diff --git a/docs/drafts/backup-file-level/BACKUP.md b/docs/drafts/backup-file-level/BACKUP.md new file mode 100644 index 0000000..24a2727 --- /dev/null +++ b/docs/drafts/backup-file-level/BACKUP.md @@ -0,0 +1,219 @@ +# malmo Backup + +> Working spec. What a hosted box backs up, how it is captured consistently, and how it is restored onto a fresh box. Resolves `NEXT.md` # Backup architecture shape and absorbs `ENVIRONMENT.md` # Logical export / restore bundle, which named this doc as its likely home. Companion to `STORAGE.md` (the on-disk layout this reads), `APP_LIFECYCLE.md` (the per-instance directory and the no-named-volumes rule), `APP_MANIFEST.md` (`storage.data_volumes` / `cache_volumes`), `SERVICE_PROVISIONING.md` (managed-service dumps), `AUTH.md` (credentials), and `ENVIRONMENT.md` (the hosted profile this ships in). + +## Scope + +**Scheduled off-site backup is a `hosted` capability.** On hosted, malmo runs the box, so malmo carries the custody obligation: a tenant hands over their photos, documents, and app state and gives up the hardware-ownership half of the pitch (`ENVIRONMENT.md` # Threat model). Custody without off-site backup is the worst version of that trade — one failed volume and the data is gone. + +**On `appliance` there is no scheduled backup in v1.** No destination, no credentials, no timer. An appliance box with a single drive and no parity (SnapRAID is deferred, `STORAGE.md` # Levels of complexity) has no protection against drive failure, and the spec says so rather than leaving it to be inferred. A user-configured destination is tracked in `NEXT.md` # Appliance backup destination — it is a real design problem (whose credentials, entered where, in a UI built for non-technical users) and not a subset of this work. + +What the appliance does inherit is everything below that is not the push agent: the backup set, the consistency rules, and the restore transaction. Because the restored artifact is a **logical bundle** rather than a disk image, it is also the vehicle for the portability promise — "it's your OS, move it home whenever you want" (`ENVIRONMENT.md` # Logical export / restore bundle). A disk image cannot serve that: the two profiles' images differ (LUKS, LAN, mDNS present on one, absent on the other), so a clone of a hosted VM will not boot on a laptop. **Restoring onto an appliance is not a v1 path**, but the bundle is designed so it can become one without a format change. + +## What is backed up + +### Roots, minus an exclude list + +The set is defined as **roots with exclusions**, not as an enumerated list of paths. This is the most important choice in the spec. + +An enumerated include list would have to name the use-case folders, and `STORAGE.md` # What apps and users actually see states that users **may rename, delete, or add folders**. An include list therefore silently omits anything a user creates — producing a backup that looks healthy and is missing data, discovered only at restore. An exclude list fails in the opposite direction: an unanticipated path is backed up, costing storage rather than data. + +**The rule: take the roots and subtract. Anything not explicitly excluded is included.** + +**On `hosted` there are three roots**, because the profile has no bind-mount tree. `ENVIRONMENT.md` # Storage (hosted) collapses the appliance layout: no mergerfs, no add/eject, and deliberately **one volume for everything** — so `/home` and `/var/lib/malmo` are real directories on the root filesystem, not views onto `/srv/malmo/`. + +| Root | Holds | +|---|---| +| `/home//` | user content | +| `/var/lib/malmo/` | brain SQLite, app instances, managed-service state | +| `/srv/malmo/shared/` | household-shared content (`internal/lifecycle/lifecycle.go` sets this root on both profiles) | + +**On `appliance` the same three logical roots exist as one tree** — `/srv/malmo/{home,state,shared}`, bind-mounted outward (`STORAGE.md` # Data drive(s)). There, the rule is to back up the underlying tree rather than the bind-mount views, or the same bytes are traversed twice under two names. + +**The bundle therefore records logical roots, not absolute paths.** The same photo is `/home/cindy/Photos/…` on hosted and `/srv/malmo/home/cindy/Photos/…` on an appliance. A bundle keyed to absolute paths can only ever restore onto the profile it came from, which forecloses the portability promise. Tagging each root by role — `user-content`, `state`, `shared` — costs nothing now and keeps the appliance path open. + +### The run must refuse to snapshot a broken box + +A run that captures an empty or drastically incomplete tree and records it as a success is worse than a failed run: with retention, good snapshots are eventually replaced by empty ones. + +The appliance has a purpose-built mechanism for the version of this caused by bind mounts racing their parent — `STORAGE.md` # Storage canary. **It does not exist on hosted**, which has no bind mounts to race and no physical drive to lose (`ENVIRONMENT.md` # Storage: "no mergerfs, no add/eject, no canary"). Hosted needs its own precondition, checked before anything is written to the repository: + +- every expected root is present and non-empty; +- the captured set is not implausibly smaller than the previous snapshot. + +Failure aborts the run and reports, rather than committing a snapshot. + +### Excluded, and why + +Paths below are given relative to the state root (`/var/lib/malmo/` on hosted). **Verify every one against the code before implementing** — `STORAGE.md`'s layout diagram is stale: it shows `brain/state.db` and `managed-services/`, while `cmd/brain/main.go` opens `/malmo.db` and `internal/lifecycle/services.go` writes services under `/services/`. An exclude keyed to a name that no longer exists silently fails to exclude, which is how live databases end up file-copied into a backup. + +| Excluded | Why | +|---|---| +| `instances/*/snapshots/` | pre-update tars (`UPDATES.md` # Pre-update snapshot), regeneratable and by definition superseded | +| every declared `cache_volumes` subtree | regenerable by construction — thumbnails, transcodes, downloaded models. The rule is the declared data tree **minus** the declared cache subtrees, since a cache commonly nests inside a data volume | +| the **live data directories of managed services** (under `services/`) | captured as logical dumps instead. A file copy of a running database is a corrupt copy | +| `catalog-cache/` | re-fetchable from the catalog | +| `seed.json` | carries the enrollment credential; a restore target is issued a fresh seed anyway, so including it only widens the blast radius of a leaked repository key | +| the restic local cache | reconstructable from the repository | +| `lost+found`, sockets, device nodes | not data | + +The OS and Docker images are excluded by construction, since neither lives under a root. The OS is a reproducible `mkosi` image and the image layers are re-pullable by digest, with the digests authoritative off-box in the catalog (`APP_STORE.md`). + +### Added from outside the roots + +Three things must be captured that live outside the roots: + +- **Logical dumps of every managed service**, per app (below). +- **Credentials** — the `/etc/shadow` entries for malmo users (# Credentials). +- **A manifest of the run itself** — the roots captured and their roles, the OS image identity, and the bundle's schema version, so a restore can refuse a bundle it does not understand rather than half-applying it. + +## Databases get dumped; files get copied + +A backup that walks a filesystem is not atomic. Anything live and database-shaped is captured as a logical dump instead of a file read. This applies to two things today and to anything database-shaped added later. + +### The brain's SQLite + +A live SQLite file copied while the brain is writing is a corrupt copy. It is captured with `VACUUM INTO` to a staging path, and the staged file is what enters the backup. + +### Managed services + +`SERVICE_PROVISIONING.md` provisions shared managed services as **brain-owned compose projects** (`malmo-svc-postgres-15`, `malmo-svc-mysql-8-0`, `malmo-svc-valkey-8`), each with its own data volume shared across every app that asked for that engine. Postgres, MySQL, MariaDB, and Valkey are built and shipped. + +Their data does **not** live under `/var/lib/malmo/instances//`. An app using a managed Postgres keeps its config and bind-mounted files in the instance directory and its rows in the shared service. Copying only instance directories restores such an app with its settings intact and its database empty — a backup that looks complete and silently is not, with nothing signalling it until a restore. + +`SERVICE_PROVISIONING.md` # At backup already states the intended shape: "Dump is included in the app's backup archive alongside its data volumes." Concretely: + +- A **logical dump**, not a file copy of the service's data volume. +- **Per app, not per engine**, so a restore can be scoped to one app without touching another's rows. +- Run through the **same throwaway one-shot client container the brain already uses to provision** — `docker run --rm --network malmo-svc-- --env-file /.env ` — so the brain still never joins a service network (`DECISIONS.md` 2026-06-02, 2026-06-15). + +`SERVICE_PROVISIONING.md` lists managed-service backup/restore as deferred and **gated on the backup design**. This spec is that gate. + +## Credentials + +`AUTH.md` makes PAM the source of truth for passwords: the brain holds no password hash and delegates verification to the host-agent. The hashes live in `/etc/shadow`, outside the backup root. Without them, a restored box brings back every user account in a state where nobody can log in. + +**v1 backs up the `/etc/shadow` entries for malmo users.** That is the whole credential set on hosted. + +- The user logs in after a restore with the password they already had. Nothing to communicate, nothing to reset, no admin handling other people's passwords. +- **Samba is not part of this on hosted.** `ENVIRONMENT.md` # Access & files states SMB is not shipped there — no LAN to serve. Samba keeps its own NT hashes in `tdbsam`, synced from `passwd` only on a live change (`AUTH.md` # Samba password backend), so a bundle restored onto an **appliance** would need it captured too or SMB breaks while dashboard and SSH work. Appliance restore is not a v1 path; the bundle schema should leave room for the field rather than pretending the profiles match. +- **This puts password hashes in the backup**, a real widening of exposure but not a new trust concession: the repository is client-side encrypted, and `ENVIRONMENT.md` # Threat model already places malmo-operated infrastructure inside the hosted trust boundary with operator-escrowed keys. +- It couples the bundle to a host file format, which is the argument for revisiting it once the successor below is available. + +**The named alternative, for when the box→cloud channel exists:** the user authenticates at malmo.network, so the control plane can push a password to a restored box over that channel and keep hashes out of the backup entirely. It is strictly better and it is not v1, because it depends on a second workstream landing and adds a credential-push operation with its own authorization rules. + +## Consistency: what is guaranteed and what is not + +Three tiers, labelled honestly rather than implied: + +1. **Managed-service data is consistent by construction.** A logical dump is a point-in-time snapshot of an app's rows. This covers the population most exposed to tearing. +2. **The brain's own state is consistent by construction**, via `VACUUM INTO`. +3. **Everything else is crash-consistent.** User content and an app's bind-mounted `data/` are captured as they lie. For user content this is close to harmless — media files are written once and rarely rewritten in place. For an app that **bundles its own database** inside `data/`, which malmo cannot see into, a restore is equivalent to restoring from a machine that lost power at that instant. Well-built databases recover from exactly that; not every bundled one will. + +**Lifecycle hooks stay deferred.** `DECISIONS.md` 2026-05-13 deferred the manifest's `hooks:` block noting that every use case named was tied to managed services or backups. Backup arriving is that trigger, and the answer is still no for v1: the managed-service dump path removes the tearing risk for the apps that matter most without asking any author to write anything, which is the better trade for a catalog whose authors are mostly upstream projects that have never heard of malmo. When hooks return as one-shot container images (`NEXT.md` # Hooks — concrete shape for return), a `pre_backup` hook refines tier 3 for apps that ship one — the same relationship `pre_update` has to the pre-update snapshot. + +**A filesystem freeze is rejected, not deferred.** It needs a snapshotting volume manager the locked ext4 layout does not provide (`DECISIONS.md` — ext4 + LUKS, not ZFS). + +The measure of this feature is successful **restores**, not successful uploads. + +## The push agent (hosted) + +### Box-initiated + +Nothing can reach into a hosted box: the lean image installs no `openssh-server`, and the control plane only talks to a box at provision time (`ENVIRONMENT.md` # How the profile is realized). Backup is therefore box-initiated push on a `malmo-backup.timer` in the quiet window, jittered so a fleet does not stampede its destination, with the outcome reported back so a silently-failing backup is visible rather than found at restore (`ENVIRONMENT.md` # Box identity). + +### Who actually runs a backup + +The timer is a host unit, but two of the three capture steps are **not host operations**, so the agent cannot be a standalone host script: + +- `VACUUM INTO` acts on the brain's live SQLite, which the brain owns. +- Managed-service dumps need the service `.env`, the service network, and `docker run --rm` — and the brain reaches Docker through the socket-proxy (`CONTROL_PLANE.md` # Docker socket exposure), which is the only sanctioned path. A host script bypassing it re-opens a boundary `DECISIONS.md` 2026-06-14 closed deliberately. + +The brain also runs containerized with `MALMO_STATE_DIR`, so brain-visible paths and host paths are not the same strings — a detail that has to be explicit wherever the two meet. + +The workable split is **the timer as the clock, the brain as the executor**: the host unit triggers a run, the brain performs the dumps and hands back a staged, consistent set, and the push happens against paths the host can read. The exact seam — a host-agent job, a brain endpoint, or a brain-internal scheduler with the timer only as a wakeup — is the first thing the build issue must settle, because it determines where the restic invocation lives and which side holds the repository credential. + +### Engine: restic + +- Client-side AES-256, so nothing readable leaves the box. +- Content-addressed deduplication, so a daily run over a mostly-unchanged home directory transfers almost nothing. +- A single static binary, shipped via `mkosi.extra` so it adds no apt package and the lean-image manifest guard still passes (`ENVIRONMENT.md` # How the profile is realized). + +**Every snapshot is logically full and physically incremental.** A run walks the whole set and stores only chunks the repository does not already hold, but the snapshot it records is a complete view of the box at that moment. There is no full/incremental chain and no base to reconstruct from, which has three consequences worth stating: every snapshot restores standalone, so one damaged snapshot never invalidates later ones; retention can drop any snapshot without orphaning another, because chunks are refcounted rather than chained; and there is no weekly-full schedule to design. The first run is the only expensive one. + +### The box appends; it never prunes + +The box holds a credential that can write to its own repository. If that credential could also delete, compromising a box would destroy its backups too. + +- The box's credential is **write-and-list**, scoped so a box can never see or touch another box's repository. +- `forget` and `prune` are **not box operations**. Retention is enforced from the hosted control plane with a separate credential. +- Credential and repository-key rotation arrive over the same authenticated channel as the status report. + +### Repository keys + +Generated per box and **escrowed by the hosted control plane**. This is not operator-blind backup and the spec does not pretend otherwise — `ENVIRONMENT.md` # Threat model already states that at-rest encryption in the hosted profile defends against a co-tenant, a leaked disk image, and an idle volume, not against the operator. Escrow is also what makes operator-driven restore possible for someone who has lost their box entirely. + +### Operational edges + +- **The first run is hours, not minutes** on a large home directory, and per-box jitter does not help one slow box. It needs its own bound; a timeout sized for the steady-state incremental will kill it. +- **A box powered off mid-run leaves a stale repository lock** that refuses the next run. Hosted suspend does exactly this. Unlock handling is explicit, not discovered as a box that quietly stopped backing up. +- **Disk pressure fails the backup exactly when it matters.** The staging dumps and the local cache need free space. Insufficient space is a reported failure with a `HEALTH.md` issue, never a silent skip. +- **A failed run is loud.** A box that stops reporting is visible operator-side; "backed up daily" is a claim that has to stay continuously true. + +## Restore + +A restore path that has never been exercised is a hope, not a backup. The acceptance for any work in this spec is a real restore. + +### Restore is its own entry path, not a replay of first-run + +A restored box has users in its brain state, so the ordinary `/setup` empty-box guard returns 409 and no wizard appears. That is the desired outcome — with credentials restored, the owner simply logs in — but it means restore is a distinct path rather than a variant of provisioning. + +### Brain state splits into portable and box-bound + +**Restoring the brain's SQLite is not a file copy.** The `box_meta` table holds box-scoped identity, and `cmd/brain/main.go` takes a frozen-identity path at boot: if a `box_id` is already present, the brain **ignores the seed entirely**. + +So restoring the old database wholesale onto a freshly provisioned box overwrites the new box's identity with the destroyed box's, and the next boot adopts it — serving the old DNS name with acme-dns credentials belonging to a box that no longer exists. Certificate issuance fails, and the box is unreachable at the name the user was actually given. The failure is silent at restore time and surfaces at the first certificate operation. + +| Class | Rows | Source on restore | +|---|---|---| +| **Box-bound** | `box_id`, `assertion_verification_key`, `enrollment` | the **new** box's seed, never the backup | +| **Owner linkage** | `sso_owner_sub`, `sso_owner_user_id` | the backup — the same portal account owns the restored box, and carrying them is what lets the owner log straight in | +| **Portable** | users, installed apps, settings, telemetry consent, first-run completion | the backup | + +Consequently, **app URLs change**: they are `..malmo.network`, so a restored box's apps are reachable at a new hostname and anything holding the old one is re-rendered on the reconcile pass. + +### The transaction + +**Preconditions, checked before anything is written:** the bundle's schema version is understood, and the target's disk is large enough for the captured set. Hosted grows its root to fill the provider disk at boot (`ENVIRONMENT.md` # Storage), so restoring onto a smaller SKU than the destroyed box fails partway through — after the box exists and the operator believes the restore is underway. It is a cheap check and an expensive omission. + +1. **A fresh, current box** is provisioned and ingests its own seed. It has a new identity; it is not a resurrection. +2. **Brain SQLite** is restored with the box-bound rows re-pointed per the table above — first, because it is the index of what everything else means. +3. **Credentials** are restored, so users can log in. +4. **Managed services** are provisioned for the engines the restored app set requires, and each app's dump is loaded into its own database with its own credential re-established. +5. **Instance directories** are restored, then apps are brought up by the ordinary reconcile path, pulling images by pinned digest. +6. **User and shared content** is restored under the root, reconciled against the users the restored brain state defines. + +**Restore is operator-triggered, by design.** The flow is: a box becomes unhealthy or dies, the customer contacts support, and an operator runs the restore. There is no self-serve restore surface, no customer-facing button, and no first-run "restore from backup" branch — `FIRST_RUN.md` stays greenfield-only, and the wizard fork it would otherwise need is not part of this work. + +**One part of it must still be a tool, not a runbook step.** Re-pointing the box-bound rows above is the step where a hand-run restore silently adopts the destroyed box's identity and breaks certificate issuance — with the failure surfacing later, at the first renewal, rather than during the restore. An operator working through an incident at speed should not be hand-editing identity rows in SQLite. The re-point is a command with the split encoded in it; the runbook calls that command. + +Self-serve file-level restore stays a later possibility and would live in the `FILES.md` surface; it is explicitly out of scope here. + +## Locked decisions + +- **Scheduled off-site backup is hosted-only in v1.** The appliance inherits the set, the consistency rules, and the restore path, but no destination. +- **The set is roots minus an exclude list**, never an enumerated include list — users may add and rename folders, and omission is the failure mode that hides until restore. Hosted has three roots; the appliance has the same three as one bind-mounted tree. +- **The bundle records logical roots, not absolute paths**, so it is not welded to the profile it came from. +- **A run refuses to snapshot a broken box.** Roots present and non-empty, and no implausible shrink against the previous snapshot. An empty snapshot must never be recorded as a success. The appliance canary is not this mechanism and does not exist on hosted. +- **The brain executes the capture; the timer is only the clock.** Dumps require the brain's SQLite and the socket-proxy Docker path. +- **Databases get dumped; files get copied.** Brain SQLite and every managed service, and anything database-shaped added later. +- **Per-app managed-service dumps**, not per-engine volume copies. +- **Consistency is tiered and stated**, not implied. Hooks stay deferred; filesystem freeze is rejected. +- **`/etc/shadow` entries are backed up** so a restore is transparent to the user; the control-plane password push is the named successor. Samba's store is an appliance-only concern, since hosted ships no SMB. +- **Brain state splits into portable and box-bound.** Box identity always comes from the new box's seed. +- **The box appends; it never prunes.** Deletion authority never lives on the box being protected. +- **Repository keys are escrowed on hosted**, consistent with the trust boundary already stated, and never claimed to be otherwise. + +## Open items + +Tracked in `NEXT.md`, not here. diff --git a/docs/drafts/backup-file-level/ENVIRONMENT-box-identity-section.md b/docs/drafts/backup-file-level/ENVIRONMENT-box-identity-section.md new file mode 100644 index 0000000..7f3093e --- /dev/null +++ b/docs/drafts/backup-file-level/ENVIRONMENT-box-identity-section.md @@ -0,0 +1,36 @@ +DRAFT — new section for docs/specs/ENVIRONMENT.md, placed immediately after "### Admin bootstrap — as built" and before "## Networking & discovery (hosted v1)". + +PREREQUISITE FIX, same PR: "### Admin bootstrap — as built" is stale relative to the code. It describes the seed as `{box_id, admin_bootstrap_secret, enrollment}` and a `/setup` gate taking a `bootstrap_secret` body field. `internal/profile/seed.go` shows the seed is actually `{box_id, assertion_verification_key, enrollment}`, and `internal/store/store.go` states the assertion key "replaces the prior one-time admin-bootstrap secret hash" — hosted admin bootstrap is portal SSO now. Correct that section before adding this one; the text below assumes the corrected version. + +--- + +## Box identity — the authenticated box→cloud channel + +Everything a hosted box says to the hosted control plane today is a special case. Certificate renewal authenticates with the acme-dns credential, which is scoped to DNS and meaningless outside it. Catalog sync is unauthenticated, because it fetches a public snapshot. Neither generalizes, so the next thing a box needs to say — a backup outcome (`BACKUP.md` # Box-initiated push) — has nothing to say it with. + +That is not a one-off gap. Fleet health, update status, disk pressure, and the credential delivery backup itself depends on are all the same shape: **the box asserts who it is, and the control plane answers with desired state.** The channel is built once. + +It also has a security payoff independent of any feature that uses it. `seed.json` carries `enrollment.password`, the long-lived acme-dns credential this doc already names as the worst-case box secret — worst-case because it can rewrite `_acme-challenge.` and so mint or MITM the box's wildcard certificate. The seed is delivered over a metadata endpoint that stays readable for the server's whole life, which is why the image carries a standing rule blocking container egress to it (# Provisioning & first-boot, #251). A **single-use, short-lived** bootstrap token makes that exposure self-limiting, and is the precondition for eventually removing the long-lived credential from the seed entirely. + +The seed's other two fields are unaffected: `box_id` stays the box's frozen identity, and `assertion_verification_key` keeps doing its own job (verifying portal SSO assertions inbound to the box). This section adds the **outbound** direction, which nothing covers today. + +### Shape + +- **The seed carries a bootstrap token**, single-use with a TTL measured in hours, alongside the fields # Admin bootstrap — as built already defines. +- **The box generates an Ed25519 keypair locally at first boot.** The private half never leaves the box and is never transmitted. +- **The box spends the token exactly once** to register its public half. The exchange is atomic control-plane-side: a replayed token is refused, and a box that already enrolled does not re-enroll. +- **Every subsequent request is signed** over a canonical string of method, path, body digest, timestamp, and nonce. +- **Rotation** is a new public key signed by the old one. **Revocation** is clearing the registered key. + +**Ed25519 signed requests rather than mTLS.** Client-certificate authentication terminates wherever TLS terminates, which on any proxied surface means the identity check happens in one process and the authorization that depends on it in another, joined by a trusted header. Signing at the request layer keeps the two together. `MALMO_NETWORK.md` made the same call for assertions in the other direction. + +### Constraints the box side must honor + +- **The canonical signing string is a wire contract**, in the same class as the seed's JSON shape and the assertion token format: the two sides meet at bytes, not at a shared type. Canonicalization ambiguity is the classic source of interop bugs that appear only for certain payloads — an empty body, a query string, a non-ASCII path — so it is specified byte-exactly and tested against fixed vectors on both sides. +- **Enrollment precedes anything that needs the channel** in boot order, and retries: first boot races DHCP exactly the way the seed fetch does (# Provisioning & first-boot). +- **Enrollment is first-boot-once, like seed ingestion.** The registered key is the box's frozen credential; re-delivering a spent token cannot re-key a provisioned box. +- **The private key is at-rest state on the box**, with the same custody question as the enrollment credential it is intended to eventually replace (`NEXT.md` # Encrypt hosted enrollment credentials at rest). +- **No secret reaches a log.** Not the bootstrap token, not the private key, not a vended credential — through seed assembly, through structured `slog` fields, or through an error string. +- **A restore is a new identity.** A box restored from backup enrolls fresh and gets a new keypair; it is not the old box (`BACKUP.md` # Restore). Any authorization to read the previous box's backup therefore cannot be anchored to box identity. + +**On `appliance`, none of this exists.** There is no control plane to report to. diff --git a/docs/drafts/backup-file-level/README.md b/docs/drafts/backup-file-level/README.md new file mode 100644 index 0000000..0bc8c6a --- /dev/null +++ b/docs/drafts/backup-file-level/README.md @@ -0,0 +1,45 @@ +# Draft: file-level backup and restore — NOT ADOPTED, DO NOT MERGE + +**Status: parked design work. Nothing here is decided, implemented, or scheduled.** + +These files are a design pass from 2026-07-19 that was **not taken forward**. The hosted product went with a simpler mechanism for v1 (a control-plane-side daily provider snapshot per box, which needs no OS-side code at all). This branch exists so the work and — more importantly — the corrections that came out of it are not lost when the design is revisited. + +**Do not merge this branch.** `docs/drafts/` is not one of the three doc homes (`docs/specs/`, `docs/progress/`, `docs/dev/`) and is not a convention this repo keeps. If this design is adopted, the files move to their real homes: `BACKUP.md` to `docs/specs/`, the box-identity section into `docs/specs/ENVIRONMENT.md`, and the issue drafts into GitHub issues. + +## What is here + +| File | What it is | +|---|---| +| `BACKUP.md` | the spec draft: backup set, consistency posture, credentials, restore transaction | +| `ENVIRONMENT-box-identity-section.md` | a proposed `ENVIRONMENT.md` section for the authenticated box→cloud channel | +| `issue-1-specs.md` | draft issue: the docs-only pass | +| `issue-2-box-identity.md` | draft issue: enrollment + request signing | +| `issue-3-backup-agent.md` | draft issue: manifest parsing, dumps, the agent | + +## The design in one paragraph + +A box captures its own data — user content, shared content, app instances, brain state — deduplicated and encrypted client-side, and pushes it off-box daily, because nothing can reach into a hosted box. Databases are dumped rather than file-copied. Restore lands the bundle on a fresh, current box. The artifact is a logical bundle rather than a disk image, which is what lets the same mechanism serve both disaster recovery and the "move it home" migration promise. + +## Why it was parked, and what would bring it back + +The v1 mechanism is bounded by the provider's **account-wide cap of 30 snapshots** (not per server, not per project), and it is whole-disk, crash-consistent, one-day-of-history, hosted-only. This design has none of those limits: it scales without a support ticket, restores a single file or a single app, costs in proportion to data rather than disk, and is the only option that could ever serve the appliance. + +It comes back when any of those limits starts to bite — most likely the snapshot cap, or the first customer who wants a single deleted file back. + +## Findings worth keeping even if the design is never adopted + +These came out of checking the drafts against the code and are true regardless of which backup mechanism ships: + +1. **Two live doc/code drifts.** `ENVIRONMENT.md` # Admin bootstrap — as built describes a seed carrying `admin_bootstrap_secret` and a `/setup` gate taking a `bootstrap_secret` field; `internal/profile/seed.go` shows `{box_id, assertion_verification_key, enrollment}` and `internal/store/store.go` records that the assertion key *replaced* that secret. Separately, `STORAGE.md`'s layout diagram shows `brain/state.db` and `managed-services/` where `cmd/brain/main.go` opens `/malmo.db` and `internal/lifecycle/services.go` writes under `/services/`. Both are worth fixing on their own. + +2. **`data_volumes` / `cache_volumes` are authored but unparsed**, and `APP_MANIFEST.md` states nesting/overlap rules nothing enforces. This blocks more than backup: `UPDATES.md` # Pre-update snapshot and `APP_LIFECYCLE.md` # Update transaction spec a pre-update tar of declared `data_volumes` as the **v1 rollback safety net** for app-driven schema migrations — a locked decision (`DECISIONS.md` 2026-05-17) that is unbuilt for exactly this missing field. + +3. **Managed-service data lives outside the instance directories.** Any backup that copies only `/var/lib/malmo/instances//` restores a database-backed app with its config intact and its rows gone — the failure that looks like success until a restore. + +4. **The hosted storage model is not the appliance's, and reading `STORAGE.md` first produces confident wrong answers.** `ENVIRONMENT.md` # Storage (hosted) removes mergerfs, the add/eject flows, and the storage canary, and puts everything on one volume — so there is no `/srv/malmo/{home,state,shared}` bind-mount tree there, and `/home` and `/var/lib/malmo` are real directories. + +5. **An include list of user folders is unsafe.** `STORAGE.md` says users may rename, delete, and add folders, so enumerating the use-case folders silently omits anything a user creates. Any future backup work should subtract from roots rather than enumerate paths. + +6. **Restoring the brain's SQLite cannot be a file copy.** `box_meta` holds box-scoped identity and `cmd/brain/main.go` ignores the seed entirely when a `box_id` is already present — so a wholesale restore makes a fresh box adopt the destroyed box's identity, and it fails silently until the first certificate operation. + +7. **Password hashes are not in any backup set by default.** PAM is the source of truth and the brain holds no hash, so a restore without `/etc/shadow` brings back accounts nobody can log into. Hosted ships no Samba, so `tdbsam` is an appliance-only concern. diff --git a/docs/drafts/backup-file-level/issue-1-specs.md b/docs/drafts/backup-file-level/issue-1-specs.md new file mode 100644 index 0000000..06838da --- /dev/null +++ b/docs/drafts/backup-file-level/issue-1-specs.md @@ -0,0 +1,80 @@ +TITLE: specs: backup + restore architecture, and the authenticated box→cloud channel + +Size: M +Area: backend +Depends on: none + +## Summary + +Docs only, no code. Writes the backup/restore architecture malmo has been deferring behind `NEXT.md` # Backup architecture shape, and specs the authenticated box→cloud channel a hosted box needs before it can report anything. Base for the two build issues that follow. + +## Spec / source of truth + +New `docs/specs/BACKUP.md`; new # Box identity section in `docs/specs/ENVIRONMENT.md`. Read `STORAGE.md`, `APP_LIFECYCLE.md`, `APP_MANIFEST.md` # Storage, `SERVICE_PROVISIONING.md`, `AUTH.md`, `UPDATES.md` # Pre-update snapshot, and `ENVIRONMENT.md` end-to-end first — this pass sits on all of them. + +## Do + +### 1. New `docs/specs/BACKUP.md` + +**Scope.** Scheduled off-site backup is **hosted-only** in v1 — that is where malmo runs the box and carries the custody obligation. State plainly that an appliance box therefore has **no backup at all** in v1: single drive, SnapRAID deferred, no destination. Add `NEXT.md` # Appliance backup destination rather than letting it read as an oversight. The appliance still inherits the backup set, the consistency rules, and the restore path, because the artifact is a logical bundle and is the same vehicle as the portability promise in `ENVIRONMENT.md` # Logical export / restore bundle. + +**The set is roots minus an exclude list, never an enumerated include list.** This is the most important choice in the spec. `STORAGE.md` # What apps and users actually see says users **may rename, delete, or add folders**, so an include list naming the use-case folders silently omits anything a user creates — a backup that looks healthy and is missing data. An exclude list fails the other way: an unanticipated path costs storage, not data. + +**Get the profile difference right — it is not one root on hosted.** `ENVIRONMENT.md` # Storage (hosted) collapses the appliance layout: no mergerfs, no add/eject, **one volume for everything**. So `/home` and `/var/lib/malmo` are real directories there, not views onto `/srv/malmo/`, and hosted has **three roots**: `/home//`, `/var/lib/malmo/`, and `/srv/malmo/shared/` (`internal/lifecycle/lifecycle.go` sets the shared root on both profiles). The appliance has the same three logical roots as one bind-mounted tree, where the rule is to back up the tree rather than the views so bytes are not traversed twice. + +Consequently **the bundle records logical roots tagged by role, not absolute paths** — the same photo is `/home/cindy/Photos/…` on hosted and `/srv/malmo/home/cindy/Photos/…` on an appliance, and a bundle keyed to absolute paths can only restore onto the profile it came from, forfeiting the portability promise. + +**A run must refuse to snapshot a broken box.** An empty or drastically incomplete capture recorded as a success is worse than a failure, because retention eventually replaces good snapshots with empty ones. Do **not** reach for the appliance's storage canary: `ENVIRONMENT.md` # Storage says hosted has "no mergerfs, no add/eject, no canary" — it detects a removed or wrong *physical* drive and has nothing to do on a cloud volume. Hosted needs its own precondition: every expected root present and non-empty, and no implausible shrink against the previous snapshot. + +Excludes (relative to the state root): `instances/*/snapshots/`, every declared `cache_volumes` subtree, the **live data directories of managed services** (dumped instead), `catalog-cache/`, `seed.json`, the engine's local cache. The OS and Docker images fall outside the roots by construction. + +**Verify these path names against the code, not against `STORAGE.md`.** Its layout diagram is stale — it shows `brain/state.db` and `managed-services/`, while `cmd/brain/main.go` opens `/malmo.db` and `internal/lifecycle/services.go` writes services under `/services/`. An exclude keyed to a name that no longer exists silently fails to exclude, which is exactly how a live database gets file-copied into a backup. Fix the diagram in this pass. + +**Databases get dumped; files get copied.** Brain SQLite via `VACUUM INTO`. Managed services via a **per-app logical dump** through the throwaway one-shot client container the brain already uses to provision — `SERVICE_PROVISIONING.md` provisions them as brain-owned compose projects with data outside `/var/lib/malmo/instances//`, so copying only instance directories restores a database-backed app with its settings intact and its rows gone. `SERVICE_PROVISIONING.md` # At backup already states the intended shape and lists this as "gated on the backup design"; this spec is that gate. + +**Credentials.** `AUTH.md` makes PAM the source of truth and the brain holds no password hash, so without this a restore brings back accounts nobody can log into. v1 backs up the **`/etc/shadow` entries** for malmo users, which is the whole credential set on hosted — `ENVIRONMENT.md` # Access & files ships **no Samba** there, so `tdbsam` is not in scope. Note that an appliance restore would need it too (it holds its own NT hashes, synced from `passwd` only on a live change, `AUTH.md` # Samba password backend), and leave room in the bundle schema rather than pretending the profiles match. Note honestly that this puts password hashes in the backup, and why that is a widening rather than a new concession (the repository is client-side encrypted and `ENVIRONMENT.md` # Threat model already places the operator inside the hosted trust boundary). Name the successor: once the box→cloud channel exists, the control plane can push a password to a restored box and keep hashes out entirely. + +**Who executes a run.** The timer is a host unit, but `VACUUM INTO` acts on the brain's live SQLite and the service dumps need the socket-proxy Docker path (`CONTROL_PLANE.md` # Docker socket exposure) — neither is a host operation, and the brain runs containerized with `MALMO_STATE_DIR` so its paths and host paths differ. Spec the split (timer as clock, brain as executor) and flag the exact seam as the build issue's first decision, since it determines where restic runs and which side holds the repository credential. + +**Consistency, tiered and stated:** managed-service data and brain state consistent by construction; everything else crash-consistent, including an app that bundles its own database inside `data/`. Hooks stay deferred — `DECISIONS.md` 2026-05-13 deferred them noting every use case was tied to managed services or backups, and backup arriving is that trigger, but the dump path covers the exposed population without asking any author to write anything. Filesystem freeze is **rejected**, not deferred: no snapshotting volume manager under the locked ext4 layout. + +**The push agent:** box-initiated (nothing can reach into a box with no sshd), restic via `mkosi.extra` so the lean-image guard still passes, jittered timer, **write-and-list credentials only — the box never prunes**, keys escrowed. Spell out that every snapshot is logically full and physically incremental: no full/incremental chain, every snapshot restores standalone, retention can drop any snapshot without orphaning another. Operational edges are requirements, not polish: the first run is hours; a box powered off mid-run leaves a stale lock; disk pressure fails the backup exactly when it matters; a failed run is loud. + +**Restore, including the part that bites.** Restore is its own entry path, not a replay of first-run: a restored box has users, so `/setup` returns 409 and no wizard appears — which is the desired outcome once credentials are restored. + +The real hazard is that **restoring the brain's SQLite cannot be a file copy.** `box_meta` holds box-scoped identity and `cmd/brain/main.go` takes a frozen-identity path at boot — if a `box_id` is present the brain **ignores the seed entirely**. Restoring the old database wholesale onto a fresh box therefore overwrites the new identity with the destroyed box's, and the next boot serves the old DNS name with acme-dns credentials belonging to a box that no longer exists. Silent at restore, surfaces at the first certificate operation. Spec the split: **box-bound** (`box_id`, `assertion_verification_key`, `enrollment`) always from the new seed; **owner linkage** (`sso_owner_sub`, `sso_owner_user_id`) from the backup so the owner logs straight in; **portable** (users, apps, settings) from the backup. Note the consequence that app URLs change, since they are `..malmo.network`. + +Then the transaction ordering: fresh box → brain state with identity re-pointed → credentials → managed services + dumps → instances via ordinary reconcile → content. Add the preconditions: the bundle's schema version is understood, and **the target disk is large enough** — hosted grows its root to fill the provider disk (`ENVIRONMENT.md` # Storage), so restoring onto a smaller SKU fails partway through, after the operator believes it is underway. + +### 2. Fix `ENVIRONMENT.md` # Admin bootstrap — as built (prerequisite) + +That section is **stale relative to the code**. It describes the seed as `{box_id, admin_bootstrap_secret, enrollment}` and a `/setup` gate taking a `bootstrap_secret` body field. `internal/profile/seed.go` shows `{box_id, assertion_verification_key, enrollment}`, and `internal/store/store.go` says the assertion key "replaces the prior one-time admin-bootstrap secret hash" — hosted admin bootstrap is portal SSO now. Correct it before adding the section below, which assumes the corrected text. + +### 3. New # Box identity section in `ENVIRONMENT.md` + +Single-use hours-TTL bootstrap token in the seed → box generates an Ed25519 keypair at first boot → spends the token once to register the public half → signs every later request over a canonical string. Rotation is a new key signed by the old; revocation is clearing the key. Ed25519 signed requests rather than mTLS, because certificate auth terminates where TLS does and splits the identity check from the authorization depending on it. + +Note this is the **outbound** direction — `assertion_verification_key` already covers inbound portal assertions and is unaffected. + +Box-side constraints to state: the canonical signing string is a **byte-exact wire contract** tested against fixed vectors (same class as the seed shape and the assertion token format); enrollment precedes anything needing the channel and retries through the first-boot DHCP race; enrollment is first-boot-once like seed ingestion; the private key is at-rest state with the same custody question as `NEXT.md` # Encrypt hosted enrollment credentials at rest; no secret ever reaches a log; a restored box enrolls fresh and is a new identity. + +Also note the standalone security payoff: a single-use token makes the seed's long-lived acme-dns credential exposure self-limiting, and is the precondition for eventually removing it from the seed. + +### 4. Doc-map and cross-reference upkeep + +- `docs/README.md` — add the `BACKUP.md` row. A spec not listed there is a bug. +- `NEXT.md` — resolve # Backup architecture shape into the new spec; add # Appliance backup destination; keep the genuinely-open items rather than deleting them. +- `SERVICE_PROVISIONING.md` — point the deferred backup/restore note at the now-existing gate. +- `DECISIONS.md` — an entry for this pass, noting explicitly that it does **not** reopen 2026-05-13 (hooks). +- `ENVIRONMENT.md` — # Logical export / restore bundle becomes a pointer; resolve the "export/restore bundle home" open question. +- `STORAGE.md` — fix the stale layout diagram (`brain/state.db` → `malmo.db`, `managed-services/` → `services/`) so the exclude list can be keyed to real names. +- `MALMO_NETWORK.md:168` and `:324` list off-site backup among paid SKUs. That framing is appliance/mesh monetization and is left as-is; flag it in the progress entry so it is a decision rather than drift. +- A `docs/progress/` entry plus its index line. + +## Touch + +`docs/specs/BACKUP.md`, `docs/specs/ENVIRONMENT.md`, `docs/specs/NEXT.md`, `docs/specs/DECISIONS.md`, `docs/specs/SERVICE_PROVISIONING.md`, `docs/README.md`, `docs/progress/`. No code. + +## Done when + +`docs/specs/BACKUP.md` exists and answers without hedging: what the backup root is and what is subtracted from it, which parts are consistent and which are only crash-consistent, how a user logs in after a restore, and which brain state is portable versus box-bound. `NEXT.md` # Backup architecture shape is resolved rather than still reserved, and the appliance gap is a named item. `ENVIRONMENT.md` # Admin bootstrap — as built matches the code, and # Box identity specs the outbound channel including byte-exactness of the signing string. `docs/README.md` lists the new spec. diff --git a/docs/drafts/backup-file-level/issue-2-box-identity.md b/docs/drafts/backup-file-level/issue-2-box-identity.md new file mode 100644 index 0000000..a7d6c33 --- /dev/null +++ b/docs/drafts/backup-file-level/issue-2-box-identity.md @@ -0,0 +1,37 @@ +TITLE: Box identity: Ed25519 enrollment and request signing for the box→cloud channel + +Size: M +Area: backend +Depends on: the specs issue (#N) + +## Summary + +A hosted box has no general-purpose way to authenticate itself to the hosted control plane. Certificate renewal uses the acme-dns credential, which is scoped to DNS; catalog sync is unauthenticated because it fetches a public snapshot. Neither generalizes, so the first thing a box needs to *report* has nothing to report with. This builds the channel once: a single-use bootstrap token in the seed, an Ed25519 keypair generated on the box, and signed requests thereafter. + +## Spec / source of truth + +`docs/specs/ENVIRONMENT.md` # Box identity (written by the specs issue). Read # Provisioning & first-boot and # Admin bootstrap — as built end-to-end first — this extends the seed contract they define, and reuses their first-boot-once ingestion pattern deliberately. + +## Do + +- **Extend seed ingestion** with the bootstrap token field, following the existing shape in `internal/` where `seed.json` is read. The seed's JSON shape is a wire contract mirrored byte-for-byte on the other side; the two repos meet at the format, not a shared Go type. +- **Generate an Ed25519 keypair at first boot** and persist it. The private half never leaves the box and is never transmitted. Persist with the same care the bootstrap-secret hash gets today: the write ordering must make a crash mid-write re-runnable next boot rather than stranding a half-state. +- **Enrollment call**, spending the token exactly once to register the public half. It must: + - run **before anything that needs the channel**, in boot order; + - **retry through the first-boot DHCP race**, the way the seed metadata fetch does (bounded, never blocking forever, exiting cleanly on a definitive negative); + - be **first-boot-once** — a box that has enrolled ignores a re-delivered token, mirroring how a persisted `box_id` makes identity frozen. +- **Request signer** producing the canonical string over method, path, body digest, timestamp, and nonce. This is the delicate part: canonicalization ambiguity produces interop bugs that appear only for certain payloads (empty body, query string, non-ASCII path, repeated header). Implement against the spec's byte-exact definition and **test against fixed vectors**, not against a round-trip with our own signer — a round-trip passes happily when both sides are wrong in the same way. +- **Rotation**: a new key signed by the old one, on a schedule. +- **Secrets never logged.** Not the token, not the private key, not a vended credential — through seed assembly, `slog` fields, or an error string. Add the negative test. + +## Touch + +`internal/` (seed ingestion and a new box-identity package), `cmd/brain/`, `dev/cloud/` if the first-boot ordering needs a unit change. Cross-platform: the signing and enrollment logic must compile everywhere and stay inner-loop testable; only host wiring is Linux-only. + +## Done when + +A box boots un-enrolled, generates a keypair, spends its token, and signs a subsequent request that verifies against the canonical string — proven in the QEMU cloud lane across a multi-boot sequence, the way the seed gate is (un-enrolled → enrolled → reboot keeps the same identity). A replayed token is refused. Fixed-vector tests cover the canonical string including the awkward payloads. No secret appears in any log at any level. + +## Notes + +The canonical signing string is co-owned with the other side of the channel and cannot be changed unilaterally once either side ships. diff --git a/docs/drafts/backup-file-level/issue-3-backup-agent.md b/docs/drafts/backup-file-level/issue-3-backup-agent.md new file mode 100644 index 0000000..c36ec83 --- /dev/null +++ b/docs/drafts/backup-file-level/issue-3-backup-agent.md @@ -0,0 +1,88 @@ +TITLE: Daily off-site backup for hosted boxes: data/cache parsing, managed-service dumps, and the backup agent + +Size: L +Area: backend +Depends on: the specs issue (#N); the status report depends on box identity (#N) + +## Summary + +Builds the backup set `BACKUP.md` defines and pushes it off-box daily on hosted. Two of its three parts are prerequisites that other already-specced-but-unbuilt work is also waiting on, so they are worth doing here rather than as backup-private plumbing. + +Consider splitting at the marked seam if this is too large for one slice — the first two sections stand alone and have value without the agent. + +## Spec / source of truth + +`docs/specs/BACKUP.md` (written by the specs issue). Also `ENVIRONMENT.md` # Storage (hosted) and # How the profile is realized — read these before `STORAGE.md`, since hosted collapses most of the appliance storage model. Then `STORAGE.md` # What apps and users actually see, `APP_MANIFEST.md` # Storage, `SERVICE_PROVISIONING.md` # At backup, `UPDATES.md` # Pre-update snapshot, `CONTROL_PLANE.md` # Docker socket exposure. + +## Do + +### 1. Parse and validate `data_volumes` / `cache_volumes` + +`internal/manifest/manifest.go` currently reads only `storage.estimated_size` and says so in a comment: the other storage keys "live in the compose, not parsed here." Nothing consumes the data/cache split, which is why two specced features are unbuilt. + +- Parse both lists into `manifest.Storage`. +- **Enforce the rules `APP_MANIFEST.md` states but nothing checks**: a `cache_volumes` path may nest inside a `data_volumes` path (the common shape is `./data` with `./data/cache`), but paths that do not nest must not otherwise overlap. Admission rejects a violation. This is the substance of the work — an unvalidated declaration silently excludes a data tree from backup, and that surfaces at restore. +- Door-2 synthetic manifests declare no `cache_volumes`; the documented fallback is best-effort backup of everything under `data/`. + +**Why this is not backup-private:** `UPDATES.md` # Pre-update snapshot and `APP_LIFECYCLE.md` # Update transaction spec a pre-update tar of declared `data_volumes` as the **v1 rollback safety net** for app-driven schema migrations — a locked decision (`DECISIONS.md` 2026-05-17) unbuilt for exactly this missing field. Same field, two consumers. + +### 2. Per-app managed-service dump path + +`SERVICE_PROVISIONING.md` provisions shared services outside the instance directory, so without this an app using a managed database backs up its config and loses its rows. + +- A **logical dump per app per service** (not a file copy of the service volume, not per-engine), so a restore can be scoped to one app. +- Run through the **existing throwaway one-shot client container** — `docker run --rm --network malmo-svc-- --env-file /.env ` — so the brain still never joins a service network (`DECISIONS.md` 2026-06-02, 2026-06-15). `internal/lifecycle/services.go` already does this for provisioning. +- Cover the shipped engines: Postgres, MySQL, MariaDB, Valkey. +- The matching **restore** direction, since a dump nobody has restored is not a backup. + +**Also not backup-private:** `UPDATES.md` and `APP_LIFECYCLE.md` both spec a managed-service dump alongside the pre-update tar, and `SERVICE_PROVISIONING.md` lists cross-version migration as gated on the same dump/restore path. + +--- seam: everything above stands alone --- + +### 3. The backup agent (hosted) + +**Settle the execution model first — it determines everything below.** The timer is a host unit, but `VACUUM INTO` acts on the brain's live SQLite and the service dumps need the service `.env`, the service network, and `docker run --rm` through the socket-proxy (`CONTROL_PLANE.md` # Docker socket exposure) — a host script bypassing it reopens a boundary `DECISIONS.md` 2026-06-14 closed. The brain also runs containerized with `MALMO_STATE_DIR`, so brain paths and host paths are different strings. The shape is timer-as-clock, brain-as-executor; the seam (host-agent job, brain endpoint, or brain-internal scheduler woken by the timer) is this issue's first decision, because it fixes where restic runs and which side holds the repository credential. + +**The set is roots minus excludes.** On hosted there are **three**: `/home//`, `/var/lib/malmo/`, and `/srv/malmo/shared/`. There is no `/srv/malmo` bind-mount tree here — `ENVIRONMENT.md` # Storage (hosted) has no mergerfs and one volume for everything, so `/home` and `/var/lib/malmo` are real directories. Do not enumerate user folders; users rename, delete, and add them, so an include list silently omits data. + +Excludes, relative to the state root: `instances/*/snapshots/`, declared `cache_volumes` subtrees, the live managed-service data directories under `services/` (dumped instead), `catalog-cache/`, `seed.json`, the engine cache. **Verify these names against the code** — `STORAGE.md`'s diagram is stale (`brain/state.db`, `managed-services/`) versus `cmd/brain/main.go` (`/malmo.db`) and `internal/lifecycle/services.go` (`/services/`). An exclude keyed to a dead name silently fails to exclude, which is how a live database gets file-copied in. + +**Record logical roots tagged by role, not absolute paths**, so the bundle is not welded to the profile it came from. + +**Refuse to snapshot a broken box.** Every expected root present and non-empty, and no implausible shrink versus the previous snapshot, checked before anything is written. Note this is **not** the appliance storage canary, which does not exist on hosted (`ENVIRONMENT.md` # Storage: "no mergerfs, no add/eject, no canary") — it detects a wrong or removed physical drive and has no analogue on a cloud volume. + +**Capture what lives outside the roots:** the brain SQLite via `VACUUM INTO` to a staging path (never a live file copy), the per-app service dumps from section 2, and the **`/etc/shadow` entries** for malmo users. Samba's `tdbsam` is *not* in scope — hosted ships no SMB (`ENVIRONMENT.md` # Access & files) — but leave room for it in the bundle schema, since an appliance restore would need it. Also record a bundle manifest: roots and their roles, the OS image identity, a schema version, so a restore can refuse a bundle it does not understand rather than half-applying it. + +**The agent:** + +- **restic in the hosted image**, shipped via `mkosi.extra` as a static binary so it adds no apt package and the lean-image manifest guard still passes. +- **`malmo-backup.timer` + service**, in the quiet window, **jittered** so a fleet does not stampede. +- **Backup configuration from the seed**; **status reported** on the authenticated box→cloud channel. +- **The box appends and never prunes.** No `forget`, no `prune`, no delete path in box-side code — retention is enforced elsewhere with a different credential, and that separation is the ransomware posture. +- **Edges are requirements, not polish:** the **first run** is hours on a large home directory and needs its own bound; a box powered off mid-run leaves a **stale repository lock** that must be handled explicitly; **insufficient disk** for staging dumps or the cache is a reported failure with a `HEALTH.md` issue, never a silent skip; a **failed run is loud**. + +### 4. Restore + +**Restore is operator-triggered**: a box dies, the customer contacts support, an operator runs the process. No self-serve surface, no first-run restore branch. But the identity re-point below is a **command**, not a documented SQL sequence — it is the step that silently breaks a restored box, and an operator mid-incident should not be hand-editing identity rows. + +**Restoring the brain SQLite is not a file copy.** `box_meta` holds box-scoped identity, and `cmd/brain/main.go:397` takes a frozen-identity path at boot: if a `box_id` is present, the brain **ignores the seed entirely**. Restoring the old database wholesale onto a fresh box adopts the destroyed box's identity, then serves the old DNS name with acme-dns credentials for a box that no longer exists — silent at restore, surfacing at the first certificate operation. + +- **Box-bound** (`box_id`, `assertion_verification_key`, `enrollment`): always from the new box's seed. +- **Owner linkage** (`sso_owner_sub`, `sso_owner_user_id`): from the backup, so the owner logs straight in. +- **Portable** (users, apps, settings, telemetry consent): from the backup. + +Restore is its own entry path — a restored box has users, so `/setup` returns 409 and no wizard appears. App URLs change, since they are `..malmo.network`; anything holding the old hostname is re-rendered on reconcile. + +Order: fresh box → brain state with identity re-pointed → credentials → services + dumps → instances via ordinary reconcile → content. + +Preconditions before writing anything: the bundle schema version is understood, and the **target disk is large enough** for the captured set. Hosted grows its root to fill the provider disk at boot, so restoring onto a smaller SKU fails partway through — after the box exists and the operator believes the restore is running. + +## Touch + +`internal/manifest/`, `internal/admission/`, `internal/lifecycle/services.go` plus a new backup package, `dev/cloud/` (image + units). Sections 1 and 2 are cross-platform and inner-loop testable; only the agent's host wiring is hosted/Linux-only. + +## Done when + +**A real restore, not a green upload.** Back up a box with a database-backed app installed, provision a fresh box, restore, and confirm: the app comes back with its **rows**, the user logs in with their **existing password**, and the restored box serves its **own** DNS name and gets a certificate — not the destroyed box's. + +Plus: a manifest with overlapping non-nesting paths is rejected at admission; a `./data` + `./data/cache` declaration backs up the data tree minus the cache subtree; a run **aborts rather than snapshotting** when a root is missing or empty; a file placed in a newly-created user folder (not one of the standard use-case folders) is present in the restore, proving the exclude-list model; the hosted image still passes the lean package-manifest check with restic present; a box interrupted mid-run backs up successfully on its next timer firing.