Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
219 changes: 219 additions & 0 deletions docs/drafts/backup-file-level/BACKUP.md

Large diffs are not rendered by default.

36 changes: 36 additions & 0 deletions docs/drafts/backup-file-level/ENVIRONMENT-box-identity-section.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
DRAFT — new section for docs/specs/ENVIRONMENT.md, placed immediately after "### Admin bootstrap — as built" and before "## Networking & discovery (hosted v1)".

PREREQUISITE FIX, same PR: "### Admin bootstrap — as built" is stale relative to the code. It describes the seed as `{box_id, admin_bootstrap_secret, enrollment}` and a `/setup` gate taking a `bootstrap_secret` body field. `internal/profile/seed.go` shows the seed is actually `{box_id, assertion_verification_key, enrollment}`, and `internal/store/store.go` states the assertion key "replaces the prior one-time admin-bootstrap secret hash" — hosted admin bootstrap is portal SSO now. Correct that section before adding this one; the text below assumes the corrected version.

---

## Box identity — the authenticated box→cloud channel

Everything a hosted box says to the hosted control plane today is a special case. Certificate renewal authenticates with the acme-dns credential, which is scoped to DNS and meaningless outside it. Catalog sync is unauthenticated, because it fetches a public snapshot. Neither generalizes, so the next thing a box needs to say — a backup outcome (`BACKUP.md` # Box-initiated push) — has nothing to say it with.

That is not a one-off gap. Fleet health, update status, disk pressure, and the credential delivery backup itself depends on are all the same shape: **the box asserts who it is, and the control plane answers with desired state.** The channel is built once.

It also has a security payoff independent of any feature that uses it. `seed.json` carries `enrollment.password`, the long-lived acme-dns credential this doc already names as the worst-case box secret — worst-case because it can rewrite `_acme-challenge.<box-id>` and so mint or MITM the box's wildcard certificate. The seed is delivered over a metadata endpoint that stays readable for the server's whole life, which is why the image carries a standing rule blocking container egress to it (# Provisioning & first-boot, #251). A **single-use, short-lived** bootstrap token makes that exposure self-limiting, and is the precondition for eventually removing the long-lived credential from the seed entirely.

The seed's other two fields are unaffected: `box_id` stays the box's frozen identity, and `assertion_verification_key` keeps doing its own job (verifying portal SSO assertions inbound to the box). This section adds the **outbound** direction, which nothing covers today.

### Shape

- **The seed carries a bootstrap token**, single-use with a TTL measured in hours, alongside the fields # Admin bootstrap — as built already defines.
- **The box generates an Ed25519 keypair locally at first boot.** The private half never leaves the box and is never transmitted.
- **The box spends the token exactly once** to register its public half. The exchange is atomic control-plane-side: a replayed token is refused, and a box that already enrolled does not re-enroll.
- **Every subsequent request is signed** over a canonical string of method, path, body digest, timestamp, and nonce.
- **Rotation** is a new public key signed by the old one. **Revocation** is clearing the registered key.

**Ed25519 signed requests rather than mTLS.** Client-certificate authentication terminates wherever TLS terminates, which on any proxied surface means the identity check happens in one process and the authorization that depends on it in another, joined by a trusted header. Signing at the request layer keeps the two together. `MALMO_NETWORK.md` made the same call for assertions in the other direction.

### Constraints the box side must honor

- **The canonical signing string is a wire contract**, in the same class as the seed's JSON shape and the assertion token format: the two sides meet at bytes, not at a shared type. Canonicalization ambiguity is the classic source of interop bugs that appear only for certain payloads — an empty body, a query string, a non-ASCII path — so it is specified byte-exactly and tested against fixed vectors on both sides.
- **Enrollment precedes anything that needs the channel** in boot order, and retries: first boot races DHCP exactly the way the seed fetch does (# Provisioning & first-boot).
- **Enrollment is first-boot-once, like seed ingestion.** The registered key is the box's frozen credential; re-delivering a spent token cannot re-key a provisioned box.
- **The private key is at-rest state on the box**, with the same custody question as the enrollment credential it is intended to eventually replace (`NEXT.md` # Encrypt hosted enrollment credentials at rest).
- **No secret reaches a log.** Not the bootstrap token, not the private key, not a vended credential — through seed assembly, through structured `slog` fields, or through an error string.
- **A restore is a new identity.** A box restored from backup enrolls fresh and gets a new keypair; it is not the old box (`BACKUP.md` # Restore). Any authorization to read the previous box's backup therefore cannot be anchored to box identity.

**On `appliance`, none of this exists.** There is no control plane to report to.
45 changes: 45 additions & 0 deletions docs/drafts/backup-file-level/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Draft: file-level backup and restore — NOT ADOPTED, DO NOT MERGE

**Status: parked design work. Nothing here is decided, implemented, or scheduled.**

These files are a design pass from 2026-07-19 that was **not taken forward**. The hosted product went with a simpler mechanism for v1 (a control-plane-side daily provider snapshot per box, which needs no OS-side code at all). This branch exists so the work and — more importantly — the corrections that came out of it are not lost when the design is revisited.

**Do not merge this branch.** `docs/drafts/` is not one of the three doc homes (`docs/specs/`, `docs/progress/`, `docs/dev/`) and is not a convention this repo keeps. If this design is adopted, the files move to their real homes: `BACKUP.md` to `docs/specs/`, the box-identity section into `docs/specs/ENVIRONMENT.md`, and the issue drafts into GitHub issues.

## What is here

| File | What it is |
|---|---|
| `BACKUP.md` | the spec draft: backup set, consistency posture, credentials, restore transaction |
| `ENVIRONMENT-box-identity-section.md` | a proposed `ENVIRONMENT.md` section for the authenticated box→cloud channel |
| `issue-1-specs.md` | draft issue: the docs-only pass |
| `issue-2-box-identity.md` | draft issue: enrollment + request signing |
| `issue-3-backup-agent.md` | draft issue: manifest parsing, dumps, the agent |

## The design in one paragraph

A box captures its own data — user content, shared content, app instances, brain state — deduplicated and encrypted client-side, and pushes it off-box daily, because nothing can reach into a hosted box. Databases are dumped rather than file-copied. Restore lands the bundle on a fresh, current box. The artifact is a logical bundle rather than a disk image, which is what lets the same mechanism serve both disaster recovery and the "move it home" migration promise.

## Why it was parked, and what would bring it back

The v1 mechanism is bounded by the provider's **account-wide cap of 30 snapshots** (not per server, not per project), and it is whole-disk, crash-consistent, one-day-of-history, hosted-only. This design has none of those limits: it scales without a support ticket, restores a single file or a single app, costs in proportion to data rather than disk, and is the only option that could ever serve the appliance.

It comes back when any of those limits starts to bite — most likely the snapshot cap, or the first customer who wants a single deleted file back.

## Findings worth keeping even if the design is never adopted

These came out of checking the drafts against the code and are true regardless of which backup mechanism ships:

1. **Two live doc/code drifts.** `ENVIRONMENT.md` # Admin bootstrap — as built describes a seed carrying `admin_bootstrap_secret` and a `/setup` gate taking a `bootstrap_secret` field; `internal/profile/seed.go` shows `{box_id, assertion_verification_key, enrollment}` and `internal/store/store.go` records that the assertion key *replaced* that secret. Separately, `STORAGE.md`'s layout diagram shows `brain/state.db` and `managed-services/` where `cmd/brain/main.go` opens `<stateDir>/malmo.db` and `internal/lifecycle/services.go` writes under `<stateDir>/services/`. Both are worth fixing on their own.

2. **`data_volumes` / `cache_volumes` are authored but unparsed**, and `APP_MANIFEST.md` states nesting/overlap rules nothing enforces. This blocks more than backup: `UPDATES.md` # Pre-update snapshot and `APP_LIFECYCLE.md` # Update transaction spec a pre-update tar of declared `data_volumes` as the **v1 rollback safety net** for app-driven schema migrations — a locked decision (`DECISIONS.md` 2026-05-17) that is unbuilt for exactly this missing field.

3. **Managed-service data lives outside the instance directories.** Any backup that copies only `/var/lib/malmo/instances/<id>/` restores a database-backed app with its config intact and its rows gone — the failure that looks like success until a restore.

4. **The hosted storage model is not the appliance's, and reading `STORAGE.md` first produces confident wrong answers.** `ENVIRONMENT.md` # Storage (hosted) removes mergerfs, the add/eject flows, and the storage canary, and puts everything on one volume — so there is no `/srv/malmo/{home,state,shared}` bind-mount tree there, and `/home` and `/var/lib/malmo` are real directories.

5. **An include list of user folders is unsafe.** `STORAGE.md` says users may rename, delete, and add folders, so enumerating the use-case folders silently omits anything a user creates. Any future backup work should subtract from roots rather than enumerate paths.

6. **Restoring the brain's SQLite cannot be a file copy.** `box_meta` holds box-scoped identity and `cmd/brain/main.go` ignores the seed entirely when a `box_id` is already present — so a wholesale restore makes a fresh box adopt the destroyed box's identity, and it fails silently until the first certificate operation.

7. **Password hashes are not in any backup set by default.** PAM is the source of truth and the brain holds no hash, so a restore without `/etc/shadow` brings back accounts nobody can log into. Hosted ships no Samba, so `tdbsam` is an appliance-only concern.
Loading