Document fleet mode: README, ADR-0007, CHANGELOG

The README gains a horizontal-scaling section with a worked config, the
load-balancer contract (point your health check at /_shipnode/ready),
and a prominent note on drainWait — the one number shipnode cannot infer
and the difference between a zero-downtime roll and a visible outage.

ADR-0007 records the design and, as importantly, its limits: why
shipnode owns a readiness endpoint instead of managing a load balancer,
why draining is a sentinel file rather than a config rewrite, why
replicas never claim the domain, why a failed roll is a state rather
than an exception, and what is deliberately absent.

Author: devalade

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to `@devalade/shipnode` will be documented here.
 ## [Unreleased]
 
 ### Added
+- **Fleet mode — one app on N servers behind your load balancer.** An `on` target resolving to more than one server makes the app a fleet: `shipnode deploy` rolls through its replicas a batch at a time, draining each out of rotation before touching it. Blue-green stays intra-replica, so the composition is blue-green within a replica, rolling across replicas. See [ADR-0007](docs/adr/0007-fleet-replication.md).
+  - **Shipnode does not manage the load balancer.** You provision it (Hetzner, DigitalOcean, ALB, nginx); shipnode owns one readiness endpoint per replica, `/_shipnode/ready`, which answers `200` normally and `503` while that replica deploys. Point your LB's health check at it. No provider APIs, no credentials — SSH remains the only requirement.
+  - **`fleet.drainWait` (default 30s) is the setting that matters.** Shipnode drains instantly; your LB only notices on its next health check, after its failure threshold. Shipnode can see neither, so you declare the wait. Below your LB's *interval × unhealthy-threshold* and the roll drops requests.
+  - Draining is a sentinel file matched by Caddy, not a config rewrite: one `touch`, no reload, immediate, and it survives a Caddy or host restart — so a replica drained by a failed roll stays drained rather than quietly rejoining the pool.
+  - Fleet replicas serve plain HTTP on `privateHost:fleet.port` and never claim the app's domain. Five replicas all requesting a certificate for one name would race Let's Encrypt. **TLS terminates at your load balancer**; pass-through TLS is out of scope.
+  - `.group(name, servers)` names a set of servers; `on` accepts a server, a group, or a list of either. Every fleet member must declare `privateHost`.
+  - A failed roll stops and reports the mixed-version fleet rather than claiming success. Updated replicas keep serving the new release, untouched ones the old, batch-mates drained but never touched go back into rotation, and **the failed replica stays drained** — out of rotation and inspectable as it died.
+  - One release id for the whole roll, so `shipnode status` can distinguish a converged fleet from a half-rolled one.
+- **`.beforeFleet()` / `.afterFleet()` — hooks that run once per roll**, on the first replica and the last, at the same lifecycle points as `preDeploy`/`postDeploy`. `preDeploy` runs once *per replica*, so a migration there runs three times on a three-server fleet; `beforeFleet` is where migrations belong. `beforeFleet` lands while the old code still serves (expand), `afterFleet` once every replica is updated (contract). A roll that fails partway never reaches `afterFleet`. Deploy and `--dry-run` warn when a fleet app declares `preDeploy`.
+- **`placement: 'primary'` on a pm2 app** pins a process to the first server the app is declared on — crons and schedulers, where N replicas means every job fires N times. Primary is a property of the *server*, not the roll, so `deploy --on web-b` cannot promote a secondary and start a second scheduler. Rejected on a pm2 app with a port: the LB would keep routing to replicas serving nothing.
+- **`SHIPNODE_<ACCESSORY>_HOST` for cross-server accessories.** Docker networks are host-local, so no connection string was ever generated and the shipnode default of `localhost` silently pointed at nothing once the database moved to its own box. Every app is now handed the address of each accessory it `dependsOn` — `127.0.0.1` when co-located, the host's `privateHost` when not — so moving an accessory to its own server is a config change, not a code change. Declared env wins.
+- **`shipnode drain` / `shipnode undrain`** for manual rotation control: take a replica out without deploying it, or return one that a failed roll left behind.
+- **`--on <server>`** on `deploy`, `rollback`, `harden`, `status`, `logs`, and `env`.
+- **`shipnode status` reports fleet convergence** — each replica's release and rotation state, and an explicit warning naming which replicas are on which release when they disagree. A replica with no release, or one left drained, counts as not converged. Skipped under `--on`, where one observation proves nothing about the others.
 - **`shipnode deploy --watch` — the development loop.** Runs one normal deploy to establish a baseline, then watches the working tree: every save is rsynced into the live release, rebuilt, reloaded, and health-probed. Each cycle is incremental — rsync receives an explicit `--files-from` list of changed paths instead of scanning the whole tree, dependencies are reinstalled only when a manifest or lockfile changes, and the health probe uses exponential backoff (100ms → 1s) rather than the deploy path's fixed 3s delay plus 2s retry gaps. Falls back to a full-tree sync when a change set is large or contains a delete, which `--files-from` cannot express. Requires `--app <name>` in a multi-app workspace; mutually exclusive with `--dry-run`.
   - This is deliberately not a release: it patches the release that is already serving, so there is no rollback target, reload can drop in-flight requests, and local deletes do not propagate until the next full deploy. Watch mode says so on startup.
   - Blue-green apps reload only the colour serving traffic (via that release's `ecosystem.web.config.cjs`), leaving the idle colour untouched so it remains a valid rollback target.
@@ -17,6 +31,15 @@ All notable changes to `@devalade/shipnode` will be documented here.
   - The debounce has a max-wait ceiling (2s). A plain debounce resets on every event, so in a repo with a background writer — a turbo daemon, a framework's own watcher — changes would sit unemitted for as long as the writing continued, and the loop would appear hung.
 
 ### Fixed
+- **Postgres and Redis were installed on every server.** `database` and `redis` are workspace-level and had no `on` target, and the per-server config spread the whole workspace, so `shipnode setup` on a three-server workspace installed and initialised Postgres three times — same user, same database name. Both now take `on` and are stripped from every other server's scoped config.
+- **Cloudflare DNS clobbered other servers' domains.** `cloudflare init` iterated the whole workspace app list rather than the apps on the connected host, so whichever server ran it last repointed *every* domain at *its* tunnel — including apps living elsewhere, which then 404'd through the catch-all. DNS and ingress are now scoped to the apps actually on that host.
+- **Accessories could start after the apps that need them.** Servers were walked in `Object.entries(config.servers)` order and each started only its own accessories, so with the app server declared first the run was: deploy `api` → health-check `api` → *then* start Postgres. The first deploy failed, and reordering the `servers` literal silently fixed it. Servers are now traversed in dependency order derived from `dependsOn`, and the cross-server warning moved from `--dry-run` into the real deploy path. A dependency cycle falls back to declaration order rather than throwing.
+- **One server failing aborted the rest.** `runRemoteCommandForTargets` had its `try/catch` outside the loop, so server 2 failing skipped server 3 and left the user with N-1 "Done" banners, one error, and no statement of which servers succeeded. Failures are now isolated per server, collected, summarised, and exit non-zero. `--app` also filters the target loop instead of only validating, so an unrelated server's connection failure no longer kills an app-scoped command.
+- **`rollback` only rolled back one server.** It went through a single-connection runner, so on a three-replica fleet it restored one server and left the other two on the bad release. It now drives the same roller a deploy uses, and its confirmation is asked once up front rather than inside the loop — where the second prompt would have arrived with half the fleet already rolled back.
+- **`harden` only hardened the first server**, and its seven prompts sat inside the connection, so fanning out would have asked each question once per server with earlier servers already changed. The answers are collected up front and applied to every server; it also takes `--on`.
+- **`shipnode deploy --dry-run --app <name>` hid cross-server dependency warnings.** It resolved them against the app-scoped config, which no longer contains the accessory's server — so the preview was quietest exactly when the dependency was remote. The real deploy path was already correct.
+- **The generated CI workflow assumed one server** — one key, one `known_hosts` entry. A host missing from `known_hosts` surfaced as an SSH failure partway through a roll, with earlier replicas already updated and the failing one left drained. The workflow now verifies every replica is a known host before deploying and prints the exact `ssh-keyscan` command to fix it.
+- **An accessory published only on loopback but consumed from another server** is now a config error rather than a runtime connection failure: no connection string can reach `127.0.0.1` on another machine.
 - **SSH keepalives on long-lived sessions** — `deploy --watch` and `monitor` sit idle between commands, where a NAT or firewall timeout would silently drop the connection and fail the next exec. Connections now send keepalives every 15s, and `connect` starts from a fresh client so reconnecting doesn't accumulate listeners.
 - **`.env` symlinks survive a rebuild** — the build-output symlink logic moved to `envSymlinkCommand` and now re-runs after every hot-sync build, since a build that wipes and recreates its output directory takes the symlink with it. The initial deploy path is unchanged.
 

README.md

@@ -214,6 +214,86 @@ export default shipnode
   .build();
 ```
 
+### Horizontal scaling (fleets)
+
+Name several servers in one `on` target and the app becomes a **fleet**: the same app on N servers, deployed by rolling through them a batch at a time.
+
+```ts
+import { shipnode, app } from '@devalade/shipnode';
+
+export default shipnode
+  .servers({
+    'web-a': { host: '1.2.3.4', user: 'deploy', privateHost: '10.0.0.11' },
+    'web-b': { host: '1.2.3.5', user: 'deploy', privateHost: '10.0.0.12' },
+    'db-1':  { host: '1.2.3.6', user: 'deploy', privateHost: '10.0.0.20' },
+  })
+  .group('web', ['web-a', 'web-b'])
+  .deployTo('/var/www/example')
+  .accessories({
+    postgres: { image: 'postgres:16', on: 'db-1', port: '0.0.0.0:5432:5432' },
+  })
+  .apps([
+    app().backend().name('api').on('web').port(3333)
+      .domain('api.example.com')
+      .dependsOn(['postgres'])
+      .fleet({ batch: 1, drainWait: 30 })
+      .worker({ name: 'mailer', command: 'node dist/worker.js' })
+      .worker({ name: 'cron', command: 'node dist/cron.js', placement: 'primary' })
+      .beforeFleet(async ({ exec }) => { await exec('pnpm db:apply'); }),
+  ])
+  .build();
+```
+
+`shipnode deploy` now rolls: drain a replica → wait `drainWait` → deploy it → health-check → put it back → next.
+
+**You bring the load balancer.** Shipnode never talks to it. Point your Hetzner / DigitalOcean / ALB / nginx load balancer at each replica's `privateHost:80` and set its health check to **`/_shipnode/ready`**. That endpoint answers `200` normally and `503` while shipnode is deploying that replica, which is the entire integration — no provider APIs, no credentials.
+
+> **`drainWait` is the setting that matters.** Shipnode drains instantly; your load balancer only notices on its next health check, after its failure threshold. Shipnode cannot see either value, so you declare how long to wait. Set it below your LB's *interval × unhealthy-threshold* and the roll will drop requests. Verify it once by `curl`ing the LB in a loop during a deploy and confirming zero non-200s.
+
+Replicas serve plain HTTP on the private network and never claim the domain — five replicas all requesting a certificate for the same name would race Let's Encrypt. **TLS terminates at your load balancer.**
+
+| Setting | Default | Description |
+|---|---|---|
+| `.group(name, servers)` | — | Name a set of servers so `on` can target them all at once |
+| `privateHost` | — | The address the LB and other servers reach this box on. Required for every fleet member |
+| `.fleet({ batch })` | `1` | Replicas taken out of rotation at once |
+| `.fleet({ drainWait })` | `30` | Seconds to wait after draining, before touching the app |
+| `.fleet({ port })` | `80` | Port a replica serves on for the load balancer |
+| `.fleet({ readyPath })` | `/_shipnode/ready` | Readiness endpoint the LB health-checks |
+
+#### Hooks that must run once
+
+`preDeploy` runs **once per replica** — a migration there runs three times on a three-server fleet. Use `.beforeFleet()` (runs on the first replica, before the new code goes live anywhere) and `.afterFleet()` (runs on the last, once every replica is updated). A roll that fails partway never reaches `afterFleet`.
+
+#### Workers that must run once
+
+`.worker({ placement: 'primary' })` pins a process to the first server in the fleet. Use it for crons and schedulers, where N copies means every job fires N times. Queue workers competing for jobs want the default (`'all'`).
+
+#### Talking to accessories on another server
+
+Each app is given `SHIPNODE_<ACCESSORY>_HOST` for every accessory it `dependsOn` — `127.0.0.1` when co-located, the accessory server's `privateHost` when not. Read it in your connection string and moving Postgres to its own box becomes a config change:
+
+```ts
+const url = `postgres://app:${pw}@${process.env.SHIPNODE_POSTGRES_HOST}:5432/app`;
+```
+
+Bind the accessory somewhere reachable (`'0.0.0.0:5432:5432'`, not `'127.0.0.1:5432:5432'`) and run `shipnode harden` to open that port to the replicas that need it — and only to them.
+
+#### Fleet commands
+
+```bash
+shipnode deploy                      # roll every replica
+shipnode deploy --on web-a           # deploy one replica
+shipnode status                      # per-replica release + rotation, flags skew
+shipnode rollback --app api          # roll the rollback across the fleet
+shipnode drain   --app api --on web-a  # take one replica out of rotation
+shipnode undrain --app api --on web-a  # put it back
+```
+
+`shipnode status` is what tells you a roll died halfway: it compares replicas and reports which ones are on which release, and which are sitting out of rotation.
+
+See [ADR-0007](docs/adr/0007-fleet-replication.md) for the design and its limits.
+
 ### All options
 
 | Method | Default | Description |
@@ -254,16 +334,20 @@ shipnode deploy                # Deploy (zero-downtime by default)
 shipnode deploy --dry-run      # Preview without making changes
 shipnode deploy --skip-build   # Skip local build step
 shipnode deploy --watch        # Deploy once, then sync + reload on every save
+shipnode deploy --on web-a     # Deploy one server (or one replica of a fleet)
 shipnode doctor                # Check local + remote config
 shipnode doctor --security     # Run security audit
-shipnode status                # Show PM2 process status
+shipnode status                # PM2 status; for a fleet, per-replica release + skew
 ```
 
 ### Release management
 
 ```bash
-shipnode rollback              # Roll back to the previous release
+shipnode rollback              # Roll back to the previous release (rolls across a fleet)
 shipnode rollback --steps 3    # Roll back 3 releases
+shipnode rollback --on web-a   # Roll back one replica
+shipnode drain --app api --on web-a    # Take a replica out of rotation
+shipnode undrain --app api --on web-a  # Put it back
 shipnode migrate               # Migrate existing deploy to zero-downtime structure
 ```