Document what the live fleet test changed

A replica answering only on its private address was documented as the
design; it was a bug. Load balancers forward the client's Host, so the
README now tells you to give a fleet app a .domain(), ADR-0007 records
why the site takes two addresses and why the http:// scheme on the domain
is load-bearing, and the CHANGELOG lists the five setup/deploy bugs the
live run surfaced.

Author: devalade

CHANGELOG.md

@@ -9,7 +9,7 @@ All notable changes to `@devalade/shipnode` will be documented here.
   - **Shipnode does not manage the load balancer.** You provision it (Hetzner, DigitalOcean, ALB, nginx); shipnode owns one readiness endpoint per replica, `/_shipnode/ready`, which answers `200` normally and `503` while that replica deploys. Point your LB's health check at it. No provider APIs, no credentials — SSH remains the only requirement.
   - **`fleet.drainWait` (default 30s) is the setting that matters.** Shipnode drains instantly; your LB only notices on its next health check, after its failure threshold. Shipnode can see neither, so you declare the wait. Below your LB's *interval × unhealthy-threshold* and the roll drops requests.
   - Draining is a sentinel file matched by Caddy, not a config rewrite: one `touch`, no reload, immediate, and it survives a Caddy or host restart — so a replica drained by a failed roll stays drained rather than quietly rejoining the pool.
-  - Fleet replicas serve plain HTTP on `privateHost:fleet.port` and never claim the app's domain. Five replicas all requesting a certificate for one name would race Let's Encrypt. **TLS terminates at your load balancer**; pass-through TLS is out of scope.
+  - Fleet replicas serve plain HTTP and never request a certificate — five replicas asking for one name would race Let's Encrypt. **TLS terminates at your load balancer**; pass-through TLS is out of scope. Each replica answers on both its `privateHost` (which the LB's health check dials) and the app's domain (which the LB forwards from the client), so give a fleet app a `.domain()`.
   - `.group(name, servers)` names a set of servers; `on` accepts a server, a group, or a list of either. Every fleet member must declare `privateHost`.
   - A failed roll stops and reports the mixed-version fleet rather than claiming success. Updated replicas keep serving the new release, untouched ones the old, batch-mates drained but never touched go back into rotation, and **the failed replica stays drained** — out of rotation and inspectable as it died.
   - One release id for the whole roll, so `shipnode status` can distinguish a converged fleet from a half-rolled one.
@@ -31,6 +31,11 @@ All notable changes to `@devalade/shipnode` will be documented here.
   - The debounce has a max-wait ceiling (2s). A plain debounce resets on every event, so in a repo with a background writer — a turbo daemon, a framework's own watcher — changes would sit unemitted for as long as the writing continued, and the loop would appear hung.
 
 ### Fixed
+- **`shipnode setup` failed at PM2 startup on the documented first run.** Deploy-user commands ran from root's home: `sudo` inherits the caller's working directory, setup runs over SSH as root, and `/root` is mode 700 — so Node's `spawn()` returned EACCES and the error named the node binary rather than the cause. Commands now run with `sudo -H` from the target user's home.
+- **`shipnode setup --no-deploy-user` was silently ignored.** Commander turns `--no-deploy-user` into `deployUser: false`, not `noDeployUser: true`, so the flag read as undefined and the user was created anyway.
+- **Caddy was never installed on fleet replicas.** Both install gates tested `app.domain`, and a fleet replica deliberately has none — but it still needs Caddy for the readiness endpoint. The first deploy died writing its site file into a directory that was never created.
+- **Caddy sites were written but never reloaded.** Only the blue-green path called `reload`, and it does so for its own colour flip — so a frontend, a recreate backend, and every fleet replica wrote a site file that never took effect until something else reloaded Caddy.
+- **The health check demanded primary-pinned processes on every replica.** A `placement: 'primary'` worker is absent from secondary replicas by design, so placement working correctly is what failed the health check there.
 - **Postgres and Redis were installed on every server.** `database` and `redis` are workspace-level and had no `on` target, and the per-server config spread the whole workspace, so `shipnode setup` on a three-server workspace installed and initialised Postgres three times — same user, same database name. Both now take `on` and are stripped from every other server's scoped config.
 - **Cloudflare DNS clobbered other servers' domains.** `cloudflare init` iterated the whole workspace app list rather than the apps on the connected host, so whichever server ran it last repointed *every* domain at *its* tunnel — including apps living elsewhere, which then 404'd through the catch-all. DNS and ingress are now scoped to the apps actually on that host.
 - **Accessories could start after the apps that need them.** Servers were walked in `Object.entries(config.servers)` order and each started only its own accessories, so with the app server declared first the run was: deploy `api` → health-check `api` → *then* start Postgres. The first deploy failed, and reordering the `servers` literal silently fixed it. Servers are now traversed in dependency order derived from `dependsOn`, and the cross-server warning moved from `--dry-run` into the real deploy path. A dependency cycle falls back to declaration order rather than throwing.

README.md

@@ -250,7 +250,9 @@ export default shipnode
 
 > **`drainWait` is the setting that matters.** Shipnode drains instantly; your load balancer only notices on its next health check, after its failure threshold. Shipnode cannot see either value, so you declare how long to wait. Set it below your LB's *interval × unhealthy-threshold* and the roll will drop requests. Verify it once by `curl`ing the LB in a loop during a deploy and confirming zero non-200s.
 
-Replicas serve plain HTTP on the private network and never claim the domain — five replicas all requesting a certificate for the same name would race Let's Encrypt. **TLS terminates at your load balancer.**
+Replicas serve plain HTTP and never request a certificate — five replicas all asking for the same name would race Let's Encrypt. **TLS terminates at your load balancer.**
+
+> **Give a fleet app a `.domain()`.** Your load balancer forwards the client's `Host` header, so each replica has to recognise it. Shipnode configures the replica to answer on both its `privateHost` (for the LB's health check) and the domain (for forwarded traffic) — over plain HTTP, so no replica ever requests a certificate. Without a domain, only requests addressed to the private IP reach your app.
 
 Each replica installs and builds, so a three-replica roll takes roughly three times a single deploy. `shipnode ci github` sizes the workflow timeout accordingly.
 

docs/adr/0007-fleet-replication.md

@@ -28,10 +28,20 @@ The alternative was rewriting the site config and reloading Caddy, which the blu
 
 `generateBackendCaddyfile` emits `api.example.com { reverse_proxy localhost:3000 }`. Put that on five boxes and all five race Let's Encrypt for a certificate on the same name.
 
-A fleet replica instead serves `http://<privateHost>:<fleet.port>` — plain HTTP on the private network, no ACME, no domain. **TLS terminates at your load balancer**, which is how a managed LB is normally run. Pass-through TLS, where each replica holds the certificate, is out of scope; the plan says so rather than half-supporting it.
+A fleet replica instead serves plain HTTP, no ACME, no certificate. **TLS terminates at your load balancer**, which is how a managed LB is normally run. Pass-through TLS, where each replica holds the certificate, is out of scope; the plan says so rather than half-supporting it.
 
 This is why every server in a fleet must declare `privateHost`. The schema enforces it.
 
+The replica's site answers to **two** addresses, and the reason is worth recording because the first implementation got it wrong:
+
+```
+http://10.0.0.11:80, http://api.example.com:80 { … }
+```
+
+Two different `Host` headers arrive on that port. The load balancer's health check dials the replica directly, so it sends the private address. Forwarded client traffic carries whatever the *client* asked for, which is the app's domain — load balancers pass the original `Host` through. Binding only the private address produced a fleet where the health check passed on every replica and every real request fell through to whatever else held port 80; on a stock install that is Caddy's welcome page, served with a cheerful `200`.
+
+The `http://` scheme on the domain is load-bearing. Without it Caddy would try to provision a certificate for that name, which is the ACME race this whole section exists to avoid. Declaring the domain does not claim it — serving it over TLS would.
+
 ## Blue-green within a replica, rolling across replicas
 
 `DeployOrchestrator` was already the right unit of work for one replica and needed no rewrite. Blue-green, health checks, release symlinks and locking all stay intra-replica; the colour flip now rewrites the fleet site's upstream port instead of the public domain's.