Add rolling deploys across a fleet of servers

An app that resolves to several servers is now rolled a batch at a time
instead of deployed to all of them at once.

shipnode does not manage the load balancer and never talks to it. It owns
one thing: a readiness endpoint per replica that answers 200 normally and
503 while that replica is being deployed. Every managed LB decides rotation
from a health check, so flipping that endpoint is enough to pull a replica
out and put it back — no provider APIs, no LB for shipnode to keep alive,
and the same code works behind Hetzner, DigitalOcean, an ALB or someone
else's nginx.

The signal is a sentinel file rather than a rewritten Caddy config. Caddy
matches on its existence, so flipping takes effect with no reload and
survives a Caddy restart or a reboot — a drained replica stays drained until
something undrains it.

fleet.drainWait is the number that matters and the one shipnode cannot infer.
It has no way to know your LB's check interval or unhealthy threshold, so it
cannot detect when traffic has actually stopped; it flips the sentinel, waits
the declared seconds, and only then touches the app. Set it too low and the
roll drops requests that were already in flight. The dry run prints it as an
explicit step for that reason.

FleetOrchestrator sits above DeployOrchestrator rather than replacing it. A
replica's deploy is the ordinary single-server deploy — same releases, same
blue-green, same health check — so the composition is blue-green within a
replica, rolling across replicas. The invariant the new layer exists to hold
is that no more than `batch` replicas are ever out of rotation, and that a
replica is out of rotation before anything touches it. deployApp now takes a
fleet-wide releaseId, because a per-replica timestamp makes a converged fleet
indistinguishable from a half-rolled one.

A failure stops the roll and resolves rather than throwing: a partly-rolled
fleet is a real state to report, not an exception to unwind. Updated replicas
keep serving the new release, untouched ones the old, drained-but-undeployed
batch-mates go back into rotation, and the replica that actually failed stays
drained so no traffic reaches it and it can be inspected as-is.

Replicas stop claiming the app's public domain, which is the sharp edge of
replicating a Caddy config: five replicas all racing Let's Encrypt for one
name. TLS terminates at the load balancer and each replica serves plain HTTP
bound to its private address. Blue-green still flips that site's upstream, so
the two compose.

Also: --on <server> scopes deploy/status/logs/restart/stop/env/unlock to one
replica, `drain`/`undrain` give manual rotation control for inspecting a
replica without deploying it, and watch mode requires --on for a fleet app
since it holds one connection and patches one live release.

Running it against a real three-server config found four things unit tests
did not, each now covered:

- The dry run printed a fleet app once per replica, because getAppsForServer
  deliberately stopped being a partition.
- Its Caddy preview showed the public-domain site, which is precisely the
  file a replica does not write.
- The deploy flow omitted the drain steps, hiding the one setting the user
  has to get right.
- `--app api --on db-1` dialled a server api does not run on and failed with
  an SSH handshake timeout instead of saying so. --on is now checked against
  what was actually asked for, before any connection is opened.

preDeploy still runs once per replica and now warns loudly when it would run
N times. Moving migrations to a run-once hook is Phase 2.

Author: devalade

src/cli/commands/deploy.ts

@@ -9,12 +9,15 @@ import type { AccessoryConfig, ShipnodeConfig, ShipnodeApp } from '../../shared/
 import { accessoryMounts } from '../../services/accessory.service.js';
 import { getPm2Name } from '../../domain/pm2/apps.js';
 import { configForAppResult, configForServer, getServerTargets, resolveServerNames, resolveSingleServerNameResult } from '../../domain/servers.js';
-import { generateBackendCaddyfile, generateFrontendCaddyfile } from '../../services/caddy.service.js';
+import { generateBackendCaddyfile, generateFleetCaddyfile, generateFrontendCaddyfile } from '../../services/caddy.service.js';
+import { appStateDir } from '../../domain/deploy/drain.js';
 import { type ServerTargetError } from '../../shared/result-errors.js';
 import { runDeployWatch } from './deploy-watch.js';
 import type { BuildLocation } from '../../domain/deploy/hot-sync.js';
+import { deployFleet, type FleetEvent } from '../../domain/deploy/fleet.js';
+import { SshConnection } from '../../infrastructure/ssh/connection.js';
 
-export async function cmdDeploy(cwd: string, options: { dryRun?: boolean; skipBuild?: boolean; app?: string; config?: string; watch?: boolean; build?: string }): Promise<void> {
+export async function cmdDeploy(cwd: string, options: { dryRun?: boolean; skipBuild?: boolean; app?: string; config?: string; watch?: boolean; build?: string; on?: string }): Promise<void> {
   let config: ShipnodeConfig;
   let targetConfig: ShipnodeConfig;
   try {
@@ -52,18 +55,45 @@ export async function cmdDeploy(cwd: string, options: { dryRun?: boolean; skipBu
     return;
   }
 
+  // Check --on against what was actually asked for, before opening any
+  // connection. Otherwise `--app api --on db-1` dials a server api does not run
+  // on and fails with an SSH timeout instead of saying so.
+  if (options.on) {
+    const reachable = getServerTargets(targetConfig).map((target) => target.name);
+    if (!reachable.includes(options.on)) {
+      ui.error(
+        options.app
+          ? `App '${options.app}' does not run on '${options.on}'. It runs on: ${reachable.join(', ')}`
+          : `Unknown server target '${options.on}'. Known targets: ${reachable.join(', ')}`,
+      );
+      process.exit(1);
+      return;
+    }
+  }
+
   // The per-server config the callback receives only carries that server's own
   // accessories, so cross-server dependencies are invisible from inside it.
   // Dependency warnings have to be resolved against the whole workspace.
   const workspaceConfig = config;
-
-  await runRemoteCommandForTargets(
+  const fleetApps = targetConfig.apps.filter((candidate) => candidate.fleet);
+  const soloApps = targetConfig.apps.filter((candidate) => !candidate.fleet);
+
+  // Naming a fleet app leaves the fan-out with nothing to do — rolling it is
+  // the whole job, and visiting other servers would deploy things not asked for.
+  const fanOut = soloApps.length > 0 || !options.app;
+
+  // Accessories and single-server apps go server by server, as they always
+  // have. Fleet apps cannot: they appear on several servers at once and must be
+  // rolled one batch at a time, so they are held back and handled after — which
+  // also means the accessories they depend on are already up.
+  if (fanOut) await runRemoteCommandForTargets(
     cwd,
     async ({ config, executor, serverName }) => {
-      const app = options.app ? config.apps.find((candidate) => candidate.name === options.app) : undefined;
+      const serverApps = config.apps.filter((candidate) => !candidate.fleet);
+      const app = options.app ? serverApps.find((candidate) => candidate.name === options.app) : undefined;
       const deployConfig = options.app
         ? { ...config, apps: app ? [app] : [] }
-        : config;
+        : { ...config, apps: serverApps };
       if (deployConfig.apps.length === 0 && Object.keys(deployConfig.accessories ?? {}).length === 0) return;
 
       ui.banner();
@@ -88,12 +118,116 @@ export async function cmdDeploy(cwd: string, options: { dryRun?: boolean; skipBu
       }
 
       ui.note(lines.join('\n'), 'Done');
-      ui.outro('Run shipnode status to check your app.');
     },
-    { configPath: options.config },
+    { configPath: options.config, serverName: options.on, appName: options.app },
+  );
+
+  for (const fleetApp of fleetApps) {
+    await rollFleetApp(cwd, workspaceConfig, fleetApp, options);
+  }
+
+  ui.outro('Run shipnode status to check your app.');
+}
+
+/**
+ * Roll one app across its replicas.
+ *
+ * Each replica gets the workspace narrowed to just this app and just that
+ * server, so the deploy that runs on it is exactly the single-server deploy —
+ * same orchestrator, same blue-green, same health check. Accessories are
+ * stripped because the fan-out above has already ensured them; leaving them in
+ * would restart the database once per replica.
+ */
+async function rollFleetApp(
+  cwd: string,
+  config: ShipnodeConfig,
+  app: ShipnodeApp,
+  options: { skipBuild?: boolean; on?: string },
+): Promise<void> {
+  const scoped = configForAppResult(config, app.name);
+  if (scoped.isErr()) {
+    ui.error(scoped.error.message);
+    process.exit(1);
+    return;
+  }
+
+  const appConfig = scoped.value;
+  let replicas = getServerTargets(appConfig).map((target) => target.name);
+  if (options.on) {
+    if (!replicas.includes(options.on)) {
+      ui.error(`App '${app.name}' does not run on '${options.on}'. It runs on: ${replicas.join(', ')}`);
+      process.exit(1);
+      return;
+    }
+    replicas = [options.on];
+  }
+
+  ui.banner();
+  ui.step(`Rolling ${chalk.bold(app.name)} across ${replicas.join(', ')}`);
+
+  if (app.hooks?.preDeploy) {
+    ui.warn(
+      `${app.name} declares a preDeploy hook, which runs once per replica — ` +
+      `${replicas.length} times for this roll. Database migrations belong somewhere that runs once.`,
+    );
+  }
+
+  for (const warning of renderDependencyWarnings(config, app)) ui.warn(warning);
+
+  const result = await deployFleet({
+    app,
+    fleet: app.fleet!,
+    replicas,
+    remotePath: appConfig.remotePath,
+    connect: async (serverName) => {
+      const ssh = new SshConnection();
+      await ssh.connect(configForServer(appConfig, serverName).ssh);
+      return { executor: ssh, close: () => ssh.disconnect() };
+    },
+    deployReplica: async ({ serverName, executor, releaseId }) => {
+      const replicaConfig = { ...configForServer(appConfig, serverName), apps: [app], accessories: {} };
+      const deployer = new DeployService(new LoggingExecutor(executor), replicaConfig);
+      await deployer.execute(cwd, options.skipBuild ?? false, releaseId);
+    },
+    onEvent: (event) => reportFleetEvent(app, event),
+  });
+
+  if (result.failed) {
+    ui.error(
+      `${app.name}: ${result.failed.server} failed and is out of rotation. ` +
+      `${result.deployed.length ? `${result.deployed.join(', ')} now on ${result.releaseId}; ` : ''}` +
+      `${result.skipped.length ? `${result.skipped.join(', ')} still on the previous release. ` : ''}` +
+      `The fleet is running mixed versions.`,
+    );
+    process.exit(1);
+    return;
+  }
+
+  ui.note(
+    [`release  ${result.releaseId}`, `servers  ${result.deployed.join(', ')}`, ...(app.domain ? [`url      https://${app.domain}`] : [])].join('\n'),
+    `${app.name} rolled`,
   );
 }
 
+function reportFleetEvent(app: ShipnodeApp, event: FleetEvent): void {
+  switch (event.type) {
+    case 'drained':
+      ui.info(`${event.servers.join(', ')} draining — waiting ${event.waitSeconds}s for the load balancer`);
+      break;
+    case 'deploying':
+      ui.step(`${app.name} → ${event.server}`);
+      break;
+    case 'undrained':
+      ui.success(`${event.server} back in rotation`);
+      break;
+    case 'failed':
+      ui.error(`${event.server}: ${event.message}`);
+      break;
+    default:
+      break;
+  }
+}
+
 /**
  * Where each watch cycle builds.
  *
@@ -122,7 +256,7 @@ function resolveBuildLocation(
 async function startWatch(
   cwd: string,
   config: ShipnodeConfig,
-  options: { skipBuild?: boolean; app?: string; build?: string },
+  options: { skipBuild?: boolean; app?: string; build?: string; on?: string },
 ): Promise<void> {
   const appName = options.app ?? (config.apps.length === 1 ? config.apps[0]?.name : undefined);
 
@@ -144,9 +278,32 @@ async function startWatch(
     return;
   }
 
-  const watchConfig = scoped.value;
+  let watchConfig = scoped.value;
   const app = watchConfig.apps[0];
 
+  // A watch session holds one SSH connection and reloads one process set, so a
+  // fleet app has to be narrowed to a single replica. Patching the live release
+  // of every replica in lockstep is not something this loop can promise.
+  const replicas = getServerTargets(watchConfig).map((target) => target.name);
+  if (replicas.length > 1) {
+    if (!options.on) {
+      ui.error(
+        `'${appName}' runs on ${replicas.length} servers (${replicas.join(', ')}). ` +
+        `Watch mode patches one live release — pick a replica with --on <server>.`,
+      );
+      process.exit(1);
+      return;
+    }
+    if (!replicas.includes(options.on)) {
+      ui.error(`'${appName}' does not run on '${options.on}'. It runs on: ${replicas.join(', ')}`);
+      process.exit(1);
+      return;
+    }
+  }
+  if (options.on) {
+    watchConfig = configForServer(watchConfig, options.on);
+  }
+
   const buildLocation = resolveBuildLocation(options, app);
   if (!buildLocation) {
     ui.error(`--build must be one of: remote, local, none (got "${options.build}")`);
@@ -215,19 +372,30 @@ function renderAppPlan(config: ShipnodeConfig, app: ShipnodeApp, skipBuild: bool
     buildRows.push(['', chalk.dim('runs on remote server')]);
   }
 
-  const steps: string[] = [
+  const steps: string[] = [];
+  if (app.fleet) {
+    const batch = app.fleet.batch === 1 ? 'one replica' : `${app.fleet.batch} replicas`;
+    steps.push(
+      `Drain ${batch} (${app.fleet.readyPath} → 503)`,
+      `Wait ${app.fleet.drainWait}s for the load balancer to notice`,
+    );
+  }
+  steps.push(
     'Acquire deploy lock',
     'Create release directory',
     'Rsync files',
     'Install dependencies',
-  ];
+  );
   if (app.hooks?.preDeploy) steps.push('Run preDeploy hook');
   steps.push('Switch symlink (atomic)');
   if (app.appType === 'backend') steps.push('Reload PM2');
   if (app.healthCheck.enabled) steps.push(`Health check ${app.healthCheck.path}`);
   steps.push('Record release', 'Clean old releases');
   if (app.hooks?.postDeploy) steps.push('Run postDeploy hook');
   steps.push('Release lock');
+  if (app.fleet) {
+    steps.push(`Undrain (${app.fleet.readyPath} → 200)`, 'Repeat for the next batch');
+  }
 
   const flowRows: [string, string][] = steps.map((step, i) => [`${i + 1}.`, step]);
 
@@ -268,11 +436,27 @@ function renderDependencyWarnings(config: ShipnodeConfig, app: ShipnodeApp): str
 }
 
 function renderCaddyPreview(config: ShipnodeConfig, app: ShipnodeApp): string | null {
+  const servePath = `${config.remotePath}/${app.name}/current`;
+  const web = app.pm2?.apps.find((pm2App) => pm2App.port !== undefined);
+
+  // A replica serves a private port and never claims the domain — showing the
+  // public site here would preview a file shipnode is not going to write.
+  if (app.fleet) {
+    const [firstReplica] = resolveServerNames(config, app.on);
+    return generateFleetCaddyfile(app, {
+      listen: app.fleet.port,
+      bind: firstReplica ? config.servers[firstReplica]?.privateHost : undefined,
+      upstream: web?.port,
+      servePath: app.appType === 'frontend' ? servePath : undefined,
+      readyPath: app.fleet.readyPath,
+      stateDir: appStateDir(config.remotePath, app.name),
+    });
+  }
+
   if (!app.domain) return null;
   if (app.appType === 'frontend') {
-    return generateFrontendCaddyfile(app, `${config.remotePath}/${app.name}/current`);
+    return generateFrontendCaddyfile(app, servePath);
   }
-  const web = app.pm2?.apps.find((pm2App) => pm2App.port !== undefined);
   if (!web?.port) return null;
   return generateBackendCaddyfile(app, web.port);
 }
@@ -289,9 +473,10 @@ export function printDryRun(config: ShipnodeConfig, skipBuild: boolean): void {
     `  ${chalk.dim('Apps'.padEnd(14))} ${config.apps.map((a) => a.name).join(', ')}`,
   ].join('\n');
 
-  const perApp = getServerTargets(config)
-    .map((target) => configForServer(config, target.name))
-    .flatMap((targetConfig) => targetConfig.apps.map((app) => renderAppPlan(config, app, skipBuild)))
+  // Per app, not per server: a fleet app runs on several servers and would
+  // otherwise be printed once for each of them.
+  const perApp = config.apps
+    .map((app) => renderAppPlan(config, app, skipBuild))
     .join('\n\n');
 
   const accessories = renderAccessoriesPlan(config);

src/cli/commands/drain.ts

@@ -0,0 +1,84 @@
+import { runRemoteCommandForTargets } from '../runner.js';
+import { ui } from '../ui.js';
+import { drain, isDrained, undrain } from '../../domain/deploy/drain.js';
+
+/**
+ * Manual rotation control.
+ *
+ * Deploying already drains and undrains each replica as it rolls. These exist
+ * for the times you want a replica out of the pool without deploying it —
+ * reproducing a bug on live data, or holding a replica back after a failed
+ * roll left it drained.
+ */
+export async function cmdDrain(
+  cwd: string,
+  options: { app?: string; on?: string; config?: string },
+): Promise<void> {
+  await setRotation(cwd, options, true);
+}
+
+export async function cmdUndrain(
+  cwd: string,
+  options: { app?: string; on?: string; config?: string },
+): Promise<void> {
+  await setRotation(cwd, options, false);
+}
+
+async function setRotation(
+  cwd: string,
+  options: { app?: string; on?: string; config?: string },
+  drained: boolean,
+): Promise<void> {
+  let touched = 0;
+
+  await runRemoteCommandForTargets(
+    cwd,
+    async ({ config, executor, serverName }) => {
+      for (const app of config.apps) {
+        if (options.app && app.name !== options.app) continue;
+        if (!app.fleet) {
+          ui.warn(`${app.name} is not behind a load balancer — nothing to drain.`);
+          continue;
+        }
+
+        if (drained) {
+          await drain(executor, config.remotePath, app.name);
+          ui.success(`${app.name} on ${serverName} is draining (${app.fleet.readyPath} now answers 503)`);
+        } else {
+          await undrain(executor, config.remotePath, app.name);
+          ui.success(`${app.name} on ${serverName} is back in rotation`);
+        }
+        touched += 1;
+      }
+    },
+    { configPath: options.config, appName: options.app, serverName: options.on },
+  );
+
+  if (touched === 0) {
+    ui.warn('No fleet apps matched.');
+    return;
+  }
+
+  if (drained) {
+    ui.info('Your load balancer removes it once its own health check notices — that is what fleet.drainWait covers.');
+  }
+}
+
+/** Drain state per replica, for `status`. */
+export async function readDrainState(
+  cwd: string,
+  options: { app?: string; config?: string },
+): Promise<void> {
+  await runRemoteCommandForTargets(
+    cwd,
+    async ({ config, executor, serverName }) => {
+      for (const app of config.apps) {
+        if (options.app && app.name !== options.app) continue;
+        if (!app.fleet) continue;
+        const out = await isDrained(executor, config.remotePath, app.name);
+        ui.info(`${app.name} on ${serverName}: ${out ? 'draining' : 'in rotation'}`);
+      }
+    },
+    { configPath: options.config, appName: options.app },
+  );
+}