Add blue-green zero-downtime deploys with safer failure recovery.

Opt-in colour swap behind Caddy keeps the old release serving until health passes; workers reload only after that, failed deploys revert current and record history, and the deploy lock is atomic via mkdir.

Author: devalade

.claude/skills/shipnode/REFERENCE.md

@@ -92,8 +92,7 @@ All commands accept `--config <path>` to use a non-default config file.
 | `.nodeVersion(v)` | `lts` | Node version via mise |
 | `.pkgManager(pm)` | auto-detected | `npm` \| `yarn` \| `pnpm` \| `bun` |
 | `.buildDir(dir)` | auto-detected | Frontend build output directory |
-| `.zeroDowntime({ keepReleases? })` | true, 5 releases | Zero-downtime mode |
-| `.legacy()` | — | In-place rsync deploy, no rollback |
+| `.zeroDowntime(altPort?)` | off; green = web port + 1 | Blue-green releases: boot the idle colour, health-check it, reload workers, then flip Caddy. Requires a web app (a `.port`). `altPort` sets the green port. |
 | `.healthCheck(path, opts?)` | `/health`, 30s, 3 retries | Post-deploy check |
 | `.noHealthCheck()` | — | Skip health check |
 | `.envFile(f)` | `.env` | Local .env file to upload |

.claude/skills/shipnode/SKILL.md

@@ -43,9 +43,9 @@ Frontend app: swap `.backend()` → `.frontend()`, add `.buildDir('dist')`, drop
 3. `shipnode env` — uploads .env
 4. `shipnode deploy`
 
-### Zero-downtime is the default
+### Releases + zero-downtime
 Releases live in `/remotePath/releases/<timestamp>/`. Current symlink is atomically switched.
-To opt out: add `.legacy()` to the builder.
+Backend apps can opt into blue-green with `.zeroDowntime()` — boot idle colour, health-check, then flip Caddy.
 
 ### Rollback
 ```bash

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All notable changes to `@devalade/shipnode` will be documented here.
 
+## [Unreleased]
+
+### Added
+- **Blue-green (zero-downtime) releases** — opt a backend web app in with `.zeroDowntime(altPort?)`. The new release boots on the idle colour's port while the old colour keeps serving; the health check runs against the new colour, and only on success does Caddy's upstream flip via a graceful `systemctl reload caddy` (drains in-flight requests, drops nothing). A failed health check leaves the old colour serving untouched — zero user impact on a bad release. Rollback becomes an instant Caddy flip back to the still-running previous colour (no restart). Off by default; the recreate path is unchanged. Blue = the web app's `port`, green = `altPort` (default `port + 1`). Web app runs ~2× (both colours resident between deploys); workers stay a single reloaded set. See ADR-0005.
+
+### Fixed
+- **Blue-green workers wait for health** — workers reload only after the new colour passes health (and before the Caddy flip), so a bad release no longer leaves workers on new code while traffic stays on the old colour.
+- **Failed deploys revert `current` and record history** — on failure after the symlink switch, `current` is restored to the previous release (when one exists) and a `status: 'failed'` entry is written to `releases.json`.
+- **Atomic deploy lock** — lock acquisition uses `mkdir` (create-or-fail) instead of a TOCTOU file check. Path is still `.shipnode/deploy.lock` (now a directory); `unlock` / monitor handle both directory and legacy file shapes.
+
 ## [3.1.0] - 2026-07-02
 
 First real production run of 3.0 surfaced a batch of bugs — all fixed — plus one new feature (incremental restic backups to Cloudflare R2 / any S3-compatible store).

CONTEXT.md

@@ -27,7 +27,7 @@ A symlink pointing to the active release. In zero-downtime mode, `/remotePath/cu
 Owns release directory creation, symlink switching, release recording, and cleanup. Operates through a `RemoteExecutor`.
 
 ### DeployLock
-A PID-based lock file preventing concurrent deployments. Lives at `/remotePath/.shipnode/deploy.lock`.
+An atomic `mkdir`-based lock preventing concurrent deployments. Lives at `/remotePath/.shipnode/deploy.lock/` (directory). Legacy file-shaped locks are still recognised.
 
 ### Zero-Downtime
 The deployment mode where a new release is staged, then atomically switched via symlink. The old release remains available until the switch succeeds and health checks pass.
@@ -36,7 +36,7 @@ The deployment mode where a new release is staged, then atomically switched via
 The deployment mode where files are rsynced directly to the remote path. No releases, no symlinks, no rollback.
 
 ### DeployOrchestrator
-Owns the invariant deployment sequence: lock → release → stage → setup → hooks → symlink → start → health check → record → cleanup → unlock. Knows nothing about app-specific details.
+Owns the invariant deployment sequence: lock → release → stage → setup → hooks → symlink → start → health check → afterHealthy → caddy flip → record → cleanup → unlock. On failure: revert `current` symlink (when a previous release exists) and record a failed release. Knows nothing about app-specific details.
 
 ### DeploymentStrategy
 An adapter that knows how to stage, prepare, and start a specific kind of application (backend, frontend, etc.). Plugs into the orchestrator at fixed lifecycle hooks.

docs/adr/0005-blue-green-zero-downtime.md

@@ -0,0 +1,33 @@
+# Blue-green releases via a two-port colour swap behind Caddy
+
+The default deploy strategy stops the whole PM2 process set and starts the new release on the same port (`pm2 delete … && pm2 start …`). Between the delete and the moment the new process is listening — install relink, framework boot, first health probe — the port is closed and inbound requests are refused. For a plain web app that is a multi-second window of dropped requests on every deploy.
+
+`zeroDowntime` opts a backend web app into blue-green releases that close that window.
+
+## Mechanism
+
+Two ports, named **blue** and **green**. Blue is the web app's configured `port`; green is `altPort` (default `port + 1`). Exactly one colour is *active* at a time — Caddy's `reverse_proxy` upstream points at it. The active colour and the port pair are persisted on the host in `<appPath>/.shipnode/deploy-state.json`.
+
+Each deploy:
+
+1. Stages the release and switches the `current` symlink (as always). The previously-started colour keeps running the old code in memory — the symlink move doesn't touch a running process.
+2. Boots the **idle** colour on its own port (`pm2 start ecosystem.web.cjs`), reaping any stale same-colour instance first. The active colour is never touched. Workers are **not** reloaded yet.
+3. Health-checks the new colour on **its** port and colour-suffixed pm2 name (`api-green`).
+4. Only on success: reloads the single worker set against the new release, then rewrites the Caddy site to the new colour's port and `systemctl reload caddy` — a graceful reload that drains in-flight requests, so the flip drops nothing. Then persists the new active colour.
+
+A failed health check throws before step 4: Caddy is untouched, workers stay on the previous release, the old colour is still serving, `current` is reverted to the previous release when one exists, and the failed colour is reaped by the next deploy. Zero user impact on a bad release — the main prize.
+
+## Why the web app is duplicated but workers are not
+
+Blue-green needs two copies of the *web* app (old + new, different ports) resident at once. Workers have no port and aren't behind Caddy; running two copies would double-process their queues. So the ecosystem is split: `ecosystem.web.cjs` holds only the colour-suffixed web app; `ecosystem.workers.cjs` holds the workers as a single set, reloaded in place **after** health passes (a brief worker blip on a successful deploy is acceptable — they're not serving HTTP). Reloading before health would leave workers on a bad release while traffic stayed on the old colour.
+
+## Rollback is a flip, not a restart
+
+Because the previous colour is still running, rollback is an instant Caddy flip back to it plus a state swap — no process restart, nothing dropped. Only one step back is supported (older colours were reaped); deeper rollbacks redeploy the target release the normal way.
+
+## Trade-offs
+
+- **~2× memory for the web app** — both colours are resident between deploys. Documented; the price of instant rollback.
+- **The port pair is fixed at the first deploy** and persisted. Later changes to the web port or `altPort` in config are ignored until the state file is cleared, so a running colour is never silently re-homed.
+- **Migration costs one restart.** The first `zeroDowntime` deploy deletes the pre-existing uncoloured web process (which holds the blue == web port) by name so the target colour can bind. After that, every deploy is zero-drop.
+- **Opt-in, default off.** The recreate path is byte-for-byte unchanged, so existing deployments are unaffected until they set `zeroDowntime`.

src/cli/commands/rollback.ts

@@ -1,11 +1,21 @@
 import { runRemoteCommandForConfig } from '../runner.js';
 import { ReleaseManager } from '../../domain/release/manager.js';
 import { HealthCheckService } from '../../services/health.service.js';
+import { CaddyService } from '../../services/caddy.service.js';
 import { ui } from '../ui.js';
 import { confirm } from '../prompt.js';
 import { loadConfig } from '../../config/loader.js';
 import { getActiveApp } from '../../domain/workspace.js';
-import { getDeploymentName, getEcosystemPath } from '../../domain/pm2/apps.js';
+import { getDeploymentName, getEcosystemPath, getWebApp } from '../../domain/pm2/apps.js';
+import {
+  readDeployState,
+  writeDeployState,
+  otherColor,
+  portFor,
+  coloredWebName,
+} from '../../domain/deploy/blue-green.js';
+import type { RemoteExecutor } from '../../domain/remote/executor.js';
+import type { ShipnodeConfig, ShipnodeApp } from '../../shared/types.js';
 import { configForAppResult } from '../../domain/servers.js';
 
 export async function cmdRollback(
@@ -30,9 +40,17 @@ export async function cmdRollback(
     async ({ config, executor }) => {
       const app = getActiveApp(config, options.app);
       const appPath = `${config.remotePath}/${app.name}`;
-      const releases = new ReleaseManager(executor, appPath, app.keepReleases);
       const stepsBack = options.steps ?? 1;
 
+      // Blue-green apps roll back by flipping Caddy back to the previous colour,
+      // which is still running — instant and zero-drop. No pm2 restart.
+      if (app.appType === 'backend' && app.zeroDowntime) {
+        await rollbackBlueGreen(executor, config, app, appPath, stepsBack);
+        return;
+      }
+
+      const releases = new ReleaseManager(executor, appPath, app.keepReleases);
+
       ui.info('Fetching release history...');
       const allReleases = await releases.listReleases();
 
@@ -89,3 +107,75 @@ export async function cmdRollback(
     },
   );
 }
+
+/**
+ * Instant blue-green rollback: the previous colour is still resident, so we
+ * flip Caddy's upstream back to it and swap the persisted active colour. No
+ * process restart, no dropped requests.
+ *
+ * Only one step back is possible — older colours were reaped by later deploys.
+ * For anything deeper, redeploy the desired release instead.
+ */
+async function rollbackBlueGreen(
+  executor: RemoteExecutor,
+  config: ShipnodeConfig,
+  app: ShipnodeApp,
+  appPath: string,
+  stepsBack: number,
+): Promise<void> {
+  if (stepsBack !== 1) {
+    throw new Error(
+      `Blue-green rollback only supports one step (the live previous colour). ` +
+      `To go further back, redeploy the desired release.`,
+    );
+  }
+
+  const state = await readDeployState(executor, appPath);
+  if (!state) {
+    throw new Error('No blue-green state found on the server — nothing to roll back to.');
+  }
+
+  const webApp = getWebApp({ ...config, apps: [app] } as ShipnodeConfig);
+  if (!webApp) {
+    throw new Error('No web app (pm2 app with a port) found for this deployment.');
+  }
+  const namespace = app.pm2?.apps[0]?.name ?? app.name;
+  const previous = otherColor(state.activeColor);
+  const previousPort = portFor(previous, state);
+  const previousName = coloredWebName(namespace, webApp.name, previous);
+
+  // The previous colour must still be online to serve traffic after the flip.
+  // Parse pm2's JSON in-process rather than relying on `node` being on the
+  // remote PATH at rollback time.
+  const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`;
+  const jlist = await executor.exec(`${mise} && mise exec -- pm2 jlist`);
+  let online = false;
+  try {
+    const entries = JSON.parse(jlist.stdout.trim()) as Array<{ name: string; pm2_env?: { status?: string } }>;
+    online = entries.some((e) => e.name === previousName && e.pm2_env?.status === 'online');
+  } catch {
+    online = false;
+  }
+  if (!online) {
+    throw new Error(
+      `Previous colour "${previousName}" is not running — cannot instant-rollback. ` +
+      `Redeploy the desired release instead.`,
+    );
+  }
+
+  ui.warn(`Active colour:   ${state.activeColor} (port ${portFor(state.activeColor, state)})`);
+  ui.warn(`Rollback target: ${previous} (port ${previousPort})`);
+
+  const ok = await confirm('Flip traffic back to the previous colour?');
+  if (!ok) {
+    ui.info('Rollback cancelled.');
+    return;
+  }
+
+  const caddy = new CaddyService(executor, config);
+  await caddy.configureBackend(app, previousPort);
+  await caddy.reload();
+  await writeDeployState(executor, appPath, { ...state, activeColor: previous });
+
+  ui.success(`Rolled back — traffic now on ${previous} (port ${previousPort})`);
+}

src/cli/commands/unlock.ts

@@ -9,13 +9,13 @@ export async function cmdUnlock(
   await runRemoteCommandForTargets(
     cwd,
     async ({ config, executor, serverName }) => {
-      const lockFile = `${config.remotePath}/.shipnode/deploy.lock`;
+      const lockPath = `${config.remotePath}/.shipnode/deploy.lock`;
 
       ui.info(`Checking for deployment lock on ${serverName} (${config.ssh.user}@${config.ssh.host})...`);
 
       const result = await executor.exec(
-        `if [ -f "${lockFile}" ]; then ` +
-          `age=$(( $(date +%s) - $(stat -c %Y "${lockFile}") )); ` +
+        `if [ -e "${lockPath}" ]; then ` +
+          `age=$(( $(date +%s) - $(stat -c %Y "${lockPath}") )); ` +
           `echo "FOUND:$age"; ` +
           `else echo "NOTFOUND"; fi`,
       );
@@ -34,7 +34,7 @@ export async function cmdUnlock(
         return;
       }
 
-      await executor.exec(`rm -f "${lockFile}"`);
+      await executor.exec(`rm -rf "${lockPath}"`);
       ui.success('Deployment lock cleared.');
     },
     { configPath: options.config, includeEmpty: true },

src/cli/monitor/poller.ts

@@ -33,10 +33,14 @@ function buildSystemSection(): string {
   ].join('; ');
 }
 
-function buildLockSection(lockFile: string): string {
+function buildLockSection(lockPath: string): string {
+  // Directory lock (atomic mkdir) stores the timestamp in acquired;
+  // legacy file locks store it as the file contents.
   return (
-    `if [ -f "${lockFile}" ]; then ` +
-    `echo "$(cat "${lockFile}" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockFile}" 2>/dev/null || date +%s) ))"; ` +
+    `if [ -d "${lockPath}" ]; then ` +
+    `echo "$(cat "${lockPath}/acquired" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockPath}" 2>/dev/null || date +%s) ))"; ` +
+    `elif [ -f "${lockPath}" ]; then ` +
+    `echo "$(cat "${lockPath}" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockPath}" 2>/dev/null || date +%s) ))"; ` +
     `else echo "none"; fi`
   );
 }

src/config/assembly.ts

@@ -34,6 +34,8 @@ type AssembleInput = {
   healthCheck?: unknown;
   envFile?: string;
   keepReleases?: number;
+  zeroDowntime?: boolean;
+  altPort?: number;
   sharedDirs?: string[];
   sharedFiles?: string[];
   buildDir?: string;

src/config/builder.ts

@@ -38,6 +38,8 @@ type BuilderState = {
   healthCheck?: Partial<HealthCheckConfig>;
   envFile?: string;
   buildDir?: string;
+  zeroDowntime?: boolean;
+  altPort?: number;
   sharedDirs?: string[];
   sharedFiles?: string[];
   appRoot?: string;
@@ -130,6 +132,16 @@ export class ShipnodeBuilder {
     return this;
   }
 
+  /**
+   * Enable zero-downtime (blue-green) releases for the web app. Optional
+   * `altPort` sets the green port (defaults to web port + 1).
+   */
+  zeroDowntime(altPort?: number): this {
+    this.config.zeroDowntime = true;
+    if (altPort !== undefined) this.config.altPort = altPort;
+    return this;
+  }
+
   sharedDirs(dirs: string[]): this {
     this.config.sharedDirs = dirs;
     return this;
@@ -343,6 +355,16 @@ export class ShipnodeAppBuilder {
     return this;
   }
 
+  /**
+   * Enable zero-downtime (blue-green) releases for this web app. Optional
+   * `altPort` sets the green port (defaults to web port + 1).
+   */
+  zeroDowntime(altPort?: number): this {
+    this.state.zeroDowntime = true;
+    if (altPort !== undefined) this.state.altPort = altPort;
+    return this;
+  }
+
   sharedDirs(dirs: string[]): this {
     this.state.sharedDirs = dirs;
     return this;