fix: harden v3 deployment workflows

Author: devalade

README.md

@@ -229,8 +229,8 @@ export default shipnode
 | `.pkgManager(pm, opts?)` | auto-detected | `npm` \| `yarn` \| `pnpm` \| `bun`; `opts.installCommand` overrides the install command |
 | `.installCommand(cmd)` | derived from pkg manager | Override the install command run on the server (e.g. `'npm ci --legacy-peer-deps'`). Equivalent to `pkgManager(pm, { installCommand: cmd })` |
 | `.buildDir(dir)` | auto-detected | Frontend build output dir |
-| `.zeroDowntime({ keepReleases? })` | true, 5 | Zero-downtime releases |
-| `.legacy()` | — | Simple in-place deploy |
+| `.zeroDowntime(altPort?)` | automatic for Caddy backends | Force blue-green releases and optionally choose the green port |
+| `.noZeroDowntime()` | — | Opt out and recreate PM2 processes during deploy |
 | `.healthCheck(path, opts?)` | `/health`, 30s, 3 retries | Post-deploy health check |
 | `.noHealthCheck()` | — | Skip health check |
 | `.envFile(f)` | `.env` | Local .env file to upload |
@@ -466,7 +466,11 @@ export default shipnode
 
 ## Zero-downtime releases
 
-By default, shipnode uses a Capistrano-style release structure:
+Backends with a domain and a PM2 web port use blue-green releases automatically. Shipnode starts the new web process on the idle port, checks it, and reloads Caddy only after it is healthy. Apps without a domain and worker-only apps keep the PM2 recreate path. Use `.noZeroDowntime()` to opt out explicitly; raw configuration may set `zeroDowntime: false`.
+
+Blue-green keeps the previous and current web processes resident, so budget about **2× the web process memory**. On an eligible Caddy backend, `.zeroDowntime(altPort?)` remains available to force the mode and choose the alternate port (default: web port + 1).
+
+Every app still uses a Capistrano-style release structure:
 
 ```
 /var/www/myapp/
@@ -479,7 +483,7 @@ By default, shipnode uses a Capistrano-style release structure:
   current -> releases/20240102T090000
 ```
 
-PM2 is reloaded against `current/` after the symlink switch. If the health check fails, you can roll back instantly with `shipnode rollback`.
+On the first migration, the new release starts on green while the existing uncoloured process keeps serving on blue. After health succeeds and Caddy reloads, Shipnode removes that legacy process. If health fails, Caddy and the legacy process remain untouched.
 
 ## Requirements
 

docs/adr/0005-blue-green-zero-downtime.md

@@ -1,8 +1,6 @@
 # Blue-green releases via a two-port colour swap behind Caddy
 
-The default deploy strategy stops the whole PM2 process set and starts the new release on the same port (`pm2 delete … && pm2 start …`). Between the delete and the moment the new process is listening — install relink, framework boot, first health probe — the port is closed and inbound requests are refused. For a plain web app that is a multi-second window of dropped requests on every deploy.
-
-`zeroDowntime` opts a backend web app into blue-green releases that close that window.
+Backend web apps behind Caddy (a domain plus a PM2 web port) use blue-green releases by default. Worker-only apps and backends without a domain retain the recreate strategy. `noZeroDowntime()` or raw `zeroDowntime: false` explicitly opts a Caddy backend out.
 
 ## Mechanism
 
@@ -13,7 +11,7 @@ Each deploy:
 1. Stages the release and switches the `current` symlink (as always). The previously-started colour keeps running the old code in memory — the symlink move doesn't touch a running process.
 2. Boots the **idle** colour on its own port (`pm2 start ecosystem.web.cjs`), reaping any stale same-colour instance first. The active colour is never touched. Workers are **not** reloaded yet.
 3. Health-checks the new colour on **its** port and colour-suffixed pm2 name (`api-green`).
-4. Only on success: reloads the single worker set against the new release, then rewrites the Caddy site to the new colour's port and `systemctl reload caddy` — a graceful reload that drains in-flight requests, so the flip drops nothing. Then persists the new active colour.
+4. Only on success: reloads the single worker set against the new release, then rewrites the Caddy site to the new colour's port and `systemctl reload caddy` — a graceful reload that drains in-flight requests. Then persists the new active colour. On the first migration only, the legacy uncoloured process is removed after this sequence.
 
 A failed health check throws before step 4: Caddy is untouched, workers stay on the previous release, the old colour is still serving, `current` is reverted to the previous release when one exists, and the failed colour is reaped by the next deploy. Zero user impact on a bad release — the main prize.
 
@@ -29,5 +27,5 @@ Because the previous colour is still running, rollback is an instant Caddy flip
 
 - **~2× memory for the web app** — both colours are resident between deploys. Documented; the price of instant rollback.
 - **The port pair is fixed at the first deploy** and persisted. Later changes to the web port or `altPort` in config are ignored until the state file is cleared, so a running colour is never silently re-homed.
-- **Migration costs one restart.** The first `zeroDowntime` deploy deletes the pre-existing uncoloured web process (which holds the blue == web port) by name so the target colour can bind. After that, every deploy is zero-drop.
-- **Opt-in, default off.** The recreate path is byte-for-byte unchanged, so existing deployments are unaffected until they set `zeroDowntime`.
+- **The first migration targets green.** A pre-existing uncoloured process can continue serving on the configured blue port through health and the Caddy reload; it is cleaned up only after the flip succeeds.
+- **Default on for Caddy backends.** Backends without a domain and worker-only apps keep recreate semantics. `.noZeroDowntime()` is the explicit builder opt-out.

src/cli/commands/ci.ts

@@ -1,5 +1,5 @@
-import { resolve } from 'path';
-import { readFile, writeFile } from 'node:fs/promises';
+import { relative, resolve, sep } from 'path';
+import { readFile, realpath, writeFile } from 'node:fs/promises';
 import { pathExists, ensureDir } from 'fs-extra';
 import { execa } from 'execa';
 import { loadConfig } from '../../config/loader.js';
@@ -56,21 +56,35 @@ async function hasBuildScript(cwd: string): Promise<boolean> {
   if (!(await pathExists(pkgPath))) return false;
   try {
     const raw = await readFile(pkgPath, 'utf8');
-    const pkg = JSON.parse(raw) as { scripts?: Record<string, string> };
-    return Boolean(pkg.scripts?.build);
+    const pkg: unknown = JSON.parse(raw);
+    if (typeof pkg !== 'object' || pkg === null || !('scripts' in pkg)) return false;
+    const scripts = pkg.scripts;
+    return typeof scripts === 'object' && scripts !== null && 'build' in scripts;
   } catch {
     return false;
   }
 }
 
-function generateWorkflow(pm: PkgManagerInfo, withBuild: boolean): string {
+function yamlSingleQuoted(value: string): string {
+  return `'${value.replace(/'/g, "''")}'`;
+}
+
+function generateWorkflow(
+  pm: PkgManagerInfo,
+  withBuild: boolean,
+  packageSpec: string,
+  deployDirectory?: string,
+): string {
   const setupPmStep = pm.setupAction
     ? `      - name: Setup ${pm.id}\n        uses: ${pm.setupAction}\n\n`
     : '';
 
   const buildStep = withBuild
     ? `\n      - name: Build\n        run: ${pm.id} run build\n`
     : '';
+  const deployWorkingDirectory = deployDirectory
+    ? `\n        working-directory: ${yamlSingleQuoted(deployDirectory)}`
+    : '';
 
   return `name: Deploy via Shipnode
 
@@ -109,25 +123,29 @@ ${buildStep}
           echo "\${{ secrets.SHIPNODE_KNOWN_HOSTS }}" >> ~/.ssh/known_hosts
 
       - name: Install Shipnode
-        run: npm install -g shipnode
+        run: npm install -g ${packageSpec}
 
       - name: Deploy
-        run: shipnode deploy
+        run: shipnode deploy${deployWorkingDirectory}
 `;
 }
 
 // ── cmdCiGithub ───────────────────────────────────────────────────────────────
 
 export async function cmdCiGithub(cwd: string): Promise<void> {
-  const pm = await detectPkgManager(cwd);
-  const withBuild = await hasBuildScript(cwd);
+  const canonicalCwd = await realpath(cwd);
+  const repositoryRoot = await resolveRepositoryRoot(canonicalCwd);
+  const deployDirectory = relative(repositoryRoot, canonicalCwd).split(sep).join('/') || undefined;
+  const pm = await detectPkgManager(repositoryRoot);
+  const withBuild = await hasBuildScript(repositoryRoot);
+  const packageSpec = await currentPackageSpec();
 
   ui.info(`Detected package manager: ${pm.id}`);
   if (withBuild) {
     ui.info('Found scripts.build in package.json — build step will be included.');
   }
 
-  const workflowDir = resolve(cwd, '.github', 'workflows');
+  const workflowDir = resolve(repositoryRoot, '.github', 'workflows');
   const workflowPath = resolve(workflowDir, 'shipnode-deploy.yml');
 
   if (await pathExists(workflowPath)) {
@@ -139,7 +157,7 @@ export async function cmdCiGithub(cwd: string): Promise<void> {
   }
 
   await ensureDir(workflowDir);
-  const content = generateWorkflow(pm, withBuild);
+  const content = generateWorkflow(pm, withBuild, packageSpec, deployDirectory);
   await writeFile(workflowPath, content, 'utf8');
 
   ui.success(`Workflow written to .github/workflows/shipnode-deploy.yml`);
@@ -155,6 +173,32 @@ export async function cmdCiGithub(cwd: string): Promise<void> {
   console.log('  Only the private key must be kept as a secret.');
 }
 
+async function resolveRepositoryRoot(cwd: string): Promise<string> {
+  try {
+    const result = await execa('git', ['rev-parse', '--show-toplevel'], { cwd });
+    const root = result.stdout.trim();
+    return root || cwd;
+  } catch {
+    return cwd;
+  }
+}
+
+async function currentPackageSpec(): Promise<string> {
+  const raw = await readFile(new URL('../../../package.json', import.meta.url), 'utf8');
+  const metadata: unknown = JSON.parse(raw);
+  if (
+    typeof metadata !== 'object' ||
+    metadata === null ||
+    !('name' in metadata) ||
+    !('version' in metadata) ||
+    typeof metadata.name !== 'string' ||
+    typeof metadata.version !== 'string'
+  ) {
+    throw new Error('Shipnode package metadata is missing name or version');
+  }
+  return `${metadata.name}@${metadata.version}`;
+}
+
 // ── cmdCiEnvSync ──────────────────────────────────────────────────────────────
 
 interface EnvVar {

src/cli/commands/doctor.ts

@@ -51,7 +51,8 @@ function checkLocal(config: ShipnodeConfig): void {
   }
 }
 
-async function checkRemote(
+/** Run remote dependency diagnostics through the same executor used by the CLI. */
+export async function checkRemote(
   config: ShipnodeConfig,
   executor: { exec: (cmd: string, opts?: { timeout?: number }) => Promise<{ stdout: string; exitCode: number }> },
 ): Promise<void> {
@@ -62,9 +63,11 @@ async function checkRemote(
   const needsCaddy = config.apps.some((app) => app.domain);
   const needsDocker = Object.keys(config.accessories ?? {}).length > 0;
 
+  const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion;
+  const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`;
   const checks = [
-    ...(needsNode ? [{ name: 'Node', cmd: 'node --version' }] : []),
-    ...(needsPm2 ? [{ name: 'PM2', cmd: 'pm2 --version' }] : []),
+    ...(needsNode ? [{ name: 'Node', cmd: `${mise}; mise exec "node@${nodeVersion}" -- node --version` }] : []),
+    ...(needsPm2 ? [{ name: 'PM2', cmd: `${mise}; mise exec "node@${nodeVersion}" -- pm2 --version` }] : []),
     ...(needsCaddy ? [{ name: 'Caddy', cmd: 'caddy version' }] : []),
     ...(needsDocker ? [{ name: 'Docker', cmd: 'docker --version' }] : []),
     { name: 'rsync', cmd: 'rsync --version' },
@@ -95,29 +98,35 @@ async function checkRemote(
   }
 }
 
-async function checkSecurity(
+/** Run the security audit and write each complete diagnostic section to `write`. */
+export async function checkSecurity(
   config: { remotePath: string },
   executor: { exec: (cmd: string) => Promise<{ stdout: string }> },
+  write: (output: string) => void = console.log,
 ): Promise<void> {
   ui.info('Running security audit...');
 
   const sshdResult = await executor.exec(
     'sudo grep -E "^(PermitRootLogin|PasswordAuthentication|Port)" /etc/ssh/sshd_config 2>/dev/null || echo "not found"',
   );
   ui.info('SSH Configuration:');
-  console.log(`  ${sshdResult.stdout.replace(/\n/g, '\n  ')}`);
+  write(indentDiagnosticOutput(sshdResult.stdout));
 
   const firewallResult = await executor.exec(
     'sudo ufw status 2>/dev/null || sudo iptables -L -n 2>/dev/null || echo "no firewall detected"',
   );
   ui.info('Firewall Status:');
-  console.log(`  ${firewallResult.stdout.split('\n').slice(0, 5).join('\n  ')}`);
+  write(indentDiagnosticOutput(firewallResult.stdout));
 
   const fail2banResult = await executor.exec('sudo fail2ban-client status 2>/dev/null || echo "fail2ban not installed"');
   ui.info('Fail2ban:');
-  console.log(`  ${fail2banResult.stdout.split('\n')[0]}`);
+  write(`  ${fail2banResult.stdout.split('\n')[0]}`);
 
   const permResult = await executor.exec(`stat -c "%a %U:%G" "${config.remotePath}/shared/.env" 2>/dev/null || echo "no .env found"`);
   ui.info('Env file permissions:');
-  console.log(`  ${permResult.stdout}`);
+  write(`  ${permResult.stdout}`);
+}
+
+function indentDiagnosticOutput(output: string): string {
+  return `  ${output.replace(/\n/g, '\n  ')}`;
 }

src/cli/commands/harden.ts

@@ -97,12 +97,12 @@ export async function cmdHarden(cwd: string, options: { config?: string }): Prom
 
       ui.heading('Fail2ban');
       const f2b = (await executor.exec(sec.fail2banCheckActiveCommand())).stdout;
-      if (!f2b.includes('ACTIVE')) {
+      if (!sec.isFail2banActive(f2b)) {
         ui.warn('Fail2ban not active.');
         if (await confirm('Install and configure fail2ban?')) {
-          await executor.exec(sec.fail2banInstallCommand());
-          await executor.exec(sec.fail2banApplyConfigCommand());
-          await executor.exec(sec.fail2banEnableCommand());
+          await executor.execOrThrow(sec.fail2banInstallCommand());
+          await executor.execOrThrow(sec.fail2banApplyConfigCommand());
+          await executor.execOrThrow(sec.fail2banEnableCommand());
           ui.success('Fail2ban installed');
           changes.push('Fail2ban: installed, sshd jail enabled');
         }

src/cli/commands/setup.ts

@@ -1,4 +1,5 @@
 import { existsSync, readFileSync } from 'fs';
+import { Result, type Result as ResultType } from 'better-result';
 import { Listr } from 'listr2';
 import { runRemoteCommandForTargets } from '../runner.js';
 import { ui } from '../ui.js';
@@ -13,6 +14,7 @@ import {
   buildRedisProbeCommand,
 } from '../../infrastructure/provisioning/commands.js';
 import { loadUsersYml, saveUsersYml, syncUsers, upsertUser } from './user.js';
+import { Pm2StartupError } from '../../shared/result-errors.js';
 
 const DEPLOY_USER = 'deploy';
 
@@ -92,13 +94,17 @@ function asUser(user: string | null, cmd: string): string {
   return `sudo -u "${user}" bash -lc '${escaped}'`;
 }
 
+function shellQuote(value: string): string {
+  return `'${value.replace(/'/g, `'"'"'`)}'`;
+}
+
 export function buildTasks(executor: RemoteExecutor, config: ShipnodeConfig, ownerUser: string | null) {
   const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion;
   const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`;
   // When a deploy user was bootstrapped, install mise/node/pm2 into their home
   // so PM2 processes run as that user and `pm2-<user>.service` matches. Without
   // this, everything lands in root's home and deploy has no pm2 on their PATH.
-  const targetHome = ownerUser ? `/home/${ownerUser}` : '$HOME';
+  const effectiveUser = ownerUser ?? config.ssh.user;
   const hasApps = config.apps.length > 0;
   const hasAccessories = Object.keys(config.accessories ?? {}).length > 0;
 
@@ -169,17 +175,16 @@ export function buildTasks(executor: RemoteExecutor, config: ShipnodeConfig, own
             // shims tools it manages itself), so we resolve pm2's real path via
             // `mise exec node@X -- which pm2` and pass that to `env PATH=... pm2`.
             title: `Configure systemd startup (${ownerUser ?? '$USER'})`,
-            task: () => executor.execOrThrow(
-              (ownerUser
-                ? `PM2_BIN=$(sudo -u "${ownerUser}" bash -lc '${mise}; mise exec "node@${nodeVersion}" -- which pm2'); ` +
-                  `[ -n "$PM2_BIN" ] && env PATH="$(dirname "$PM2_BIN"):$PATH" "$PM2_BIN" startup systemd -u "${ownerUser}" --hp "${targetHome}" || true`
-                : `${mise}; mise exec "node@${nodeVersion}" -- pm2 startup systemd -u $USER --hp $HOME || true`) +
-              ` && ` +
-              asUser(
+            task: async () => {
+              const result = await configurePm2Startup(
+                executor,
+                effectiveUser,
                 ownerUser,
-                `${mise}; mise exec "node@${nodeVersion}" -- pm2 save --force || true`,
-              ),
-            ),
+                nodeVersion,
+                mise,
+              );
+              if (result.isErr()) throw result.error;
+            },
           },
         ], { concurrent: false }),
       }] : []),
@@ -259,11 +264,10 @@ export function buildTasks(executor: RemoteExecutor, config: ShipnodeConfig, own
             title: `Create release structure at ${config.remotePath}`,
             task: () => executor.execOrThrow(
               'SUDO=""; [ "$EUID" -ne 0 ] && SUDO="sudo"; ' +
-              `$SUDO mkdir -p "${config.remotePath}" && ` +
-              (ownerUser
-                ? `$SUDO chown "${ownerUser}:${ownerUser}" "${config.remotePath}" && ` +
-                  `sudo -u "${ownerUser}" mkdir -p "${config.remotePath}/releases" "${config.remotePath}/shared" "${config.remotePath}/.shipnode"`
-                : `mkdir -p "${config.remotePath}/releases" "${config.remotePath}/shared" "${config.remotePath}/.shipnode"`),
+              `OWNER=${shellQuote(effectiveUser)}; GROUP=$(id -gn "$OWNER"); ` +
+              `$SUDO install -d -m 0755 -o "$OWNER" -g "$GROUP" ` +
+              `${shellQuote(config.remotePath)} ${shellQuote(`${config.remotePath}/releases`)} ` +
+              `${shellQuote(`${config.remotePath}/shared`)} ${shellQuote(`${config.remotePath}/.shipnode`)}`,
             ),
           },
           ...(config.apps.some((app) => app.domain) ? [{
@@ -282,3 +286,36 @@ export function buildTasks(executor: RemoteExecutor, config: ShipnodeConfig, own
     { rendererOptions: { collapseErrors: false } },
   );
 }
+
+async function configurePm2Startup(
+  executor: RemoteExecutor,
+  effectiveUser: string,
+  runAsUser: string | null,
+  nodeVersion: string,
+  mise: string,
+): Promise<ResultType<void, Pm2StartupError>> {
+  const resolvePm2 = runAsUser
+    ? `sudo -u "${runAsUser}" bash -lc '${mise}; mise exec "node@${nodeVersion}" -- which pm2'`
+    : `${mise}; mise exec "node@${nodeVersion}" -- which pm2`;
+  const savePm2 = asUser(
+    runAsUser,
+    `${mise}; mise exec "node@${nodeVersion}" -- pm2 save --force`,
+  );
+  const unit = `pm2-${effectiveUser}.service`;
+  const command =
+    'SUDO=""; [ "$EUID" -ne 0 ] && SUDO="sudo"; ' +
+    `OWNER=${shellQuote(effectiveUser)}; HOME_DIR=$(getent passwd "$OWNER" | cut -d: -f6); ` +
+    `PM2_BIN=$(${resolvePm2}); ` +
+    `[ -n "$HOME_DIR" ] && [ -x "$PM2_BIN" ] && ` +
+    `$SUDO env PATH="$(dirname "$PM2_BIN"):$PATH" "$PM2_BIN" startup systemd -u "$OWNER" --hp "$HOME_DIR" && ` +
+    `$SUDO test -f ${shellQuote(`/etc/systemd/system/${unit}`)} && ` +
+    `$SUDO systemctl is-enabled --quiet ${shellQuote(unit)} && ` +
+    savePm2;
+
+  const result = await executor.exec(command);
+  if (result.exitCode !== 0) {
+    const detail = (result.stderr || result.stdout).trim() || `command exited with ${result.exitCode}`;
+    return Result.err(new Pm2StartupError({ user: effectiveUser, detail }));
+  }
+  return Result.ok(undefined);
+}