fix(cli): deploy fresh-host bootstrap — proxy start_or_run per host, kamal network, real setup() (#3162)

Wave 2a of the #2957 deploy-orchestration campaign:

- DEP-5a: the proxy boot guard was details() || boot(), but details()
  (docker ps --filter) exits 0 whether or not the proxy exists, so
  boot() was unreachable and kamal-proxy never started on a fresh host.
  Replaced with Kamal's Proxy##start_or_run shape:
  docker start kamal-proxy || docker run ... (ProxyCommands.start_or_run).
- DEP-5b: the guard was dispatched to ONE host via $dispatchAny; the
  proxy now boots on every proxy-fronted host via $dispatch/onEach.
- DEP-5c: DockerCommands.create_network had zero call sites while every
  app/proxy/accessory docker run joins --network kamal. New
  ensure_network() (inspect probe || create, idempotent) is dispatched
  to all app hosts during deploy and to accessory hosts during setup.
- setup() was literally 'return deploy(opts)'. It now runs a real setup
  phase: kamal network create + accessory boot on each accessory's
  hosts, then the full deploy (Kamal setup semantics). Deploy body
  extracted to $deploy() so setup's bootstrap commands survive into
  dryRunOutput().
- Proxy gating: kamal-proxy deploy fired for EVERY role; it is now
  gated to proxy-fronted roles via Role.runningProxy() (explicit
  role-level proxy: boolean/hash wins, default true only for web),
  matching Kamal's Role##running_proxy?.

Verified with FakeSshPool specs (fresh-host sequence ordering,
multi-host proxy iteration, setup!=deploy, proxy gating) and dry-run
flows; real SSH/docker-remote is unverifiable in-harness. Full CLI
suite in the lucee7 docker harness: 1005 pass / 0 fail / 2 tolerated
docker-env artifacts (SshClientSpec/SshPoolSpec docker-not-found).

Refs #2957

Signed-off-by: Peter Amiri <peter@alurium.com>

Author: bpamiri

changelog.d/deploy-fresh-host-bootstrap.fixed.md

@@ -0,0 +1 @@
+- `wheels deploy` now works on fresh hosts: the kamal-proxy boot guard `details() || boot()` never reached `boot()` (`docker ps` exits 0 whether or not the proxy exists) and was dispatched to only one host — the proxy is now booted via Kamal's `docker start kamal-proxy || docker run ...` (`ProxyCommands.start_or_run()`) on every proxy-fronted host; the `kamal` docker network, previously never created (zero `create_network` call sites while every app/proxy/accessory `docker run` joins `--network kamal`), is now idempotently ensured on every host before the first consumer; `wheels deploy setup` is a real setup phase (network create + accessory boot on accessory hosts, then deploy) instead of a literal `deploy()` alias; and `kamal-proxy deploy` registration is gated to proxy-fronted roles (role-level `proxy:` boolean, defaulting to the `web` role) instead of firing for every job/worker role (#2957)

cli/lucli/Module.cfc

@@ -2244,7 +2244,7 @@ component extends="modules.BaseModule" {
 	 *   wheels deploy rollback v1              - roll back to version v1
 	 *   wheels deploy config                   - print resolved config as YAML
 	 *   wheels deploy init                     - create config stub
-	 *   wheels deploy setup                    - full setup (Phase 2 adds accessories)
+	 *   wheels deploy setup                    - one-time bootstrap (network + accessories) + deploy
 	 *   wheels deploy bootstrap                - install Docker on every host
 	 *   wheels deploy exec "uname -a"          - run a command on every host
 	 *   wheels deploy version                  - show version pinning

cli/lucli/services/deploy/cli/DeployMainCli.cfc

@@ -56,6 +56,15 @@ component {
 
     public string function deploy(required struct opts) {
         arrayClear(variables.dryRunBuffer);
+        return $deploy(arguments.opts);
+    }
+
+    /**
+     * Deploy body, shared by deploy() and setup(). Does NOT clear the
+     * dry-run buffer — the public verbs do, so setup()'s pre-deploy
+     * bootstrap commands survive into dryRunOutput().
+     */
+    private string function $deploy(required struct opts) {
         var cfg = variables.loader.load(
             arguments.opts.configPath,
             {destination: arguments.opts.destination ?: ""}
@@ -66,6 +75,7 @@ component {
         var app = new modules.wheels.services.deploy.commands.AppCommands(cfg);
         var proxy = new modules.wheels.services.deploy.commands.ProxyCommands(cfg);
         var builder = new modules.wheels.services.deploy.commands.BuilderCommands(cfg);
+        var dockerCmds = new modules.wheels.services.deploy.commands.DockerCommands(cfg);
         var lock = new modules.wheels.services.deploy.commands.LockCommands(cfg);
         var hooks = new modules.wheels.services.deploy.commands.HookCommands(
             cfg,
@@ -93,16 +103,31 @@ component {
 
             try {
                 $dispatch(hosts, builder.pull(ver), dryRun);
-                $dispatchAny(hosts, proxy.details() & " || " & proxy.boot(), dryRun);
+                // Fresh-host bootstrap (#2957 DEP-5c): every `docker run`
+                // below joins --network kamal, so the network must exist
+                // before the first consumer. ensure_network is idempotent.
+                $dispatch(hosts, dockerCmds.ensure_network("kamal"), dryRun);
+                // Boot (or start) kamal-proxy on EVERY proxy-fronted host
+                // (#2957 DEP-5a/5b). The old `details() || boot()` guard
+                // never booted anything — `docker ps` exits 0 regardless —
+                // and was dispatched to only one host via $dispatchAny.
+                var proxyHosts = $proxyHosts(cfg);
+                if (arrayLen(proxyHosts)) {
+                    $dispatch(proxyHosts, proxy.start_or_run(), dryRun);
+                }
 
                 for (var role in cfg.roles()) {
                     for (var host in role.hosts()) {
                         $dispatch([host], app.run(role, ver), dryRun);
-                        $dispatch(
-                            [host],
-                            proxy.deploy(role, app.container_name(role, ver) & ":" & appPort),
-                            dryRun
-                        );
+                        // Only proxy-fronted roles register with kamal-proxy —
+                        // job/worker roles serve no traffic (#2957).
+                        if (role.runningProxy()) {
+                            $dispatch(
+                                [host],
+                                proxy.deploy(role, app.container_name(role, ver) & ":" & appPort),
+                                dryRun
+                            );
+                        }
                     }
                 }
             } finally {
@@ -179,9 +204,34 @@ component {
         );
     }
 
+    /**
+     * One-time server bootstrap + first deploy (Kamal `setup` semantics):
+     * create the kamal docker network on every accessory host, boot each
+     * accessory on its hosts, then run a full deploy (which bootstraps the
+     * network + proxy on the app hosts). Previously a literal alias for
+     * deploy(), so fresh hosts never got their accessories (#2957).
+     */
     public string function setup(required struct opts) {
-        // Phase 2 will add accessory boot; for Phase 1 this equals deploy.
-        return deploy(arguments.opts);
+        arrayClear(variables.dryRunBuffer);
+        var cfg = variables.loader.load(
+            arguments.opts.configPath,
+            {destination: arguments.opts.destination ?: ""}
+        );
+        var dryRun = arguments.opts.dryRun ?: false;
+
+        if (arrayLen(cfg.accessories())) {
+            var dockerCmds = new modules.wheels.services.deploy.commands.DockerCommands(cfg);
+            var accCmds = new modules.wheels.services.deploy.commands.AccessoryCommands(cfg);
+            for (var acc in cfg.accessories()) {
+                // Accessories join --network kamal too, and may live on
+                // hosts outside the app roles — ensure the network there
+                // before the accessory container runs (#2957 DEP-5c).
+                $dispatch(acc.hosts(), dockerCmds.ensure_network("kamal"), dryRun);
+                $dispatch(acc.hosts(), accCmds.run(acc), dryRun);
+            }
+        }
+
+        return $deploy(arguments.opts);
     }
 
     /**
@@ -451,8 +501,8 @@ component {
 
     /**
      * Dispatch a single command to "any one" host — used for operations
-     * that only need to happen once across the fleet (lock acquire/release,
-     * proxy boot check). FakeSshPool.onAny records exactly one call.
+     * that only need to happen once across the fleet (lock acquire/release).
+     * FakeSshPool.onAny records exactly one call.
      */
     private void function $dispatchAny(
         required array hosts,
@@ -544,6 +594,22 @@ component {
         return out;
     }
 
+    /**
+     * Distinct hosts of every proxy-fronted role (Role.runningProxy()),
+     * in declaration order. Each of these needs its own kamal-proxy
+     * container (#2957 DEP-5b).
+     */
+    private array function $proxyHosts(required any cfg) {
+        var out = [];
+        for (var role in arguments.cfg.roles()) {
+            if (!role.runningProxy()) continue;
+            for (var h in role.hosts()) {
+                if (!arrayContains(out, h)) arrayAppend(out, h);
+            }
+        }
+        return out;
+    }
+
     private struct function $roleHosts(required any cfg) {
         var out = {};
         for (var role in arguments.cfg.roles()) {

cli/lucli/services/deploy/commands/DockerCommands.cfc

@@ -18,4 +18,16 @@ component extends="Base" {
     public string function create_network(required string name) {
         return docker("network", "create", arguments.name);
     }
+
+    /**
+     * Idempotent network create — `docker network create` exits nonzero
+     * when the network already exists, so deploy/setup guard it with an
+     * inspect probe (exit 0 only when the network is present). Ruby Kamal
+     * rescues the "already exists" error instead; a shell guard is the
+     * commands-are-strings equivalent (#2957 DEP-5c).
+     */
+    public string function ensure_network(required string name) {
+        return "docker network inspect #arguments.name# >/dev/null 2>&1 || "
+             & create_network(arguments.name);
+    }
 }

cli/lucli/services/deploy/commands/ProxyCommands.cfc

@@ -71,6 +71,19 @@ component extends="Base" {
         return docker("start", variables.PROXY_CONTAINER_NAME);
     }
 
+    /**
+     * Fresh-host-safe boot, mirroring Kamal's Proxy#start_or_run
+     * (`combine start, run, by: "||"`): `docker start` succeeds when the
+     * container already exists (running start is a no-op, stopped start
+     * resumes it), and the full `docker run` fires only on a truly fresh
+     * host. The previous guard — `details() || boot()` — never reached
+     * boot() because `docker ps --filter` exits 0 whether or not anything
+     * matches (#2957 DEP-5a).
+     */
+    public string function start_or_run() {
+        return start() & " || " & boot();
+    }
+
     public string function stop() {
         return docker("stop", variables.PROXY_CONTAINER_NAME);
     }

cli/lucli/services/deploy/config/Config.cfc

@@ -93,6 +93,7 @@ component {
 					var roleRaw = {name: roleName, hosts: hosts};
 					if (structKeyExists(entry, "env")) roleRaw.env = entry.env;
 					if (structKeyExists(entry, "cmd")) roleRaw.cmd = entry.cmd;
+					if (structKeyExists(entry, "proxy")) roleRaw.proxy = entry.proxy;
 					arrayAppend(out, new Role(roleRaw));
 				}
 			}

cli/lucli/services/deploy/config/Role.cfc

@@ -33,4 +33,21 @@ component {
 		return variables.raw.cmd ?: "";
 	}
 
+	/**
+	 * Whether this role's containers are fronted by kamal-proxy.
+	 *
+	 * Mirrors Kamal's Role#running_proxy? (lib/kamal/configuration/role.rb):
+	 * an explicit role-level `proxy:` boolean wins; a `proxy:` hash (proxy
+	 * options) opts the role in; otherwise only the default "web" role runs
+	 * behind the proxy. Job/worker roles must not receive proxy boot or
+	 * `kamal-proxy deploy` commands (#2957).
+	 */
+	public boolean function runningProxy() {
+		if (structKeyExists(variables.raw, "proxy")) {
+			if (isBoolean(variables.raw.proxy)) return variables.raw.proxy;
+			if (isStruct(variables.raw.proxy)) return true;
+		}
+		return lCase(name()) == "web";
+	}
+
 }

cli/lucli/tests/specs/deploy/cli/DeployMainCliSpec.cfc

@@ -3,6 +3,8 @@ component extends="wheels.wheelstest.system.BaseSpec" {
     function beforeAll() {
         variables.fixture = expandPath("/cli/lucli/tests/_fixtures/deploy/configs/minimal.yml");
         variables.proxyFixture = expandPath("/cli/lucli/tests/_fixtures/deploy/configs/with-proxy.yml");
+        variables.multiRoleFixture = expandPath("/cli/lucli/tests/_fixtures/deploy/configs/full.yml");
+        variables.accessoriesFixture = expandPath("/cli/lucli/tests/_fixtures/deploy/configs/with-accessories.yml");
     }
 
     function run() {
@@ -709,6 +711,104 @@ component extends="wheels.wheelstest.system.BaseSpec" {
                 expect(out).notToInclude(":3000");
             });
 
+            // Regression suite for #2957 (Wave 2a) — fresh-host bootstrap.
+            // (DEP-5a) the proxy boot guard was `details() || boot()`; details()
+            // is `docker ps --filter ...` which exits 0 whether or not the proxy
+            // exists, so boot() was unreachable and kamal-proxy never started on
+            // a fresh host. (DEP-5b) the guard was dispatched to ONE host via
+            // $dispatchAny while every proxy-fronted host needs its own proxy.
+            // (DEP-5c) `docker network create kamal` had zero call sites while
+            // app/proxy/accessory runs all require `--network kamal`. setup()
+            // was literally `return deploy(opts)` — no accessory boot. And
+            // proxy.deploy fired for EVERY role, including non-fronted job roles.
+
+            it("deploy boots kamal-proxy via docker start || docker run, not the dead docker ps guard (##2957 DEP-5a)", () => {
+                var fake = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc = new cli.lucli.services.deploy.cli.DeployMainCli(fake);
+                dc.deploy({configPath: variables.fixture, version: "v1"});
+                var cmds = $cmds(fake);
+                expect($anyInclude(cmds, "docker start kamal-proxy || docker run")).toBeTrue();
+                expect($anyInclude(cmds, "docker ps --filter name=kamal-proxy || ")).toBeFalse();
+            });
+
+            it("deploy creates the kamal network before any --network kamal consumer (##2957 DEP-5c)", () => {
+                var fake = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc = new cli.lucli.services.deploy.cli.DeployMainCli(fake);
+                dc.deploy({configPath: variables.fixture, version: "v1"});
+                var cmds = $cmds(fake);
+                var networkIdx = 0; var consumerIdx = 0;
+                for (var i = 1; i <= arrayLen(cmds); i++) {
+                    if (!networkIdx && findNoCase("docker network create kamal", cmds[i])) networkIdx = i;
+                    if (!consumerIdx && findNoCase("--network kamal", cmds[i])) consumerIdx = i;
+                }
+                expect(networkIdx).toBeGT(0);
+                expect(consumerIdx).toBeGT(networkIdx);
+            });
+
+            it("deploy boots the proxy on EVERY proxy-fronted host, not just one (##2957 DEP-5b)", () => {
+                var fake = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc = new cli.lucli.services.deploy.cli.DeployMainCli(fake);
+                dc.deploy({configPath: variables.multiRoleFixture, version: "v1"});
+                var bootHosts = $hostsFor(fake, "docker start kamal-proxy || docker run");
+                // full.yml: web role = 1.1.1.1 + 1.1.1.2; workers = 1.1.1.3 + 1.1.1.4.
+                expect(bootHosts).toInclude("1.1.1.1");
+                expect(bootHosts).toInclude("1.1.1.2");
+                expect(bootHosts).notToInclude("1.1.1.3");
+                expect(bootHosts).notToInclude("1.1.1.4");
+            });
+
+            it("deploy gates kamal-proxy deploy to proxy-fronted roles only (##2957)", () => {
+                var fake = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc = new cli.lucli.services.deploy.cli.DeployMainCli(fake);
+                dc.deploy({configPath: variables.multiRoleFixture, version: "v1"});
+                var proxyDeployHosts = $hostsFor(fake, "kamal-proxy deploy");
+                expect(proxyDeployHosts).toInclude("1.1.1.1");
+                expect(proxyDeployHosts).toInclude("1.1.1.2");
+                expect(proxyDeployHosts).notToInclude("1.1.1.3");
+                expect(proxyDeployHosts).notToInclude("1.1.1.4");
+                // ...while the app containers still run on every role's hosts.
+                var runHosts = $hostsFor(fake, "docker run --detach --restart unless-stopped --name app-");
+                expect(runHosts).toInclude("1.1.1.3");
+                expect(runHosts).toInclude("1.1.1.4");
+            });
+
+            it("setup boots accessories before the app deploy; plain deploy does not (##2957 setup!=deploy)", () => {
+                // setup: accessory containers run, before the app container.
+                var fake = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc = new cli.lucli.services.deploy.cli.DeployMainCli(fake);
+                dc.setup({configPath: variables.accessoriesFixture, version: "v1"});
+                var cmds = $cmds(fake);
+                var accIdx = 0; var appIdx = 0;
+                for (var i = 1; i <= arrayLen(cmds); i++) {
+                    if (!accIdx && findNoCase("--name demo-db", cmds[i])) accIdx = i;
+                    if (!appIdx && findNoCase("--name demo-web-v1", cmds[i])) appIdx = i;
+                }
+                expect(accIdx).toBeGT(0);
+                expect($anyInclude(cmds, "--name demo-redis")).toBeTrue();
+                expect(appIdx).toBeGT(accIdx);
+                // The accessory host (1.2.3.5) gets the network created too.
+                expect($hostsFor(fake, "docker network create kamal")).toInclude("1.2.3.5");
+
+                // plain deploy: no accessory boot.
+                var fake2 = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc2 = new cli.lucli.services.deploy.cli.DeployMainCli(fake2);
+                dc2.deploy({configPath: variables.accessoriesFixture, version: "v1"});
+                var cmds2 = $cmds(fake2);
+                expect($anyInclude(cmds2, "--name demo-db")).toBeFalse();
+                expect($anyInclude(cmds2, "--name demo-redis")).toBeFalse();
+            });
+
+            it("setup --dry-run buffers network create + accessory boot + proxy boot + app run (##2957)", () => {
+                var fake = new cli.lucli.services.deploy.lib.FakeSshPool();
+                var dc = new cli.lucli.services.deploy.cli.DeployMainCli(fake);
+                var out = dc.setup({configPath: variables.accessoriesFixture, version: "v1", dryRun: true});
+                expect(arrayLen(fake.calls())).toBe(0);
+                expect(out).toInclude("docker network create kamal");
+                expect(out).toInclude("--name demo-db");
+                expect(out).toInclude("docker start kamal-proxy || docker run");
+                expect(out).toInclude("--name demo-web-v1");
+            });
+
             // Regression for #2671 — git's stderr ("fatal: not a git repository...") used to leak through as the version string.
             it("$gitShortSha() returns 'unknown' when run outside a git repo", () => {
                 var nonGitDir = getTempDirectory() & "/wheels-2671-main-" & createUUID();
@@ -733,4 +833,21 @@ component extends="wheels.wheelstest.system.BaseSpec" {
         for (var s in arguments.arr) if (findNoCase(arguments.needle, s)) return true;
         return false;
     }
+
+    private array function $cmds(required any fake) {
+        var out = [];
+        for (var c in arguments.fake.calls()) arrayAppend(out, c.cmd ?: "");
+        return out;
+    }
+
+    /** Distinct hosts that received a command containing needle, in call order. */
+    private array function $hostsFor(required any fake, required string needle) {
+        var out = [];
+        for (var c in arguments.fake.calls()) {
+            if (findNoCase(arguments.needle, c.cmd ?: "") && !arrayContains(out, c.host ?: "")) {
+                arrayAppend(out, c.host ?: "");
+            }
+        }
+        return out;
+    }
 }

cli/lucli/tests/specs/deploy/commands/DockerCommandsSpec.cfc

@@ -0,0 +1,28 @@
+component extends="wheels.wheelstest.system.BaseSpec" {
+
+    function beforeAll() {
+        variables.cfg = new cli.lucli.services.deploy.config.ConfigLoader()
+            .load(expandPath("/cli/lucli/tests/_fixtures/deploy/configs/minimal.yml"));
+    }
+
+    function run() {
+        describe("DockerCommands", () => {
+
+            it("create_network() emits docker network create", () => {
+                var cmd = new cli.lucli.services.deploy.commands.DockerCommands(variables.cfg)
+                    .create_network("kamal");
+                expect(cmd).toBe("docker network create kamal");
+            });
+
+            // #2957 DEP-5c — `docker network create` exits nonzero when the
+            // network already exists, so the deploy/setup flows need an
+            // idempotent guard (inspect probe || create) to be re-runnable.
+            it("ensure_network() guards create with an inspect probe so reruns are idempotent (##2957)", () => {
+                var cmd = new cli.lucli.services.deploy.commands.DockerCommands(variables.cfg)
+                    .ensure_network("kamal");
+                expect(cmd).toInclude("docker network inspect kamal");
+                expect(cmd).toInclude(" || docker network create kamal");
+            });
+        });
+    }
+}