Skip to content

Commit 7d0ca8f

Browse files
committed
Add deploy --watch, a development loop over the live release.
One full deploy establishes a baseline, then every save is rsynced into the release that is already running, rebuilt, reloaded and health-probed. Cycles are incremental: rsync receives an explicit --files-from list, deps reinstall only when a manifest moves, and the health probe uses exponential backoff instead of the deploy path's fixed delays. Measured ~7.8s edit-to-live on a TanStack Start app against a real VPS. This is deliberately not a release. It patches what is serving, so there is no rollback target, deletes do not propagate, and reload can drop in-flight requests. The session says so on startup. --build <remote|local|none> selects where each cycle builds, because backends are not all alike: some build on the server, others build locally and upload the artifact (Nitro/TanStack Start/Nuxt deployed with --skip-build). Build output is watched only under `none` — when shipnode itself runs the build, reacting to its writes would rebuild forever. Testing this against a real 37-app workspace found four defects that the unit tests could not, each fixed with a regression test: - Build output was ignored unconditionally, so local-build projects synced source the app never runs. - Watching our own build output fed each cycle back into itself. Ignoring build *directories* was not enough — a build also regenerates files inside the source tree (routeTree.gen.ts) and drops temp files at the repo root. Fixed by suppressing the watcher around builds we run, which holds for any framework's codegen. Observed in the wild as a pm2 reload every ~8s. - The debounce reset on every event with no ceiling, so a repo with a background writer never emitted a batch and the loop looked hung. - Ctrl-C during a cycle exited before the lock was released, leaving a lock no process owned and blocking later deploys. Also: SSH keepalives so long-lived sessions survive an idle NAT timeout, the watcher honours .shipnodeignore, local builds run from appRoot in a monorepo, the .env build-output symlink logic is shared so it survives a rebuild, and unlock gains --yes (it blocked on stdin with no TTY, which is exactly where a stuck lock needs clearing).
1 parent 7e2eca4 commit 7d0ca8f

17 files changed

Lines changed: 2204 additions & 38 deletions

File tree

.claude/skills/shipnode/SKILL.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ shipnode run "pnpm db:apply" [--app name]
173173
shipnode run bash
174174
shipnode deploy --dry-run
175175
shipnode deploy --skip-build
176+
shipnode deploy --watch [--app name] # dev loop: patches the live release, no rollback target
176177
shipnode unlock # stuck .shipnode/deploy.lock (directory)
177178
shipnode config show [--app name]
178179
shipnode user add alice --key ~/.ssh/alice.pub --sudo

CHANGELOG.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,22 @@ All notable changes to `@devalade/shipnode` will be documented here.
44

55
## [Unreleased]
66

7+
### Added
8+
- **`shipnode deploy --watch` — the development loop.** Runs one normal deploy to establish a baseline, then watches the working tree: every save is rsynced into the live release, rebuilt, reloaded, and health-probed. Each cycle is incremental — rsync receives an explicit `--files-from` list of changed paths instead of scanning the whole tree, dependencies are reinstalled only when a manifest or lockfile changes, and the health probe uses exponential backoff (100ms → 1s) rather than the deploy path's fixed 3s delay plus 2s retry gaps. Falls back to a full-tree sync when a change set is large or contains a delete, which `--files-from` cannot express. Requires `--app <name>` in a multi-app workspace; mutually exclusive with `--dry-run`.
9+
- This is deliberately not a release: it patches the release that is already serving, so there is no rollback target, reload can drop in-flight requests, and local deletes do not propagate until the next full deploy. Watch mode says so on startup.
10+
- Blue-green apps reload only the colour serving traffic (via that release's `ecosystem.web.config.cjs`), leaving the idle colour untouched so it remains a valid rollback target.
11+
- The deploy lock is held for each cycle, so a concurrent `shipnode deploy` can never interleave with a sync — it is skipped with a notice instead.
12+
- **`--build <remote|local|none>`** controls where each cycle builds. `remote` (backend default) builds on the server. `local` builds here (from `appRoot` in a monorepo) and ships the artifact. `none` (implied by `--skip-build`) only syncs and reloads, which is what pairs with a framework's own watch mode. Projects that build locally and upload the bundle — Nitro/TanStack Start/Nuxt apps deployed with `--skip-build` — need `local` or `none`; on `remote` their `.output/` is ignored and the loop would ship source the app never runs.
13+
- The watcher is suppressed while a build shipnode runs is writing. Ignoring build *directories* is not sufficient: a TanStack Start/Nitro build regenerates files inside the source tree (`routeTree.gen.ts`) and drops temp files at the repo root, which are indistinguishable from a developer's edit by path alone. Left unguarded this feeds each cycle back into itself — observed in the wild as a `pm2 reload` of the live process every ~8 seconds. Gating on *when* we build fixes it for any framework's codegen.
14+
- Build output is watched only when shipnode is not the thing writing it (`none`). Under `local` the cycle runs the build itself, so watching its output would feed those writes back in and rebuild forever; that mode instead syncs by full-tree rsync, which detects the fresh artifact without the watcher reporting it.
15+
- The watcher reads `.shipnodeignore` and skips those directories, so a build cache like `.nitro/` no longer costs a full lock/sync/reload/probe cycle to transfer nothing.
16+
- Ctrl-C during a cycle releases the deploy lock instead of leaking it. Exiting straight from the signal handler skipped the `finally` that releases it, leaving a lock no process owned and blocking every later deploy until someone ran `shipnode unlock`. A second Ctrl-C still exits immediately.
17+
- The debounce has a max-wait ceiling (2s). A plain debounce resets on every event, so in a repo with a background writer — a turbo daemon, a framework's own watcher — changes would sit unemitted for as long as the writing continued, and the loop would appear hung.
18+
19+
### Fixed
20+
- **SSH keepalives on long-lived sessions**`deploy --watch` and `monitor` sit idle between commands, where a NAT or firewall timeout would silently drop the connection and fail the next exec. Connections now send keepalives every 15s, and `connect` starts from a fresh client so reconnecting doesn't accumulate listeners.
21+
- **`.env` symlinks survive a rebuild** — the build-output symlink logic moved to `envSymlinkCommand` and now re-runs after every hot-sync build, since a build that wipes and recreates its output directory takes the symlink with it. The initial deploy path is unchanged.
22+
723
## [3.2.0-alpha.0] - 2026-07-11
824

925
### Added

CONTEXT.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@ Stages source files, installs dependencies on the remote host, builds remotely,
4747
### FrontendStrategy
4848
Builds locally, stages build output, and relies on Caddy to serve static files.
4949

50+
### HotSync
51+
One iteration of `deploy --watch`: rsync changed files into the **live** release, reinstall only when a dependency manifest moved, rebuild, reload the running processes, probe health. Deliberately *not* a Release — it mutates what is already serving, so there is no rollback target. Trades the release pipeline's safety for a sub-second edit-to-live loop.
52+
53+
### ProjectWatcher
54+
The local file-watching seam behind `deploy --watch`. Emits debounced batches of changed repo-relative paths. Prefers recursive `fs.watch` and falls back to an mtime-scan poller where that is unavailable.
55+
5056
### Hook
5157
User-provided function that runs at a fixed point in the deployment lifecycle. `preDeploy` runs before the app goes live; `postDeploy` runs after cleanup.
5258

@@ -70,4 +76,5 @@ Owns config-loading and SSH lifecycle for CLI commands. Each command is pure bus
7076
- **Config seam**: `assembleConfig` is the only entry point from raw config → trusted config.
7177
- **Remote seam**: `RemoteExecutor` is the only way services talk to the remote host.
7278
- **Deployment seam**: `DeployOrchestrator` + `DeploymentStrategy` split invariant sequence from app-specific behaviour.
79+
- **Watch seam**: `ProjectWatcher` (local file events) and `HotSync` (remote patch + reload) are independent; the watch session owns the deploy lock, cycle serialisation, and reconnection.
7380
- **CLI seam**: `runRemoteCommand` / `runLocalCommand` separate connection ceremony from command logic.

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,7 @@ shipnode setup # Install Node, PM2, Caddy, fail2ban on server
253253
shipnode deploy # Deploy (zero-downtime by default)
254254
shipnode deploy --dry-run # Preview without making changes
255255
shipnode deploy --skip-build # Skip local build step
256+
shipnode deploy --watch # Deploy once, then sync + reload on every save
256257
shipnode doctor # Check local + remote config
257258
shipnode doctor --security # Run security audit
258259
shipnode status # Show PM2 process status

src/cli/commands/deploy-watch.ts

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
import chalk from 'chalk';
2+
import { readFile } from 'fs/promises';
3+
import { resolve } from 'path';
4+
import type { ShipnodeConfig, ShipnodeApp } from '../../shared/types.js';
5+
import { SshConnection } from '../../infrastructure/ssh/connection.js';
6+
import { LoggingExecutor } from '../../infrastructure/ssh/logging-executor.js';
7+
import { DeployService } from '../../services/deploy.service.js';
8+
import { HealthCheckService } from '../../services/health.service.js';
9+
import { DeployLock } from '../../domain/release/manager.js';
10+
import { HotSync, type BuildLocation, type HotSyncResult } from '../../domain/deploy/hot-sync.js';
11+
import { parseIgnoreFileDirs, watchProject, type ProjectWatcher } from '../../domain/deploy/watcher.js';
12+
import { LockError } from '../../shared/errors.js';
13+
import { ui } from '../ui.js';
14+
15+
/**
16+
* `shipnode deploy --watch` — the development loop.
17+
*
18+
* One full deploy establishes a coherent baseline (dependencies installed,
19+
* release health-checked, Caddy wired), then every local edit is rsynced into
20+
* that live release and the processes are reloaded. The first deploy is the
21+
* slow one; each edit after it is a hot sync.
22+
*
23+
* The session owns three things the hot sync itself does not: the deploy lock
24+
* (so a concurrent `shipnode deploy` can never interleave with a sync),
25+
* serialisation of cycles (edits arriving mid-sync are coalesced into the next
26+
* one), and reconnecting an SSH session that dropped while idle.
27+
*/
28+
export async function runDeployWatch(
29+
cwd: string,
30+
config: ShipnodeConfig,
31+
app: ShipnodeApp,
32+
options: { buildLocation: BuildLocation },
33+
): Promise<void> {
34+
const ssh = new SshConnection();
35+
await ssh.connect(config.ssh);
36+
37+
const executor = new LoggingExecutor(ssh);
38+
const lock = new DeployLock(ssh, config.remotePath);
39+
40+
let watcher: ProjectWatcher | undefined;
41+
42+
const hotSync = new HotSync(
43+
config,
44+
app,
45+
executor,
46+
cwd,
47+
new HealthCheckService(executor, config),
48+
{
49+
buildLocation: options.buildLocation,
50+
// Deliberately late-bound: the watcher is created after the baseline
51+
// deploy, but the build that deploy runs must be suppressed too.
52+
suppressWatch: {
53+
pause: () => watcher?.pause(),
54+
resume: () => watcher?.resume(),
55+
},
56+
},
57+
);
58+
59+
let stopping = false;
60+
let lockHeld = false;
61+
62+
/**
63+
* Release the deploy lock before exiting.
64+
*
65+
* Ctrl-C lands while a cycle may be holding the lock, and exiting there
66+
* would skip the `finally` that releases it — leaving a lock no process
67+
* owns, which blocks every later deploy until someone runs `shipnode
68+
* unlock`. A second Ctrl-C bypasses this and exits immediately, so a wedged
69+
* release can never trap the user.
70+
*/
71+
const shutdown = (): void => {
72+
if (stopping) {
73+
process.exit(130);
74+
}
75+
stopping = true;
76+
watcher?.close();
77+
78+
const finish = (): never => {
79+
ssh.disconnect();
80+
ui.outro('Watch stopped.');
81+
process.exit(0);
82+
};
83+
84+
if (!lockHeld) finish();
85+
86+
process.stdout.write(chalk.dim(' releasing deploy lock…\n'));
87+
const timeout = setTimeout(() => {
88+
ui.warn('Could not release the deploy lock. Run `shipnode unlock` before the next deploy.');
89+
finish();
90+
}, 5000);
91+
92+
void lock
93+
.release()
94+
.catch(() => {
95+
ui.warn('Could not release the deploy lock. Run `shipnode unlock` before the next deploy.');
96+
})
97+
.finally(() => {
98+
clearTimeout(timeout);
99+
finish();
100+
});
101+
};
102+
103+
process.on('SIGINT', shutdown);
104+
process.on('SIGTERM', shutdown);
105+
106+
try {
107+
ui.banner();
108+
ui.step(`Watching ${chalk.bold(app.name)}${config.ssh.user}@${config.ssh.host}`);
109+
ui.warn(
110+
'Watch mode patches the release that is serving traffic — no new release, ' +
111+
'no rollback target, and reload can drop in-flight requests. Use plain ' +
112+
'`shipnode deploy` for anything you need to be able to roll back.',
113+
);
114+
115+
// The baseline deploy runs with skipBuild for local/none, so a local build
116+
// has to happen here or the first release ships stale build output.
117+
if (options.buildLocation === 'local') {
118+
ui.step('Building locally…');
119+
await hotSync.buildLocally();
120+
}
121+
122+
ui.step(`Initial deploy… ${chalk.dim(`(build: ${options.buildLocation})`)}`);
123+
// The baseline deploy builds on the server only when the loop will too.
124+
await new DeployService(executor, config).execute(cwd, options.buildLocation !== 'remote');
125+
126+
let busy = false;
127+
const pending = new Set<string>();
128+
129+
const drain = async (): Promise<void> => {
130+
busy = true;
131+
try {
132+
while (pending.size > 0 && !stopping) {
133+
const batch = [...pending];
134+
pending.clear();
135+
await runCycle(batch);
136+
}
137+
} finally {
138+
busy = false;
139+
}
140+
};
141+
142+
const runCycle = async (batch: string[]): Promise<void> => {
143+
const label = batch.length === 1 ? batch[0] : `${batch.length} files`;
144+
process.stdout.write(`${chalk.dim('│')} ${chalk.cyan('⟳')} ${label}\n`);
145+
146+
try {
147+
await withLock(lock, async () => {
148+
lockHeld = true;
149+
const result = await syncWithReconnect(ssh, config, hotSync, batch);
150+
reportCycle(result);
151+
}, () => {
152+
lockHeld = false;
153+
});
154+
} catch (error) {
155+
if (error instanceof LockError) {
156+
process.stdout.write(
157+
`${chalk.dim('│')} ${chalk.yellow('⏸')} another deploy holds the lock — skipped\n`,
158+
);
159+
return;
160+
}
161+
// A failed sync is expected during development (a type error, a crashed
162+
// boot). Report it and keep watching; the next save retries.
163+
const message = error instanceof Error ? error.message : String(error);
164+
process.stdout.write(`${chalk.dim('│')} ${chalk.red('✗')} ${message}\n`);
165+
}
166+
};
167+
168+
watcher = watchProject(cwd, {
169+
// Watch build output only when nothing in this process writes it — that
170+
// is, when the developer or their framework owns the build (`none`).
171+
// Under `local` *we* run the build, so watching its output would feed
172+
// our own writes back in as changes and rebuild forever. Watching is not
173+
// the same as syncing: `local` still ships the artifact, via a full-tree
174+
// rsync that detects it without the watcher's help.
175+
watchBuildOutput: options.buildLocation === 'none',
176+
ignoredDirs: await readIgnoredDirs(cwd),
177+
onBatch: (paths) => {
178+
for (const path of paths) pending.add(path);
179+
if (!busy) void drain();
180+
},
181+
onError: (error) => ui.warn(`watcher: ${error.message}`),
182+
});
183+
184+
ui.success(
185+
`Ready — editing files in ${cwd} syncs to the live release` +
186+
(watcher.mode === 'poll' ? chalk.dim(' (polling: recursive fs.watch unavailable)') : ''),
187+
);
188+
process.stdout.write(chalk.dim(' Ctrl-C to stop.\n'));
189+
190+
// Hold the process open; the watcher drives everything from here.
191+
await new Promise<never>(() => {});
192+
} catch (error) {
193+
watcher?.close();
194+
ssh.disconnect();
195+
throw error;
196+
}
197+
}
198+
199+
/**
200+
* Directory names from `.shipnodeignore`, so the watcher does not wake up for
201+
* paths rsync would refuse to transfer anyway.
202+
*/
203+
async function readIgnoredDirs(cwd: string): Promise<string[]> {
204+
try {
205+
return parseIgnoreFileDirs(await readFile(resolve(cwd, '.shipnodeignore'), 'utf8'));
206+
} catch {
207+
return [];
208+
}
209+
}
210+
211+
/**
212+
* Acquire the deploy lock, run `body`, and always release it.
213+
*
214+
* `onReleased` runs after the lock is gone so the caller can stop tracking it —
215+
* the signal handler uses that to know whether it still has a lock to clean up.
216+
*/
217+
async function withLock(
218+
lock: DeployLock,
219+
body: () => Promise<void>,
220+
onReleased: () => void = () => {},
221+
): Promise<void> {
222+
await lock.acquire();
223+
try {
224+
await body();
225+
} finally {
226+
await lock.release();
227+
onReleased();
228+
}
229+
}
230+
231+
/**
232+
* Run a hot sync, reconnecting once if the SSH session died while idle.
233+
*
234+
* A watch session sits idle for long stretches; even with keepalives a laptop
235+
* suspend or a NAT timeout can drop the connection. Reconnecting beats making
236+
* the user restart the session.
237+
*/
238+
async function syncWithReconnect(
239+
ssh: SshConnection,
240+
config: ShipnodeConfig,
241+
hotSync: HotSync,
242+
batch: string[],
243+
): Promise<HotSyncResult> {
244+
if (!ssh.isConnected()) {
245+
process.stdout.write(`${chalk.dim('│')} ${chalk.yellow('⟲')} reconnecting…\n`);
246+
await ssh.connect(config.ssh);
247+
}
248+
249+
try {
250+
return await hotSync.run(batch);
251+
} catch (error) {
252+
if (ssh.isConnected()) throw error;
253+
process.stdout.write(`${chalk.dim('│')} ${chalk.yellow('⟲')} connection lost — reconnecting…\n`);
254+
await ssh.connect(config.ssh);
255+
return hotSync.run(batch);
256+
}
257+
}
258+
259+
function reportCycle(result: HotSyncResult): void {
260+
const parts: string[] = [
261+
`${result.transferredFiles} file${result.transferredFiles === 1 ? '' : 's'}`,
262+
];
263+
if (result.mode === 'full') parts.push('full scan');
264+
if (result.installed) parts.push('installed');
265+
if (result.built) parts.push('built');
266+
if (result.reloaded) parts.push('reloaded');
267+
268+
const seconds = (result.durationMs / 1000).toFixed(2);
269+
const health =
270+
result.health === 'passed'
271+
? chalk.green('healthy')
272+
: result.health === 'failed'
273+
? chalk.red('unhealthy')
274+
: chalk.dim('no health check');
275+
276+
process.stdout.write(
277+
`${chalk.dim('│')} ${chalk.green('✓')} ${parts.join(', ')} · ${health} · ${chalk.bold(`${seconds}s`)}\n`,
278+
);
279+
280+
if (result.health === 'failed' && result.healthError) {
281+
process.stdout.write(`${chalk.dim('│')} ${chalk.dim(result.healthError.split('\n')[0])}\n`);
282+
}
283+
}

0 commit comments

Comments
 (0)