Skip to content

Commit bdf8d36

Browse files
committed
Scale the CI timeout with replica count; record the build-once decision
A roll installs and builds on every replica in turn, inside one job. The generated workflow hard-coded 30 minutes, which is comfortable for one server and tight for four — and a timeout mid-roll is the worst place to stop, leaving the fleet on mixed versions with a replica drained. ADR-0007 now records build-once as considered and declined rather than merely absent. With a committed lockfile the per-replica builds are equivalent, so it is a wall-clock cost rather than a correctness gate, and both implementations cost more than they save: building on the primary and copying needs SSH between servers, a new trust edge in a tool whose premise is that it needs nothing but your SSH access to each box; building locally fails silently for native modules when the developer's platform differs from the servers'.
1 parent 3841ade commit bdf8d36

4 files changed

Lines changed: 21 additions & 3 deletions

File tree

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,8 @@ export default shipnode
252252
253253
Replicas serve plain HTTP on the private network and never claim the domain — five replicas all requesting a certificate for the same name would race Let's Encrypt. **TLS terminates at your load balancer.**
254254

255+
Each replica installs and builds, so a three-replica roll takes roughly three times a single deploy. `shipnode ci github` sizes the workflow timeout accordingly.
256+
255257
| Setting | Default | Description |
256258
|---|---|---|
257259
| `.group(name, servers)` || Name a set of servers so `on` can target them all at once |

docs/adr/0007-fleet-replication.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,4 +84,10 @@ Accessories stay single-node. Shipnode replicating Postgres is out of scope, and
8484
- **A shipnode-managed load balancer** — a Cloudflare tunnel with N connectors, or an edge Caddy node. The drain contract is designed so either could be added later as an alternative drain implementation without disturbing the roller.
8585
- **Pass-through TLS.** Replicas serve plain HTTP; the LB terminates.
8686
- **Replicated accessories.** One Postgres, on one server.
87-
- **Building once and shipping the artifact.** Each replica still builds. With a committed lockfile the builds are equivalent, so this is a time optimisation rather than a correctness gate, and the two ways to implement it — building locally, or copying between servers — each impose an infrastructure requirement that has not been chosen.
87+
- **Building once and shipping the artifact.** Each replica installs and builds, so an N-replica roll costs N builds. This was considered and declined.
88+
89+
The case for it is that a floating dependency could resolve differently on each replica and produce N different artifacts. With a committed lockfile — the normal case — it cannot, so this is a wall-clock cost rather than a correctness gate. Frontends are already unaffected: `FrontendStrategy` builds locally and rsyncs only `buildDir`.
90+
91+
Both implementations impose an infrastructure requirement that outweighs the saving. Building on the primary and copying to its peers needs SSH *between servers*, a new trust edge in a tool whose whole premise is that it needs nothing but your SSH access to each box. Building locally needs the developer's machine or CI runner to produce server-compatible binaries, which fails silently for native modules — `sharp`, `esbuild`, `better-sqlite3`, Prisma engines — when someone deploys from macOS arm64 to linux x64.
92+
93+
Revisit if fleets get large enough that build time dominates, or if shipnode grows a container image path where the artifact is already portable by construction.

src/cli/commands/ci.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,11 @@ function generateWorkflow(
177177
: '';
178178
const commandOptions = renderCommandOptions(options);
179179

180+
// A roll installs and builds on every replica in turn, inside one job. Thirty
181+
// minutes is comfortable for one server and tight for four, and a timeout
182+
// mid-roll leaves the fleet on mixed versions with a replica drained.
183+
const deployTimeoutMinutes = 30 * Math.max(1, hosts.length);
184+
180185
const knownHostsScan = hosts.length > 0 ? hosts.join(' ') : '<your-server>';
181186
// Fail before the roll rather than during it. A host missing from
182187
// known_hosts otherwise surfaces as an SSH failure on replica three, with
@@ -238,7 +243,7 @@ concurrency:
238243
jobs:
239244
deploy:
240245
runs-on: ubuntu-latest
241-
timeout-minutes: 30
246+
timeout-minutes: ${deployTimeoutMinutes}
242247
environment: ${yamlSingleQuoted(environment)}
243248
244249
steps:

tests/unit/ci-github.test.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,9 @@ describe('ci github', () => {
200200
expect(workflow).toContain('Verify every replica is a known host');
201201
expect(workflow).toContain("for host in '10.0.0.11' '10.0.0.12'");
202202
expect(workflow).toContain('ssh-keyscan 10.0.0.11 10.0.0.12');
203+
// Each replica installs and builds in turn inside one job; a timeout
204+
// mid-roll leaves the fleet on mixed versions with a replica drained.
205+
expect(workflow).toContain('timeout-minutes: 60');
203206
});
204207

205208
it('omits the replica check for a single-server workspace', async () => {
@@ -216,6 +219,8 @@ describe('ci github', () => {
216219

217220
await cmdCiGithub(root, { app: 'api', syncEnv: true });
218221

219-
expect(await readWorkflow(root)).not.toContain('Verify every replica is a known host');
222+
const workflow = await readWorkflow(root);
223+
expect(workflow).not.toContain('Verify every replica is a known host');
224+
expect(workflow).toContain('timeout-minutes: 30');
220225
});
221226
});

0 commit comments

Comments
 (0)