Skip to content

fix: pin app compose commands so an app op can never take down the core stack#161

Merged
max-tet merged 5 commits into
mainfrom
fix/clayde/pin-app-compose-commands
Jul 20, 2026
Merged

fix: pin app compose commands so an app op can never take down the core stack#161
max-tet merged 5 commits into
mainfrom
fix/clayde/pin-app-compose-commands

Conversation

@ClaydeCode

@ClaydeCode ClaydeCode commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #160.

Problem

Every app compose invocation was unpinned — docker compose <cmd> with only cwd, no -f, no -p. Compose v2 derives both the compose file and the project name from the working directory and walks up the tree when cwd has no compose file. App dirs live under the core dir:

/core/docker-compose.yml                        <- CORE stack (postgres, traefik, shard_core, web-terminal)
/core/installed_apps/<app>/docker-compose.yml   <- per-app stack

So an app command run from installed_apps/<app> with a missing compose file resolved to /core/docker-compose.yml, project name core — and stop/down then stopped shard_core itself through the mounted docker socket. That is the root cause of the recent full-shard outage: core stopped itself while uninstalling a malformed custom app.

Fix

  • New app_compose_command(app_dir) in shard_core/util/subprocess.py builds a command pinned with -f <app_dir>/docker-compose.yml -p <project> --project-directory <app_dir>.
  • It refuses to run when the app's compose file is missing (ComposeFileNotFound) — compose never gets the chance to walk up.
  • Defense in depth: ComposeProjectNotAllowed when the resolved project would be core (or is empty).
  • Every app compose call now goes through it: up --no-start, up -d, down, stop, pause, unpause in app_tools.py, plus the pause-time ps -q lookup in memory_pressure.reclaim_compose_stack.

Project name compatibility

The pinned -p must keep matching the project name compose previously derived from the app dir name, or already-installed app stacks would be orphaned. normalize_project_name mirrors compose's own normalization (lowercase, drop everything outside [a-z0-9_-], strip leading _/-), verified empirically against docker compose config (v5.0.2): My_App.v2-xmy_appv2-x, _-Foofoo. This matters because custom-app names come from the uploaded zip filename and can contain uppercase or dots.

Blast radius of the new error

Failing loudly is the point, and the existing callers already contain it: _uninstall_app and _reinstall_app wrap stop/shutdown in try/except Exception (logged), and the installation worker loop catches every exception per task. So uninstalling a malformed app now logs the error, removes the app dir and DB row, and leaves core running — instead of taking the shard offline. docker_stop_all_apps / docker_shutdown_all_apps gather with return_exceptions=True.

Tests

New tests/test_app_compose_pinning.py (16 tests, unit tier — no Docker):

  • Acceptance from the issue: with installed_apps/<app>/docker-compose.yml deleted, each of docker_create_app_containers, docker_start_app, docker_pause_app, docker_unpause_app, docker_stop_app, docker_shutdown_app raises ComposeFileNotFound and runs no compose command at all (subprocess never called), so the core project cannot be touched.
  • With the compose file present, the command carries -f and -p for the app.
  • Helper-level: missing file raises, core project rejected, project-name normalization cases.

Verified to have teeth: checked out the pre-fix app_tools.py / memory_pressure.py and re-ran the file — 7 failed, 9 passed. With the fix: 16 passed.

One existing test needed fixing, and the reason is the guard doing its job: test_reclaim_compose_stack_reclaims_each_container called reclaim_compose_stack for an app dir that never existed, which now raises ComposeFileNotFound. That state is unreachable in production — reclaim only runs right after a successful compose pause — so the test now creates the compose file it implicitly assumed.

Note: docker_start_app's @throttle(5) is global across apps and tests — a start triggered by an earlier test would silently drop the call under test and fail it spuriously, so the throttle clock is reset in an autouse fixture.

Local run: 2 failed, 188 passed — the memory_pressure one above (now fixed) and test_app_lifecycle.py::test_app_starts_and_stops, which fails only on this dev box because another container already binds host port 80 (Bind for 0.0.0.0:80 failed: port is already allocated); it passed on CI's clean runner. CI's first run: 1 failed, 189 passed — only the memory_pressure test, fixed in the last commit.

Recommended reading order

  1. shard_core/util/subprocess.py — the pinned-command helper and its guards
  2. shard_core/service/app_tools.py — all app compose calls routed through it
  3. shard_core/service/memory_pressure.py — the one remaining cwd-based call
  4. agents.md
  5. tests/test_app_compose_pinning.py
  6. tests/test_memory_pressure.py

Compose v2 discovers both the compose file and the project name from the
working directory, walking up the tree when cwd has no compose file. App dirs
live under the core dir, so an app command run with cwd=installed_apps/<app>
and a missing compose file resolves to /core/docker-compose.yml with project
name "core" — the live core stack.

app_compose_command builds an explicitly pinned command (-f, -p,
--project-directory), refuses to run when the app's compose file is missing,
and guards against a resolved project name of "core". The project name mirrors
compose's own normalization of the directory name so existing app stacks keep
their identity.
All app compose invocations relied on cwd for file and project discovery. When
an app's docker-compose.yml is missing (e.g. a malformed custom-app zip),
compose walked up to the core compose file and ran against project "core" —
an app stop/down then stopped shard_core itself, taking the shard offline.

Route every app compose call (including the pause-time `ps -q` lookup in
memory_pressure) through app_compose_command, which pins -f/-p and refuses to
run at all when the app's compose file is absent.
Covers issue #160's acceptance: with installed_apps/<app>/docker-compose.yml
deleted, every app operation errors on the missing file and runs no compose
command at all, so the core project is never touched. Verified to fail against
the pre-fix cwd-based calls (7 failures).

The project-name normalization cases were checked against `docker compose
config` (v5.0.2) so pinned -p keeps matching the project compose derived from
the app dir name for already-installed apps.
@ClaydeCode
ClaydeCode requested a review from max-tet July 14, 2026 21:27
reclaim_compose_stack now goes through app_compose_command, which refuses to
run without the app's compose file. The test called it for an app dir that
never existed — a state unreachable in production, since reclaim only runs
right after a successful `compose pause`. Create the compose file the test
implicitly assumed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P0] Unpinned compose commands: an app stop/down can escalate to the core project and kill shard_core

2 participants