Skip to content

Commit 42b9e0a

Browse files
Merge pull request #128 from NASA-IMPACT/feature/mmgis-deployment-skill
Add mmgis-deployment skill: scripted local deployments with seeded golden baseline
2 parents 700e711 + cba765a commit 42b9e0a

113 files changed

Lines changed: 10095 additions & 890 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
name: mmgis-deployment
3+
description: Use when deploying MMGIS locally — standing up or booting a dev instance, running several MMGIS deployments or git worktrees in parallel, tearing one down, or debugging a local deployment that won't start or serve.
4+
---
5+
6+
# Deploying MMGIS locally
7+
8+
## Overview
9+
10+
A **deployment** is one MMGIS instance: a server process on a `PORT`, a database named `DB_NAME`, an `.env`, and its own `node_modules`, in one directory. All deployments share a single Postgres+PostGIS container and coexist simply by using different `PORT` and `DB_NAME` values. A git worktree is just one convenient way to get an independent directory — a separate clone, or the main checkout itself, works the same.
11+
12+
Each deployment has its **own database**, so its Configure page, users, and missions are fully independent and persistent. New deployments start from a frozen baseline database (`mmgis_golden`) so they boot as a working app (admin user + a mission) with no manual setup.
13+
14+
To understand the internals (the single-instance deploy and the multi-instance pattern), read `references/deployment-model.md`.
15+
16+
## Prerequisites
17+
18+
- The shared DB container must be running. From the main checkout: `npm run db:start`.
19+
- `mmgis_golden` must exist. On a fresh machine run `scripts/seed-golden.sh` once — it builds the baseline (admin/admin + the `arst` mission) from the committed seed via a temporary server; no existing database needed. Set `MAPBOX_TOKEN` (env or main `.env`) first if you want basemaps to render — tokens are never committed to git. `scripts/refresh-golden.sh` instead re-snapshots the baseline from a live database you already have.
20+
21+
## Routing — which script for which intent
22+
23+
Scripts live in `scripts/` next to this file. Invoke them by path; each takes a deployment directory (default: current dir) or, for `create`/`teardown`, a name.
24+
25+
| Intent | Command |
26+
|--------|---------|
27+
| Understand how local deploy works | read `references/deployment-model.md` |
28+
| Provision a new deployment | `scripts/create.sh <name>` (new worktree) or `scripts/create.sh --here` |
29+
| Boot the dashboard | `scripts/start.sh <dir>` — prints the dashboard URL when healthy |
30+
| Stop the server | `scripts/stop.sh <dir>` |
31+
| See everything running | `scripts/list.sh` |
32+
| Diagnose a sick deployment | `scripts/doctor.sh <dir>` |
33+
| Run tests | `scripts/test.sh <dir> [unit\|e2e\|all]` |
34+
| Remove a deployment | `scripts/teardown.sh <name\|dir>` |
35+
| Bootstrap the baseline on a fresh machine | `scripts/seed-golden.sh` |
36+
| Re-baseline from a live DB | `scripts/refresh-golden.sh [source-db]` |
37+
38+
A typical new-feature flow: `create.sh <name>``start.sh <dir>` → open the dashboard URL. When done: `teardown.sh <name>`.
39+
40+
## Autonomy and safety
41+
42+
Run these **autonomously** when the user's intent is clear: `create`, `start`, `stop`, `list`, `doctor`, `test`, and `seed-golden` when no golden exists (it refuses to overwrite one without `--force`).
43+
44+
**Confirm with the user first** — show exactly what will change, then wait for explicit approval — before:
45+
46+
- `teardown.sh` — drops a database and removes a worktree. It refuses when the deployment has uncommitted or unpushed work unless `--force`; never bypass that for the user without surfacing the warning. Show its printed plan and get a yes.
47+
- `refresh-golden.sh`, or `seed-golden.sh --force` — both overwrite the baseline all future deployments clone from.
48+
49+
## Common mistakes
50+
51+
- **Using `npm start` in a coexisting deployment.** Its `prestart` hook starts another DB container fighting for port 5432. Coexisting deployments use `start:no-docker` (which `start.sh` does). The main checkout may still use `npm start`.
52+
- **Cloning a deployment DB from the live `mmgis` database.** Postgres can't use a connected database as a clone template. Always clone the frozen `mmgis_golden` (`create.sh` does this).
53+
- **Expecting `refresh-golden` to update existing deployments.** It only changes the baseline for *future* clones; existing deployments keep their own databases.
54+
- **Reaching for `FORCE_CONFIG_PATH`.** It forces one read-only mission and bypasses the Configure system — special-case only, not how normal deployments are configured.
55+
- **Running e2e while another server holds port 8888.** Playwright would attach to it and test the wrong code. `test.sh e2e` guards against this.
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# MMGIS local deployment model
2+
3+
How MMGIS runs on a developer machine — both the simple single-instance case and the multi-instance pattern this skill automates.
4+
5+
## What a deployment is
6+
7+
A deployment is one MMGIS instance: a server process on a `PORT`, a database named `DB_NAME`, an `.env`, and its own `node_modules`, in one directory. The only backing service required is **one Postgres+PostGIS container**. STAC / TiTiler / tipg / veloserver are optional (`--profile stac`, `--profile veloserver`) and off by default — not needed for a dashboard or for tests.
8+
9+
## The three run modes (from package.json)
10+
11+
- **`npm start`** — has a `prestart` hook that runs `docker-compose -f docker-compose.db.yml up -d --wait` to bring up a DB container, then `node scripts/init-db.js && node scripts/server.js`. The vanilla single-machine path.
12+
- **`npm run start:no-docker`**`init-db.js` + `server.js` with **no** `prestart` hook (npm only fires `prestart` before `start`). Assumes a DB is already reachable. **This is what coexisting deployments use**, because the `prestart` container always binds host port 5432 and would collide when a shared container already holds it.
13+
- **`npm run start:test`**`NODE_ENV=test PORT=8888 node scripts/server.js`. What Playwright auto-launches for e2e.
14+
15+
## Database
16+
17+
- **Auto-init.** `init-db.js` creates the `$DB_NAME` database, the `postgis` + `btree_gist` extensions, the `session` table, and spatial indexes. Idempotent ("already exists → nothing to do").
18+
- **Auto-schema.** `server.js` runs Sequelize `.sync()`, which creates all tables on boot. **No manual migrations.**
19+
- **Cross-branch schema.** `.sync()` reconciles *additive* changes automatically. Dropping/renaming columns across branches is the only case needing manual care.
20+
21+
## Ports
22+
23+
- In `NODE_ENV=development`: the API + WebSocket run on `PORT`; the **dashboard UI is served by webpack-dev-server on `PORT+1`**. `/` on `PORT` redirects to `PORT+1`. So dev uses **two** adjacent ports.
24+
- In test/production: everything is on `PORT` (no `+1`).
25+
- The API healthcheck is `GET /api/utils/healthcheck` on `PORT` (returns "Alive and Well!"). It comes up before webpack finishes its first compile, so the API can be healthy while the dashboard is still compiling for a few seconds.
26+
27+
## Coexisting many deployments
28+
29+
They differ only in `PORT` and `DB_NAME`, against one shared container:
30+
31+
- **One shared Postgres container.** `docker-compose.db.yml` hardcodes host `5432:5432`, so only one can run. Start it once from the main checkout (`npm run db:start`); every deployment points at `localhost:5432` with a distinct `DB_NAME`. Coexisting deployments launch with `start:no-docker`.
32+
- **Ports stepped by 10** (dev burns `PORT` and `PORT+1`), starting at 8888 and skipping any already in an `.env` or listening.
33+
- **`.env` and `node_modules` are gitignored** — never present in a fresh worktree; each deployment provisions its own (`npm install --force`; `--force` is required by this dependency tree, not `--legacy-peer-deps`).
34+
35+
## Config: how a deployment gets a working dashboard
36+
37+
A fresh database has no admin and no missions, so the landing page would be empty. Two ways to populate it:
38+
39+
- **Golden clone (default).** A frozen baseline database `mmgis_golden` holds a baseline admin (`admin`, permission `111`) and a mission. Each new deployment's database is created as `CREATE DATABASE <db> TEMPLATE mmgis_golden`, so it boots already populated and then evolves independently. Postgres requires the template to have no active connections — that's why `mmgis_golden` is frozen (nothing connects to it) and why you must never clone from the live `mmgis` database.
40+
- **First-run signup.** Without a golden, the `/first_signup` endpoint makes the first account created a full admin (permission `111`); you then build missions in the Configure page.
41+
42+
Where the golden itself comes from — it lives only in the local Docker volume, so each machine builds its own:
43+
44+
- **Seed (fresh machine).** `seed-golden.sh` builds it from the committed `seed/baseline-mission.json`: it boots a temporary server against a scratch database (Sequelize creates the schema), seeds an admin + the baseline mission through MMGIS's own APIs (`first_signup``login``/api/configure/add`), then renames the scratch DB to `mmgis_golden`. The basemap token is injected at seed time from `MAPBOX_TOKEN` (env or the main checkout's `.env`) — **tokens are never committed to git**; the seed file holds a `{{MAPBOX_TOKEN}}` placeholder.
45+
- **Snapshot (existing machine).** `refresh-golden.sh` re-baselines from a live database via `pg_dump` (default source: `mmgis`).
46+
47+
Note the golden does **not** survive `docker-compose down -v` or deleting the `mmgis_mmgis-local-db` volume — that wipes every database. Re-seed or re-snapshot afterward.
48+
49+
`FORCE_CONFIG_PATH=<file.json>` forces a single read-only mission from a file and **bypasses the Configure system** — a special case, not how normal deployments are configured.
50+
51+
## Tests
52+
53+
- **Unit (`npm run test:unit`)** — pure JS, no server, no database. Run anywhere in parallel, no collisions.
54+
- **E2E (`npm run test:e2e`)** — Playwright auto-starts the server via `start:test`, which hardcodes `PORT=8888`, and `reuseExistingServer` is on locally. If another server is already on 8888, Playwright attaches to it and tests the wrong code. Run e2e only when 8888 is free (the `test.sh` helper enforces this). Parallel e2e across deployments would need app changes and is out of scope.
55+
56+
## Gotchas
57+
58+
- **CRLF in `.env`.** Some `.env` files use Windows line endings; a naive shell read yields values with a trailing `\r` (e.g. role `mmgis\r` → "role does not exist"). Always strip CR when reading `.env` in shell.
59+
- **Stray DB containers.** Running plain `npm start` in a worktree spins up that worktree's *own* DB container (e.g. `mmgis-<name>-db-1`). These can linger (often in "Created" state) and confuse "which container is the shared one." The shared one is whichever publishes host port 5432.
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Troubleshooting MMGIS deployments
2+
3+
`doctor.sh` points here. Find the symptom, apply the fix.
4+
5+
## Server won't start / healthcheck never passes
6+
7+
| Symptom | Cause | Fix |
8+
|---------|-------|-----|
9+
| `start.sh` times out; log shows DB connection error | Shared DB container not running | `npm run db:start` in the main checkout |
10+
| Log: `role "mmgis\r" does not exist` or auth fails on a clean DB | CRLF in `.env` (trailing `\r` in values) | The scripts strip CR when reading `.env`; if you hand-edit, save with LF endings |
11+
| Log: `database "<name>" does not exist` and it isn't created | `DB_NAME` mismatch, or DB never cloned | `doctor.sh`; if DB missing, `create.sh` (clones golden) or clone manually |
12+
| `start.sh` says "already up on port N" | A server (maybe a stale one) is on that port | `stop.sh <dir>`, or find the listener with `lsof -nP -iTCP:N -sTCP:LISTEN` |
13+
| API healthy but dashboard (PORT+1) returns 000/connection refused | webpack-dev-server still compiling | Wait ~30–90s for the first compile; re-curl |
14+
15+
## Port conflicts
16+
17+
| Symptom | Cause | Fix |
18+
|---------|-------|-----|
19+
| Two deployments show the same PORT in `list.sh` | They were configured with the same `PORT` (e.g. both copied 8888) | Edit one deployment's `.env` to a free port (`create.sh` picks free ports automatically for new deployments) |
20+
| `start.sh` fails immediately, port in use | Another deployment or stray process holds the port | `lsof -nP -iTCP:<port> -sTCP:LISTEN`; stop the owner |
21+
| Plain `npm start` failed with port 5432 in use | `npm start`'s `prestart` tried to start a second DB container | Use `start:no-docker` (what `start.sh` does); remove the stray container `docker rm mmgis-<name>-db-1` |
22+
23+
## Database / golden
24+
25+
| Symptom | Cause | Fix |
26+
|---------|-------|-----|
27+
| `create.sh`: "mmgis_golden does not exist" | Baseline never created on this machine | `seed-golden.sh` (builds from the committed seed — works on a fresh machine) or `refresh-golden.sh` (snapshots a live `mmgis`) |
28+
| Basemap blank in a fresh deployment | Golden was seeded without `MAPBOX_TOKEN` | Set `MAPBOX_TOKEN` and `seed-golden.sh --force`, or paste a token into the mission's basemap settings in Configure |
29+
| Clone fails: "source database is being accessed by other users" | Tried to use a live DB as a TEMPLATE | Clone only from frozen `mmgis_golden`, never from `mmgis`; if refreshing golden, the dump/restore path avoids this |
30+
| New deployment opens to an empty landing page | DB cloned from a golden that had no mission, or `first_signup` not done | Refresh golden from a DB that has the mission, or sign up the first admin and build a mission |
31+
| `DROP DATABASE` fails: in use | Server still connected | `stop.sh <dir>` first; `teardown.sh` stops the server before dropping |
32+
33+
## Teardown
34+
35+
| Symptom | Cause | Fix |
36+
|---------|-------|-----|
37+
| "refusing: uncommitted or unpushed work" | Real source changes or local-only commits in the worktree | Review the listed files/commits; commit or push them, or re-run with `--force` if you truly want to discard |
38+
| `git worktree remove` fails: "contains modified or untracked files" | Build churn (node_modules, lockfile) in the worktree | Expected — `teardown.sh` falls back to `--force` for the worktree removal step automatically |
39+
| Branch still exists after teardown | `teardown.sh` removes the worktree + database but leaves the branch | Delete it yourself if unwanted: `git branch -D <branch>` |
40+
41+
## Tests
42+
43+
| Symptom | Cause | Fix |
44+
|---------|-------|-----|
45+
| e2e passes but didn't seem to test your changes | Playwright `reuseExistingServer` attached to another server on 8888 | `test.sh e2e` refuses when 8888 is busy; stop the other server and re-run |
46+
| e2e can't start its server | Port 8888 occupied | Free it; `test.sh` reports the conflict |
47+
| Unit tests fail to import modules | `node_modules` missing/incomplete | `npm install --force` in the deployment |
48+
49+
## Stray containers
50+
51+
`docker ps -a | grep postgis` may show extra `mmgis-<name>-db-1` containers from accidental `npm start` runs. The shared container is the one publishing host port 5432 (`docker ps --format '{{.Names}} {{.Ports}}'`). Remove strays: `docker rm <name>` (add `-f` if running).

0 commit comments

Comments
 (0)