feat: replace ENV with YAML config (#1755)

* wip: initial commit

* refactor(config): move compass.yaml to repo root and use typed config directly

* refactor(self-host): inline web build config and simplify installer

* fix(e2e): write playwright compass config at config-load time

The e2e webServer (dev.ts) now calls loadCompassConfig() instead of
reading process.env directly. Write a minimal .playwright-compass.yaml
at playwright.config.ts import time and pass its path via
COMPASS_CONFIG_FILE so the webServer finds it without a compass.yaml in
the repo root.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(config): correct ENV_FILE → CONFIG_FILE and tighten timezone schema

Two bugs caught in code review:

1. validate_existing_env_secrets() guarded on \$ENV_FILE which was
   removed in the env-to-yaml migration. The guard always returned early,
   silently skipping all secret placeholder checks on re-installs.
   Changed to \$CONFIG_FILE.

2. CompassConfigSchema accepted any string for runtime.timezone but the
   backend EnvSchema enforced z.enum(["Etc/UTC","UTC"]).  The mismatch
   produced a confusing TZ enum error at backend startup instead of a
   clear parse error at YAML load time. Moved the enum constraint into
   the shared schema so both web and backend catch it early.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(e2e): include ports section in playwright test config

dev.ts reads WEB_PORT from config.ports?.web (falls back to 9080),
so the test compass.yaml must declare ports.web to match the port
Playwright waits on.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(scripts): replace getCliConfig() with module-level cliConfig

Load and parse compass.yaml once at module init instead of re-reading
the file on every call. Callers now access typed fields directly
(cliConfig.urls.backendApi, etc.). DEV_BROWSER remains a plain
process.env read since it is not part of the config schema.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(ci): build staging compass.yaml from individual secrets/vars

Replaces the COMPASS_YAML blob secret with granular GitHub vars
(non-sensitive URLs, client IDs) and secrets (credentials, tokens).
The printf pattern mirrors Dockerfile.web — each field is a separate
auditable/rotatable value.

* fix(deploy): correct environment variable assignments for staging deployment

* docs: remove migration note from local-development.md

The .env.local → compass.yaml migration steps belong in the PR
description, not the long-term developer doc. The doc now covers
steady-state setup only.

* chore: cleanup configs

* refactor(config): reorganize YAML into web/backend containers

Move scattered fields into logical containers:
- web.port (was ports.web)
- backend.port, backend.apiUrl, backend.originsAllowed, backend.compassToken
  (were ports.backend, urls.backendApi, urls.cors, tokens.compassSync)
- tokens section now only holds googleCalendarNotification (optional)

Replace getGcalWebhookBaseURL() utility with a schema-level field:
GCAL_WEBHOOK_BASEURL is now always set during config parsing
(googleWebhook ?? backendApiUrl), making the fallback explicit and
removing the need for a lazy accessor function.

Also fixes two pre-existing bugs found during the refactor:
- google-import.service.ts imported from a non-existent api-base-url.util
- gcal.service.test.ts mocked that same phantom module; now mocks CONFIG directly
- google.oauth.client, supertokens.middleware, config.controller imported
  isGoogleConfigured from config.constants instead of config.util

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tests): update env.constants → config.constants across test files

- Replace all imports of ENV from env.constants with CONFIG as ENV
  from config.constants (compatible alias, no test logic changed)
- Fix google-watch-token.test.ts: mock config.constants instead of
  the now-removed env.constants module
- Export parseConfigFromEnv (was private parseConfig) for test access
- Rewrite config.constants.test.ts to match current behavior:
  - GCAL_WEBHOOK_BASEURL now falls back to BASEURL instead of undefined
  - Non-HTTPS webhook URLs are accepted (HTTPS check is in superRefine)
  - Remove getApiBaseURL tests (function no longer exists)
  - Import isGoogleConfigured from config.util
- Fix migration/seeder import paths for env.constants → config.constants

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): replace ENV with CONFIG

* refactor(config): dissolve urls section into web and backend containers

Move urls.frontend → web.url, urls.googleWebhook → backend.googleWebhook,
urls.health → backend.healthUrl. The urls group no longer exists; all URL
fields now live under the container they belong to.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(config): remove unused backend.healthUrl field

The field was only read by the self-host shell scripts as an optional
override for the health probe URL; the backend never consumed it at
runtime. The default (http://localhost:<port>/api/health) is sufficient.
Also fix stale urls.frontend / urls.health / ports.backend references
in the self-host/compass helper.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(config): move googleWebhook and notificationToken under google section

tokens.googleCalendarNotification → google.notificationToken
backend.googleWebhook → google.webhookUrl

Both fields are only relevant when Google is configured, so they belong
under the google group. Removes the now-empty tokens section entirely.
Also brings packages/backend/compass.yaml up to date with the current
schema (it still used ports/urls/tokens from before earlier migrations).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs: remove github section from config.md

* fix(config): walk up directory tree when searching for compass.yaml

Scripts run from a subdirectory (e.g. bun dev:web cd-s into packages/web)
would fail to find compass.yaml at the repo root because the old lookup
only checked process.cwd() exactly. Walking up mirrors how dotenv and
similar tools discover config files regardless of invocation directory.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(config): fail fast on REPLACE_WITH_* placeholder values at startup

Standardise example YAML placeholders to REPLACE_WITH_* (uppercase +
underscore) and add a recursive scanner in parseCompassConfigText that
fires after Zod validation. Any string value containing REPLACE_WITH_ —
including values embedded inside URIs — is collected and thrown as a
single error listing every affected field path. Since all entry points
(backend, scripts, web dev) call parseCompassConfigText, the guard fires
everywhere. The compass-self-host-placeholder Google values intentionally
do not use this prefix so leaving Google unconfigured stays valid.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs: add comments for compass token

* docs: update example config and doc

* refactor(config): remove Google OAuth placeholder constants in favor of empty/undefined values

* API Error: 529 {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"},"request_id":"req_011Cb3RMWeJgeK1XQGWyx1LF"}

* fix(self-host): use localhost URLs by default and prevent config loss with existing volumes

* fix(self-host): simplify installer check by removing gcal check

no need to overcomplicate with a standalone function for one key

* fix(backend): initialize mongo after connect resolves

* test: cleanup playwright config

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: tyler-dane

.github/workflows/deploy-staging.yml

@@ -25,14 +25,47 @@ jobs:
     steps:
       - name: Deploy release to staging
         env:
+          # Non-sensitive
+          BACKEND_API_URL: ${{ vars.STAGING_BACKEND_API_URL }}
+          FRONTEND_URL: ${{ vars.STAGING_FRONTEND_URL }}
+          GOOGLE_CLIENT_ID: ${{ vars.STAGING_GOOGLE_CLIENT_ID }}
           RELEASE_TAG: ${{ inputs.tag }}
+          SSH_USER: ${{ vars.STAGING_SSH_USER }}
+          SSH_HOST: ${{ vars.STAGING_SSH_HOST }}
+          SUPERTOKENS_URI: ${{ vars.STAGING_SUPERTOKENS_URI }}
+          # Sensitive 
+          COMPASS_SYNC_TOKEN: ${{ secrets.STAGING_COMPASS_SYNC_TOKEN }}
+          GCAL_NOTIFICATION_TOKEN: ${{ secrets.STAGING_GCAL_NOTIFICATION_TOKEN }}
+          GOOGLE_CLIENT_SECRET: ${{ secrets.STAGING_GOOGLE_CLIENT_SECRET }}
+          MONGO_URI: ${{ secrets.STAGING_MONGO_URI }}
           SSH_KEY: ${{ secrets.STAGING_SSH_KEY }}
-          SSH_HOST: ${{ secrets.STAGING_SSH_HOST }}
-          SSH_USER: ${{ secrets.STAGING_SSH_USER }}
+          SUPERTOKENS_KEY: ${{ secrets.STAGING_SUPERTOKENS_KEY }}
         run: |
           echo "Deploying Compass ${RELEASE_TAG} to staging"
           mkdir -p ~/.ssh
           echo "$SSH_KEY" > ~/.ssh/staging_key
           chmod 600 ~/.ssh/staging_key
           ssh-keyscan -H "$SSH_HOST" >> ~/.ssh/known_hosts
+          printf '%s\n' \
+            'runtime:' \
+            '  nodeEnv: production' \
+            '  timezone: Etc/UTC' \
+            'web:' \
+            "  url: \"${FRONTEND_URL}\"" \
+            'backend:' \
+            "  apiUrl: \"${BACKEND_API_URL}\"" \
+            '  originsAllowed:' \
+            "    - \"${FRONTEND_URL}\"" \
+            "  compassToken: \"${COMPASS_SYNC_TOKEN}\"" \
+            'mongo:' \
+            "  uri: \"${MONGO_URI}\"" \
+            'supertokens:' \
+            "  uri: \"${SUPERTOKENS_URI}\"" \
+            "  key: \"${SUPERTOKENS_KEY}\"" \
+            'google:' \
+            "  clientId: \"${GOOGLE_CLIENT_ID}\"" \
+            "  clientSecret: \"${GOOGLE_CLIENT_SECRET}\"" \
+            "  notificationToken: \"${GCAL_NOTIFICATION_TOKEN}\"" \
+            | ssh -i ~/.ssh/staging_key "$SSH_USER@$SSH_HOST" \
+                "umask 077 && mkdir -p ~/compass && cat > ~/compass/compass.yaml"
           ssh -i ~/.ssh/staging_key "$SSH_USER@$SSH_HOST" "cd ~/compass && ./compass update"

.github/workflows/publish-docker-images.yml

@@ -83,10 +83,10 @@ jobs:
           context: .
           file: self-host/Dockerfile.web
           push: true
-          # BASEURL and GOOGLE_CLIENT_ID are baked into the web bundle at build time.
+          # backend.apiUrl and google.clientId are baked into the web bundle at build time.
           # The published image ships with localhost defaults, which work for local installs.
           # Users who need a custom API domain or real Google credentials must rebuild
-          # the web image locally using the build: blocks in docker-compose.yml.
+          # the web image locally using the build: blocks in compose.yaml.
           build-args: |
             BASEURL=http://localhost:3000/api
             GOOGLE_CLIENT_ID=compass-self-host-placeholder.apps.googleusercontent.com

.github/workflows/test-unit.yml

@@ -74,6 +74,6 @@ jobs:
 
       - name: Run ${{ matrix.project }} tests
         env:
-          TZ: ${{ vars.TZ }}
+          TZ: Etc/UTC
         run: |
           bun run test:${{ matrix.project }}

.gitignore

@@ -14,6 +14,8 @@ credentials.txt
 *.mongodb
 *.tsbuildinfo
 *.mjs
+compass.yaml
+.playwright-compass.yaml
 
 ########
 # DIRS #
@@ -55,4 +57,4 @@ packages/backend/.prod.env
 
 
 tmp/
-!.env.local.example
+!compass.example.yaml

AGENTS.md

@@ -5,12 +5,14 @@
 - Frontend-only work usually starts with `bun run dev:web`; it does not require
   backend services.
 - Backend, auth, MongoDB, Google sync, and SSE work require
-  `packages/backend/.env.local`. Bootstrap with:
+  a `compass.yaml` at the repo root. Bootstrap with:
 
 ```bash
-cp packages/backend/.env.local.example packages/backend/.env.local
+cp compass.example.yaml compass.yaml
 ```
 
+- `compass.yaml` contains secrets. Do not commit it.
+
 - Avoid defaulting to `bun run test`; use the focused package test first.
 - Formatting is handled by the repo-local Codex Stop hook after each agent turn.
 - Use `bun run lint` and relevant verification before push or handoff.

bun.lock

@@ -0,0 +1,51 @@
+# Compass Config
+
+# Notes
+# - Copy this file to compass.yaml and replace values for your local setup.
+# - Do not commit this file; it contains secrets.
+# - For detailed explanations, see: https://docs.compasscalendar.com/docs/config
+
+compose:
+  version: latest
+
+runtime:
+  nodeEnv: development
+  timezone: Etc/UTC
+  logLevel: debug
+
+web:
+  port: 9080
+  url: http://localhost:9080
+
+backend:
+  port: 3000
+  apiUrl: http://localhost:3000/api
+  originsAllowed:
+    - http://localhost:3000
+    - http://localhost:9080
+  compassToken: REPLACE_WITH_COMPASS_TOKEN # any string will do
+
+mongo:
+  username: compass
+  password: local-mongo-password
+  replicaSetKey: local-mongo-replica-set-key
+  uri: mongodb+srv://admin:REPLACE_WITH_MONGO_PASSWORD@cluster0.m99yy.mongodb.net/dev_calendar?authSource=admin&retryWrites=true&w=majority&tls=true
+
+supertokens:
+  uri: REPLACE_WITH_SUPERTOKENS_URI
+  key: REPLACE_WITH_SUPERTOKENS_KEY
+
+# google:
+#   clientId: REPLACE_WITH_GOOGLE_CLIENT_ID # e.g. your-id.apps.googleusercontent.com
+#   clientSecret: REPLACE_WITH_GOOGLE_CLIENT_SECRET
+#   channelExpirationMin: 10
+#   webhookUrl: REPLACE_WITH_GOOGLE_WEBHOOK_URL # e.g. https://example.trycloudflare.com/api
+#   notificationToken: REPLACE_WITH_NOTIFICATION_TOKEN
+
+# email:
+#   kitApiSecret: REPLACE_WITH_KIT_API_SECRET
+#   kitUserTagId: REPLACE_WITH_KIT_USER_TAG_ID
+
+# posthog:
+#   key: REPLACE_WITH_POSTHOG_KEY
+#   host: REPLACE_WITH_POSTHOG_HOST

compass.example.yaml

@@ -16,8 +16,8 @@ Primary file:
 
 Environment loading:
 
-- `bun run cli` loads `packages/backend/.env.local`.
-- Keep local development variables in `packages/backend/.env.local` (bootstrap from `.env.local.example`).
+- `bun run cli` loads `compass.yaml` from the repo root.
+- Keep local development config in `compass.yaml` at the repo root (bootstrap from `compass.example.yaml`).
 
 ## CLI URL Resolution Contract
 
@@ -27,14 +27,14 @@ Primary source:
 
 How API base URLs are resolved:
 
-- local (`--environment local`): returns `BASEURL` directly (trailing slash removed)
-- staging/production: derives `https://<domain>/api` from `FRONTEND_URL`
+- local (`--environment local`): returns `backend.apiUrl` directly (trailing slash removed)
+- staging/production: derives `https://<domain>/api` from `web.url`
 
 Fallback behavior:
 
-- if `FRONTEND_URL` points at `localhost`, CLI prompts for a domain and builds `https://<domain>/api`
-- if `FRONTEND_URL` is already a non-localhost URL, CLI uses that hostname directly
-- local mode does not prompt for a domain; it depends on `BASEURL`
+- if `web.url` points at `localhost`, CLI prompts for a domain and builds `https://<domain>/api`
+- if `web.url` is already a non-localhost URL, CLI uses that hostname directly
+- local mode does not prompt for a domain; it depends on `backend.apiUrl`
 
 ## Commands To Know
 

docs/CI-CD/cli-and-maintenance-commands.md

@@ -73,7 +73,7 @@ publishing images.
 Source: [`.github/workflows/deploy-staging.yml`](../../.github/workflows/deploy-staging.yml)
 
 The deploy workflow SSHes into the staging VPS and runs `./compass update`,
-which pulls the Docker Hub image tag configured by the staging `.env` file and
+which pulls the Docker Hub image tag configured by the staging `compass.yaml` file and
 restarts the stack. The workflow accepts a release tag input so the Actions logs
 show which release triggered or motivated the deploy.
 

docs/CI-CD/workflows.md

@@ -0,0 +1,78 @@
+# Compass YAML Configuration
+
+Compass uses `compass.yaml` for self-hosting and local development. The file is visible, diffable, and contains secrets, so keep it out of git and back it up with the Docker volumes.
+
+Examples:
+
+- local development: `packages/backend/compass.example.yaml`
+- self-hosting: `self-host/compass.example.yaml`
+
+## Top-Level Sections
+
+| key | Default | Description |
+|---|---|---|
+| `compose.version` | `latest` | Docker image tag used by the self-host compose stack. Pin this for reproducible installs. |
+| `runtime.nodeEnv` | `production` | Runtime mode. Self-hosted installs should use `production`; local development uses `development`. |
+| `runtime.timezone` | `Etc/UTC` | Backend timezone. Only `Etc/UTC` and `UTC` are accepted. |
+| `runtime.logLevel` | `info` | Winston log level. |
+
+## Web
+
+| key | Required | Description |
+|---|---|---|
+| `web.port` | `9080` | Host port bound to the web container on `127.0.0.1`. |
+| `web.url` | Yes | Public frontend URL as seen by the backend. Example: `https://compass.example.com`. |
+
+## Backend
+
+| key | Required | Description |
+|---|---|---|
+| `backend.port` | `3000` | Host port bound to the backend container on `127.0.0.1`. |
+| `backend.apiUrl` | Yes | Public API URL. Example: `https://compass.example.com/api`. This is baked into the web bundle when the web image is rebuilt. |
+| `backend.originsAllowed` | Yes | YAML list of allowed CORS origins. Include `web.url`. |
+| `backend.compassToken` | Yes | Bearer token protecting internal sync endpoints. |
+
+## MongoDB
+
+| key | Required | Description |
+|---|---|---|
+| `mongo.username` | Self-host | MongoDB root username created on first container startup. Must match `mongo.uri`. |
+| `mongo.password` | Self-host | MongoDB root password. Changing it after first startup requires a MongoDB user migration. |
+| `mongo.replicaSetKey` | Self-host | MongoDB replica set key for the single-node `rs0` replica set. |
+| `mongo.uri` | Yes | Backend MongoDB connection string. Self-hosted installs must include `authSource=admin` and `replicaSet=rs0`. |
+
+## SuperTokens
+
+SuperTokens handles user-sessions for us.
+
+| key | Required | Description |
+|---|---|---|
+| `supertokens.uri` | Yes | SuperTokens Core URL as seen by the backend. Self-hosted Docker uses `http://supertokens:3567`. |
+| `supertokens.key` | Yes | API key shared by backend and SuperTokens Core. |
+| `supertokens.postgres.user` | Self-host | Postgres user for the SuperTokens database container. |
+| `supertokens.postgres.password` | Self-host | Postgres password for the SuperTokens database container. |
+| `supertokens.postgres.database` | Self-host | Postgres database name for SuperTokens. |
+
+## Google
+These values are only necessary if you want to enable Google Oauth and/or 2-way sync between Compass and Google Calendar
+
+Both `google.clientId` and `google.clientSecret` must be real values for Google features to activate. Setting only one causes backend startup to fail.
+
+| key | Required | Description |
+|---|---|---|
+| `google.clientId` | No | Google OAuth client ID. Rebuild the web image after changing it. |
+| `google.clientSecret` | No | Google OAuth client secret. Backend-only. |
+| `google.channelExpirationMin` | No | Google Calendar watch channel lifetime in minutes. |
+| `google.webhookUrl` | No | Public HTTPS API URL for Google Calendar push notifications. When omitted, Compass uses `backend.apiUrl`. |
+| `google.notificationToken` | Required for HTTPS Google webhooks | Token used to verify Google Calendar webhook requests. |
+
+See [Google Calendar](./google-calendar.md) for full setup instructions.
+
+## Optional Integrations
+
+| key | Required | Description |
+|---|---|---|
+| `email.kitApiSecret` | No | Kit.com API secret key. |
+| `email.kitUserTagId` | No | Kit.com tag ID applied to users on signup. |
+| `posthog.key` | No | PostHog project key injected into the web bundle. |
+| `posthog.host` | No | PostHog host injected into the web bundle. |