Merged branch feature/postgres into develop

Author: mike-bestcat

AGENTS.md

@@ -62,8 +62,10 @@ scaffold with
 mise run add-migration "The name of the migration"
 ```
 
-This will generate a new file in src/core/migrations/versions, link it in the sequence of migrations,
-and you can then edit it there, based on the patterns established.
+This will generate a new file in src/core/migrations/sqlite/versions
+and in sqr/core/migrations/postgres/versions, 
+link it in the sequence of migrations, and you can then edit it
+there, based on the patterns established.
 
 ### Lint, fix & check commands
 

docs/adrs/0008.webapi-backend-blend-via-config-global.md

@@ -0,0 +1,63 @@
+# ADR 0008: WebAPI backend blend controlled via `Config.global`
+
+## Status
+
+Accepted (2026-05-03)
+
+## Context
+
+The WebAPI process wires several replaceable infrastructure pieces: where
+domain data lives, how telemetry is exported, which implementation backs search
+queries, and whether CRM side effects run against a real provider. Operators
+need a single, explicit place to choose that **blend** per deployment, without
+scattering ad hoc rules across `env`, hosting flags, and implicit defaults.
+
+## Decision
+
+1. **Control surface**  
+   Four environment variables live in `src/Config.global` (loaded with other
+   universe-wide settings when building global configuration). Each variable
+   selects one implementation for that concern. Together they define the
+   **backend blend** for the WebAPI (and any other backend components that share
+   the same top-level wiring and configuration load in that process).
+
+2. **Global scope**  
+   The chosen values apply to **all** backend composition for that run: one
+   blend per process, not per workspace, user, or request.
+
+3. **Variables and allowed values**
+
+   | Variable | Allowed values | Meaning |
+   <!-- markdownlint-disable MD013 -->
+   | -------- | -------------- | ------- |
+   | `WEBAPI_STORAGE_ENGINE` | `sqlite`, `postgres` | Primary domain persistence engine |
+   | `WEBAPI_TELEMETRY` | `sentry`, `local` | Telemetry sink (e.g. Sentry vs local/no-op style) |
+   | `WEBAPI_SEARCH` | `sql`, `algolia` | Search query / document storage engine for search |
+   | `WEBAPI_CRM` | `wix`, `noop` | CRM integration vs no-op |
+   <!-- markdownlint-enable MD013 -->
+
+   Values are lowercase string tokens. Invalid or missing values should be
+   rejected at startup with a clear error once wiring reads these variables.
+
+4. **Credentials and URLs**  
+   Provider-specific secrets and connection strings remain in service-level
+   config (for example `src/webapi/Config.project` and related env). The
+   `WEBAPI_*` flags only choose **which** implementation to construct; they do
+   not replace DSNs, API keys, or database URLs.
+
+## Conventions by universe
+
+- **Thrive** (`UNIVERSE=thrive`): use the hosted-style blend—`postgres`,
+  `sentry`, `algolia`, and `wix` for the four `WEBAPI_*` variables respectively.
+- **Other universes** (not thrive): use the local-first blend—`sqlite`, `local`,
+  `sql`, and `noop`.
+- **Dev** (`UNIVERSE=dev`): no fixed convention; teams may set any valid
+  combination while developing or testing.
+
+## Consequences
+
+- Operators set the four `WEBAPI_*` entries in `Config.global` (or equivalent
+  injected env) for each deployment’s intended blend.
+- Composition code in the WebAPI entrypoint should branch on these settings (or
+  on typed values derived from them) so behavior stays aligned with the declared
+  configuration instead of re-deriving the same choices from unrelated signals.

infra/self-hosted/compose.yaml

@@ -1,20 +1,54 @@
 name: jupiter
 
 services:
+  webapi-postgres:
+    profiles:
+      - storage-engine-postgres
+    image: postgres:${POSTGRES_VERSION:-18}
+    restart: always
+    environment:
+      POSTGRES_USER: ${JUPITER_DEV_POSTGRES_USER:-jupiter}
+      POSTGRES_PASSWORD: ${JUPITER_DEV_POSTGRES_PASSWORD:-secret}
+      POSTGRES_DB: ${JUPITER_DEV_POSTGRES_DB:-jupiter}
+    volumes:
+      # Postgres 18+ images expect one mount at /var/lib/postgresql (not .../data); see docker-library/postgres#1259.
+      - postgres_data:/var/lib/postgresql
+    healthcheck:
+      # Must match POSTGRES_USER / POSTGRES_DB (from JUPITER_DEV_POSTGRES_*), not hardcoded jupiter.
+      test: ["CMD-SHELL", "pg_isready -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
+      interval: 5s
+      timeout: 5s
+      retries: 10
+    ports:
+      # Host port for psql / GUI tools (Compose v2.20+): see WEBAPI_POSTGRES_PORT from srv.sh / mise run run:srv.
+      # Use a non-zero value for a stable mapping; :0 assigns an ephemeral host port.
+      - "${WEBAPI_POSTGRES_PORT:-0}:5432"
   webapi:
     environment:
-      - SQLITE_DB_URL=sqlite+aiosqlite:////data/jupiter.sqlite
+      - SQLITE_DB_URL=${SQLITE_DB_URL:-sqlite+aiosqlite:////data/jupiter.sqlite}
+      - POSTGRES_DB_URL=${POSTGRES_DB_URL:-}
+      - ALEMBIC_INI_PATH=${ALEMBIC_INI_PATH:-../core/migrations/alembic.sqlite.ini}
+      - ALEMBIC_MIGRATIONS_PATH=${ALEMBIC_MIGRATIONS_PATH:-../core/migrations/sqlite}
       - UNIVERSE=${UNIVERSE:?Check https://docs.get-thriving.com/how-tos/self-hosting}
       - ENV=${ENV:-production}
       - INSTANCE=${INSTANCE:?Check https://docs.get-thriving.com/how-tos/self-hosting}
       - PORT=2000
       - AUTH_TOKEN_SECRET=${AUTH_TOKEN_SECRET:?Check https://docs.get-thriving.com/how-tos/self-hosting}
+      - WEBAPI_STORAGE_ENGINE=${WEBAPI_STORAGE_ENGINE:-sqlite}
+      - WEBAPI_TELEMETRY=${WEBAPI_TELEMETRY:-local}
+      - WEBAPI_SEARCH=${WEBAPI_SEARCH:-sql}
+      - WEBAPI_CRM=${WEBAPI_CRM:-noop}
+      - POSTGRES_VERSION=${POSTGRES_VERSION:-18}
+    depends_on:
+      webapi-postgres:
+        condition: service_healthy
+        required: false
     restart: always
     image: ${DOCKER_IMAGE_WEBAPI:-getthriving/webapi:${VERSION:-latest}}
     ports:
     - "${WEBAPI_PORT:-0}:2000"
     volumes:
-      - db:/data
+      - sqlite_data:/data
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:2000/healthz"]
       interval: 30s
@@ -95,6 +129,8 @@ services:
       - PUBLIC_NAME=${PUBLIC_NAME}
     restart: always
     image: ${DOCKER_IMAGE_DOCS:-getthriving/docs:${VERSION:-latest}}
+    ports:
+      - "${DOCS_PORT:-0}:8000"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:8000/healthz"]
       interval: 30s
@@ -125,6 +161,8 @@ services:
       timeout: 10s
       retries: 5
 volumes:
-  db:
-    name: db-${INSTANCE:-dev}
+  sqlite_data:
+    name: sqlite_data-${INSTANCE:-dev}
+  postgres_data:
+    name: postgres_data-${INSTANCE:-dev}
 

itests/api/big_plans.test.py

@@ -451,6 +451,10 @@ def test_api_big_plan_create_inbox_task_visible_in_inbox(
         },
         timeout=10,
     )
+
+    print(create_response.status_code)
+    print(create_response.content)
+
     assert create_response.status_code == 200
     created_ref_id = create_response.json()["new_inbox_task"]["ref_id"]
 

itests/api/search.test.py

@@ -108,7 +108,8 @@ def _search(
 
 
 def _ref_ids_from_search(data: Any) -> list[str]:  # type: ignore
-    return [m["summary"]["ref_id"] for m in data.get("matches", [])]
+    # JSON numbers decode as int; API clients use str ref_ids — normalize for ``in`` checks.
+    return [str(m["summary"]["ref_id"]) for m in data.get("matches", [])]
 
 
 def _todo_ref_ids_from_search(data: Any) -> list[str]:  # type: ignore

itests/output.json

@@ -0,0 +1 @@
+{"content": {"suites": {"0": {"status": {"total_pass": 0, "total_skip": 0, "total_xpass": 0, "total_xfail": 0, "total_rerun": 0, "total_fail": 0, "total_error": 3}, "tests": {"0": {"status": "ERROR", "message": "conftest.py:85: in api_url\n    raise Exception(\"API_URL is not set\")\nE   Exception: API_URL is not set\n", "test_name": "test_webui_time_plan_link_untracked_inbox_tasks[chromium]", "rerun": "0"}, "1": {"status": "ERROR", "message": "conftest.py:85: in api_url\n    raise Exception(\"API_URL is not set\")\nE   Exception: API_URL is not set\n", "test_name": "test_webui_time_plan_link_untracked_big_plans[chromium]", "rerun": "0"}, "2": {"status": "ERROR", "message": "conftest.py:85: in api_url\n    raise Exception(\"API_URL is not set\")\nE   Exception: API_URL is not set\n", "test_name": "test_webui_time_plan_show_activity_doneness[chromium]", "rerun": "0"}}, "suite_name": "itests/webui/entities/time_plans.test.py"}}}, "date": "May 09, 2026", "start_time": 1778353928.159845, "total_suite": 1, "status": "FAIL", "status_list": {"pass": "0", "fail": "0", "skip": "0", "error": "3", "xpass": "0", "xfail": "0", "rerun": "0"}, "total_tests": "3"}