From 23a9179f2c7258f8a7d75a88ba442febc3c78642 Mon Sep 17 00:00:00 2001 From: Ashwin Date: Sat, 6 Jun 2026 17:22:06 +0530 Subject: [PATCH 1/2] fix: prevent DB and Redis connection exhaustion by implementing SSE stream timeouts, explicit connection cleanup, and documenting production deployment safety. --- PRODUCTION_DEPLOY.md | 232 ++++++++++++++++++ backend/common/tests/test_notification_sse.py | 29 +++ backend/common/views/notification_views.py | 100 ++++++-- 3 files changed, 342 insertions(+), 19 deletions(-) create mode 100644 PRODUCTION_DEPLOY.md diff --git a/PRODUCTION_DEPLOY.md b/PRODUCTION_DEPLOY.md new file mode 100644 index 000000000..dac973bf7 --- /dev/null +++ b/PRODUCTION_DEPLOY.md @@ -0,0 +1,232 @@ +# Production Deploy & Postgres Connection Safety + +This runbook covers running the backend under ASGI in production **without +exhausting the shared PostgreSQL cluster**. It exists because of the 2026-06-05 +incident where the app leaked connections until the whole shared cluster hit +`max_connections` (`TooManyConnectionsError: sorry, too many clients already`) +and took down unrelated tenants. + +There are three independent layers of protection. Apply **all three** — each one +catches what the others miss: + +1. **App-level** — bounded SSE streams + explicit connection close (already in + code; see `common/views/notification_views.py`). +2. **Server-level** — uvicorn worker recycling + graceful shutdown. +3. **Database-level** — a per-app role with a hard `CONNECTION LIMIT` (the + non-negotiable safety net — it's the only layer that *guarantees* one app + can never exhaust a shared cluster again). + +--- + +## 1. Database role with a hard connection cap (do this first) + +The app must connect as a **non-superuser** role (superusers bypass RLS — see +`RLS_SETUP.md`) that **also has a `CONNECTION LIMIT`**. The cap is what makes a +future leak fail *this app only*, instead of the whole cluster. + +```sql +-- Run as a Postgres admin/superuser, once per cluster. + +-- The application role. Non-superuser, NOBYPASSRLS, capped. +-- 3 uvicorn workers + a few Celery workers comfortably fit under 30. +ALTER ROLE bottlecrm WITH NOSUPERUSER NOBYPASSRLS CONNECTION LIMIT 30; + +-- If the role doesn't exist yet: +-- CREATE ROLE bottlecrm LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS CONNECTION LIMIT 30; + +-- The enterprise super-admin dashboard role (BYPASSRLS, separate alias in +-- crm_enterprise.settings DATABASES["superadmin"]). Cap it tightly — the +-- dashboard is low-traffic and this role's connections also count against +-- the cluster. It MUST keep BYPASSRLS but does NOT need to be superuser. +ALTER ROLE bottlecrm_superadmin WITH NOSUPERUSER BYPASSRLS CONNECTION LIMIT 5; +``` + +Sizing rule of thumb: `CONNECTION LIMIT` ≥ `(uvicorn workers × ~2) + +(celery workers × concurrency) + headroom`, and the **sum of all app caps on +the cluster** must stay below `max_connections` minus the +`superuser_reserved_connections` slots. With the app-level SSE fix in place, +steady-state usage is a small multiple of the worker count, not per-open-tab. + +Verify the role is safe: + +```bash +cd backend +uv run python manage.py manage_rls --verify-user # asserts non-superuser +``` + +Watch live connection counts (run as admin): + +```sql +SELECT usename, count(*) , max(backend_start) +FROM pg_stat_activity +GROUP BY usename +ORDER BY count(*) DESC; +``` + +--- + +## 2. uvicorn: worker recycling + graceful shutdown + +Run the ASGI app (`crm.asgi:application`) with worker recycling so a slow leak +can never accumulate indefinitely, and a bounded graceful-shutdown so deploys +don't leave **orphaned workers** holding connections (the other half of the +incident — those required `kill -9`). + +```bash +cd backend +uv run uvicorn crm.asgi:application \ + --host 0.0.0.0 --port 8000 \ + --workers 3 \ + --limit-max-requests 2000 \ # recycle each worker after N requests — caps any per-request leak + --timeout-graceful-shutdown 30 \ # don't wait forever for SSE streams to drain on restart + --timeout-keep-alive 10 \ + --proxy-headers --forwarded-allow-ips='*' # behind nginx/ELB +``` + +Notes: +- `--limit-max-requests` makes uvicorn replace a worker after it has served N + requests, closing all its DB connections on the way out. Pick a value that + recycles roughly every 1–2 hours of traffic. +- `--timeout-graceful-shutdown 30` bounds how long a worker waits for in-flight + requests (including SSE streams) before being force-killed. Combined with the + app-level `MAX_STREAM_SECONDS = 300`, streams self-close well within this, so + workers drain cleanly on deploy instead of orphaning. +- Keep `--workers` modest and let the DB `CONNECTION LIMIT` be the real ceiling. +- `gunicorn -k uvicorn.workers.UvicornWorker` is a fine alternative; use + `--max-requests 2000 --max-requests-jitter 200 --graceful-timeout 30` there. + +--- + +## 3. systemd unit + +`/etc/systemd/system/bottlecrm-api.service`: + +```ini +[Unit] +Description=BottleCRM API (uvicorn/ASGI) +After=network.target + +[Service] +Type=exec +User=bottlecrm +Group=bottlecrm +WorkingDirectory=/srv/bottlecrm/backend +EnvironmentFile=/srv/bottlecrm/backend/.env +# uv resolves the project venv automatically. +ExecStart=/usr/local/bin/uv run uvicorn crm.asgi:application \ + --host 127.0.0.1 --port 8000 \ + --workers 3 \ + --limit-max-requests 2000 \ + --timeout-graceful-shutdown 30 \ + --timeout-keep-alive 10 \ + --proxy-headers --forwarded-allow-ips='*' + +# Graceful stop: SIGTERM lets workers finish draining; systemd force-kills +# after TimeoutStopSec, so a hung worker can't linger as an orphan. +KillSignal=SIGTERM +TimeoutStopSec=45 +Restart=always +RestartSec=5 + +# Hardening (optional but recommended) +NoNewPrivileges=true +ProtectSystem=full +PrivateTmp=true + +[Install] +WantedBy=multi-user.target +``` + +Celery worker unit `/etc/systemd/system/bottlecrm-celery.service` (Celery keeps +its own persistent connections — bound concurrency so it stays under the role +cap): + +```ini +[Unit] +Description=BottleCRM Celery worker +After=network.target redis.service + +[Service] +Type=exec +User=bottlecrm +WorkingDirectory=/srv/bottlecrm/backend +EnvironmentFile=/srv/bottlecrm/backend/.env +ExecStart=/usr/local/bin/uv run celery -A crm worker \ + --loglevel=INFO --concurrency=4 --max-tasks-per-child=500 +KillSignal=SIGTERM +TimeoutStopSec=60 +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +`--max-tasks-per-child=500` recycles Celery child processes periodically, the +same idea as `--limit-max-requests` for the web tier. + +Apply: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now bottlecrm-api bottlecrm-celery +sudo systemctl restart bottlecrm-api # zero-orphan rolling restart +``` + +--- + +## 4. (Optional) pgbouncer — pooling in front of the cluster + +On a shared cluster, putting **pgbouncer** between the app and Postgres caps +server-side connections regardless of how many clients the app opens. Use +**transaction** pooling (works with Django's autocommit; note RLS uses +`set_config(..., is_local=false)` at session scope, so verify the RLS context is +re-applied per checkout — the middleware sets and resets it every request, which +is compatible, but test tenant isolation under pgbouncer before rolling out). + +`pgbouncer.ini`: + +```ini +[databases] +crm_db = host=127.0.0.1 port=5432 dbname=crm_db + +[pgbouncer] +listen_addr = 127.0.0.1 +listen_port = 6432 +auth_type = scram-sha-256 +auth_file = /etc/pgbouncer/userlist.txt +pool_mode = transaction +max_client_conn = 200 +default_pool_size = 20 ; server-side conns per (user,db) — the real ceiling +reserve_pool_size = 5 +server_idle_timeout = 60 +``` + +Point the app at pgbouncer by setting `DBPORT=6432` (and `DBHOST=127.0.0.1`). +Keep the database-level `CONNECTION LIMIT` anyway — defense in depth. + +--- + +## Post-deploy verification + +```bash +# 1. App connects as a capped non-superuser +cd backend && uv run python manage.py manage_rls --verify-user + +# 2. Connection count stays bounded under load (run as DB admin, watch over time) +watch -n 30 "psql -c \"SELECT count(*) FROM pg_stat_activity WHERE usename='bottlecrm'\"" + +# 3. SSE streams self-terminate: open the app, leave a tab on a page with the +# notifications stream, confirm the connection count does NOT climb with +# open-tab time (it should plateau, recycling ~every MAX_STREAM_SECONDS=300s). + +# 4. Rolling restart leaves no orphans +sudo systemctl restart bottlecrm-api +ps -ef | grep '[u]vicorn' # only the new master + 3 workers, no stragglers +``` + +If connections ever climb again: check `pg_stat_activity` for the `query` / +`state` / `backend_start` of the `bottlecrm` rows — long-idle (`idle` with old +`backend_start`) connections point back to a streaming/long-lived view holding a +connection; see `common/views/notification_views.py` for the pattern to follow +(close the request connection before a long-lived response, bound its lifetime). diff --git a/backend/common/tests/test_notification_sse.py b/backend/common/tests/test_notification_sse.py index 0b91ce116..95df839c0 100644 --- a/backend/common/tests/test_notification_sse.py +++ b/backend/common/tests/test_notification_sse.py @@ -14,6 +14,7 @@ from rest_framework.test import APIClient from common.models import Notification, Org, Profile, User +from common.views import notification_views from common.views.notification_views import ( _format_keepalive, _format_sse, @@ -119,6 +120,34 @@ async def runner(): assert payload["link"] == "/cases/abc" assert payload["data"] == {"comment_excerpt": "hi"} + def test_stream_self_terminates_at_deadline(self): + """The stream must end on its own once MAX_STREAM_SECONDS elapses, so a + long-open browser tab cannot pin a worker thread / DB connection + forever (the connection-exhaustion regression).""" + async def runner(): + pubsub = _StubPubSub() # no messages -> keepalive-only loop + gen = _stream_events( + "notif:o:p", recipient_id=self.profile.id, pubsub=pubsub + ) + first = await asyncio.wait_for(gen.__anext__(), timeout=1) + stopped = False + try: + # Deadline already passed -> next pull should end the stream. + await asyncio.wait_for(gen.__anext__(), timeout=2) + except StopAsyncIteration: + stopped = True + await gen.aclose() + return first, stopped + + original = notification_views.MAX_STREAM_SECONDS + notification_views.MAX_STREAM_SECONDS = 0 + try: + first, stopped = asyncio.run(runner()) + finally: + notification_views.MAX_STREAM_SECONDS = original + assert first == b": keepalive\n\n" + assert stopped, "stream did not terminate at its deadline" + def test_drops_message_for_other_recipient(self): n_other = Notification.objects.create( org=self.org, recipient=self.other_profile, verb="other" diff --git a/backend/common/views/notification_views.py b/backend/common/views/notification_views.py index c0cce8e79..21a91bc21 100644 --- a/backend/common/views/notification_views.py +++ b/backend/common/views/notification_views.py @@ -16,6 +16,7 @@ import asyncio import json import logging +import time from asgiref.sync import sync_to_async from django.http import StreamingHttpResponse @@ -36,6 +37,17 @@ KEEPALIVE_SECONDS = 15 +# Upper bound on a single SSE connection's lifetime. The browser's EventSource +# transparently reconnects when the server closes the stream, so capping this +# is invisible to the user but critical for the server: under ASGI the *entire* +# stream runs inside one request (one `ThreadSensitiveContext`), so an +# unbounded `while True` stream pins a worker thread — and any DB/redis +# connection bound to it — for as long as the tab stays open (hours). Left +# unbounded this is what slowly exhausts a shared Postgres cluster and leaves +# workers undrainable on deploy. 5 minutes bounds the leak window and lets +# workers recycle. +MAX_STREAM_SECONDS = 300 + DEFAULT_LIMIT = 20 MAX_LIMIT = 100 @@ -141,38 +153,67 @@ async def _aget_serialized(notif_id, recipient_id): """Fetch a notification by id, scoped to recipient. Returns dict or None.""" @sync_to_async def fetch(): - notif = Notification.objects.filter( - pk=notif_id, recipient_id=recipient_id - ).first() - if notif is None: - return None - return NotificationSerializer(notif).data + from django.db import connection + + try: + notif = Notification.objects.filter( + pk=notif_id, recipient_id=recipient_id + ).first() + if notif is None: + return None + return NotificationSerializer(notif).data + finally: + # This runs in asgiref's shared thread-sensitive executor, outside + # Django's request/response cycle — so `close_old_connections` + # never fires for it. Without this explicit close the connection + # dangles open (the SSE tests need TransactionTestCase precisely + # because of it). Close it so each fetch reclaims its connection. + connection.close() return await fetch() +async def _aclose_redis(obj): + """Close a redis.asyncio client/pubsub across versions (aclose vs close).""" + if obj is None: + return + closer = getattr(obj, "aclose", None) or getattr(obj, "close", None) + if closer is None: + return + try: + result = closer() + if asyncio.iscoroutine(result): + await result + except Exception: # pragma: no cover - best effort + pass + + async def _open_pubsub(channel: str): """Open a redis.asyncio pubsub subscribed to ``channel``. - Returns ``None`` if Redis is unreachable; the stream then runs in - keepalive-only mode (the frontend's polling-since path provides - backfill). + Returns ``(client, pubsub)``. Both are ``None`` if Redis is unreachable; + the stream then runs in keepalive-only mode (the frontend's polling-since + path provides backfill). The caller owns closing BOTH the pubsub and the + underlying client connection pool — closing only the pubsub leaks the + client's pooled connections, one per dropped SSE stream. """ try: import redis.asyncio as aioredis # type: ignore except ImportError: # pragma: no cover - return None + return None, None from django.conf import settings url = getattr(settings, "CELERY_BROKER_URL", None) or "redis://localhost:6379/0" + client = None try: client = aioredis.from_url(url) pubsub = client.pubsub() await pubsub.subscribe(channel) - return pubsub + return client, pubsub except Exception as exc: logger.warning("SSE redis subscribe failed (%s); keepalive-only mode", exc) - return None + await _aclose_redis(client) + return None, None async def _stream_events(channel: str, recipient_id, *, pubsub=None): @@ -182,12 +223,18 @@ async def _stream_events(channel: str, recipient_id, *, pubsub=None): opens its own redis.asyncio pubsub via :func:`_open_pubsub`. """ owns_pubsub = pubsub is None + client = None if owns_pubsub: - pubsub = await _open_pubsub(channel) + client, pubsub = await _open_pubsub(channel) + deadline = time.monotonic() + MAX_STREAM_SECONDS try: # Initial comment so the client confirms the stream opened. yield _format_keepalive() while True: + # Bounded lifetime: end the stream so the worker thread and its + # DB/redis connections are released. EventSource reconnects. + if time.monotonic() >= deadline: + return if pubsub is None: await asyncio.sleep(KEEPALIVE_SECONDS) yield _format_keepalive() @@ -211,12 +258,15 @@ async def _stream_events(channel: str, recipient_id, *, pubsub=None): continue yield _format_sse("notification", payload) finally: - if owns_pubsub and pubsub is not None: - try: - await pubsub.unsubscribe(channel) - await pubsub.close() - except Exception: # pragma: no cover - best effort - pass + if owns_pubsub: + if pubsub is not None: + try: + await pubsub.unsubscribe(channel) + except Exception: # pragma: no cover - best effort + pass + await _aclose_redis(pubsub) + # Close the underlying client pool too — not just the pubsub. + await _aclose_redis(client) def _drive_async_gen(agen): @@ -271,6 +321,18 @@ def get(self, request, *args, **kwargs): profile_id = request.profile.id channel = notif_mod.channel_for(org_id, profile_id) + # Release THIS request thread's DB connection before streaming begins. + # Under ASGI the entire stream runs inside one request's + # ThreadSensitiveContext, so the connection opened by auth/middleware + # would otherwise stay checked out for the full (possibly hours-long) + # stream — one leaked Postgres connection per open browser tab, which + # is what exhausts the shared cluster. The stream's own reads go + # through a separate executor (see `_aget_serialized`) and close + # themselves, so dropping this one is safe. + from django.db import connection + + connection.close() + response = StreamingHttpResponse( _drive_async_gen(_stream_events(channel, profile_id)), content_type="text/event-stream", From 2cca92aba57cf31271fdd6fcc55f5826acf522fc Mon Sep 17 00:00:00 2001 From: Ashwin Date: Sat, 6 Jun 2026 17:24:32 +0530 Subject: [PATCH 2/2] chore: remove production deployment runbook --- PRODUCTION_DEPLOY.md | 232 ------------------------------------------- 1 file changed, 232 deletions(-) delete mode 100644 PRODUCTION_DEPLOY.md diff --git a/PRODUCTION_DEPLOY.md b/PRODUCTION_DEPLOY.md deleted file mode 100644 index dac973bf7..000000000 --- a/PRODUCTION_DEPLOY.md +++ /dev/null @@ -1,232 +0,0 @@ -# Production Deploy & Postgres Connection Safety - -This runbook covers running the backend under ASGI in production **without -exhausting the shared PostgreSQL cluster**. It exists because of the 2026-06-05 -incident where the app leaked connections until the whole shared cluster hit -`max_connections` (`TooManyConnectionsError: sorry, too many clients already`) -and took down unrelated tenants. - -There are three independent layers of protection. Apply **all three** — each one -catches what the others miss: - -1. **App-level** — bounded SSE streams + explicit connection close (already in - code; see `common/views/notification_views.py`). -2. **Server-level** — uvicorn worker recycling + graceful shutdown. -3. **Database-level** — a per-app role with a hard `CONNECTION LIMIT` (the - non-negotiable safety net — it's the only layer that *guarantees* one app - can never exhaust a shared cluster again). - ---- - -## 1. Database role with a hard connection cap (do this first) - -The app must connect as a **non-superuser** role (superusers bypass RLS — see -`RLS_SETUP.md`) that **also has a `CONNECTION LIMIT`**. The cap is what makes a -future leak fail *this app only*, instead of the whole cluster. - -```sql --- Run as a Postgres admin/superuser, once per cluster. - --- The application role. Non-superuser, NOBYPASSRLS, capped. --- 3 uvicorn workers + a few Celery workers comfortably fit under 30. -ALTER ROLE bottlecrm WITH NOSUPERUSER NOBYPASSRLS CONNECTION LIMIT 30; - --- If the role doesn't exist yet: --- CREATE ROLE bottlecrm LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS CONNECTION LIMIT 30; - --- The enterprise super-admin dashboard role (BYPASSRLS, separate alias in --- crm_enterprise.settings DATABASES["superadmin"]). Cap it tightly — the --- dashboard is low-traffic and this role's connections also count against --- the cluster. It MUST keep BYPASSRLS but does NOT need to be superuser. -ALTER ROLE bottlecrm_superadmin WITH NOSUPERUSER BYPASSRLS CONNECTION LIMIT 5; -``` - -Sizing rule of thumb: `CONNECTION LIMIT` ≥ `(uvicorn workers × ~2) + -(celery workers × concurrency) + headroom`, and the **sum of all app caps on -the cluster** must stay below `max_connections` minus the -`superuser_reserved_connections` slots. With the app-level SSE fix in place, -steady-state usage is a small multiple of the worker count, not per-open-tab. - -Verify the role is safe: - -```bash -cd backend -uv run python manage.py manage_rls --verify-user # asserts non-superuser -``` - -Watch live connection counts (run as admin): - -```sql -SELECT usename, count(*) , max(backend_start) -FROM pg_stat_activity -GROUP BY usename -ORDER BY count(*) DESC; -``` - ---- - -## 2. uvicorn: worker recycling + graceful shutdown - -Run the ASGI app (`crm.asgi:application`) with worker recycling so a slow leak -can never accumulate indefinitely, and a bounded graceful-shutdown so deploys -don't leave **orphaned workers** holding connections (the other half of the -incident — those required `kill -9`). - -```bash -cd backend -uv run uvicorn crm.asgi:application \ - --host 0.0.0.0 --port 8000 \ - --workers 3 \ - --limit-max-requests 2000 \ # recycle each worker after N requests — caps any per-request leak - --timeout-graceful-shutdown 30 \ # don't wait forever for SSE streams to drain on restart - --timeout-keep-alive 10 \ - --proxy-headers --forwarded-allow-ips='*' # behind nginx/ELB -``` - -Notes: -- `--limit-max-requests` makes uvicorn replace a worker after it has served N - requests, closing all its DB connections on the way out. Pick a value that - recycles roughly every 1–2 hours of traffic. -- `--timeout-graceful-shutdown 30` bounds how long a worker waits for in-flight - requests (including SSE streams) before being force-killed. Combined with the - app-level `MAX_STREAM_SECONDS = 300`, streams self-close well within this, so - workers drain cleanly on deploy instead of orphaning. -- Keep `--workers` modest and let the DB `CONNECTION LIMIT` be the real ceiling. -- `gunicorn -k uvicorn.workers.UvicornWorker` is a fine alternative; use - `--max-requests 2000 --max-requests-jitter 200 --graceful-timeout 30` there. - ---- - -## 3. systemd unit - -`/etc/systemd/system/bottlecrm-api.service`: - -```ini -[Unit] -Description=BottleCRM API (uvicorn/ASGI) -After=network.target - -[Service] -Type=exec -User=bottlecrm -Group=bottlecrm -WorkingDirectory=/srv/bottlecrm/backend -EnvironmentFile=/srv/bottlecrm/backend/.env -# uv resolves the project venv automatically. -ExecStart=/usr/local/bin/uv run uvicorn crm.asgi:application \ - --host 127.0.0.1 --port 8000 \ - --workers 3 \ - --limit-max-requests 2000 \ - --timeout-graceful-shutdown 30 \ - --timeout-keep-alive 10 \ - --proxy-headers --forwarded-allow-ips='*' - -# Graceful stop: SIGTERM lets workers finish draining; systemd force-kills -# after TimeoutStopSec, so a hung worker can't linger as an orphan. -KillSignal=SIGTERM -TimeoutStopSec=45 -Restart=always -RestartSec=5 - -# Hardening (optional but recommended) -NoNewPrivileges=true -ProtectSystem=full -PrivateTmp=true - -[Install] -WantedBy=multi-user.target -``` - -Celery worker unit `/etc/systemd/system/bottlecrm-celery.service` (Celery keeps -its own persistent connections — bound concurrency so it stays under the role -cap): - -```ini -[Unit] -Description=BottleCRM Celery worker -After=network.target redis.service - -[Service] -Type=exec -User=bottlecrm -WorkingDirectory=/srv/bottlecrm/backend -EnvironmentFile=/srv/bottlecrm/backend/.env -ExecStart=/usr/local/bin/uv run celery -A crm worker \ - --loglevel=INFO --concurrency=4 --max-tasks-per-child=500 -KillSignal=SIGTERM -TimeoutStopSec=60 -Restart=always -RestartSec=5 - -[Install] -WantedBy=multi-user.target -``` - -`--max-tasks-per-child=500` recycles Celery child processes periodically, the -same idea as `--limit-max-requests` for the web tier. - -Apply: - -```bash -sudo systemctl daemon-reload -sudo systemctl enable --now bottlecrm-api bottlecrm-celery -sudo systemctl restart bottlecrm-api # zero-orphan rolling restart -``` - ---- - -## 4. (Optional) pgbouncer — pooling in front of the cluster - -On a shared cluster, putting **pgbouncer** between the app and Postgres caps -server-side connections regardless of how many clients the app opens. Use -**transaction** pooling (works with Django's autocommit; note RLS uses -`set_config(..., is_local=false)` at session scope, so verify the RLS context is -re-applied per checkout — the middleware sets and resets it every request, which -is compatible, but test tenant isolation under pgbouncer before rolling out). - -`pgbouncer.ini`: - -```ini -[databases] -crm_db = host=127.0.0.1 port=5432 dbname=crm_db - -[pgbouncer] -listen_addr = 127.0.0.1 -listen_port = 6432 -auth_type = scram-sha-256 -auth_file = /etc/pgbouncer/userlist.txt -pool_mode = transaction -max_client_conn = 200 -default_pool_size = 20 ; server-side conns per (user,db) — the real ceiling -reserve_pool_size = 5 -server_idle_timeout = 60 -``` - -Point the app at pgbouncer by setting `DBPORT=6432` (and `DBHOST=127.0.0.1`). -Keep the database-level `CONNECTION LIMIT` anyway — defense in depth. - ---- - -## Post-deploy verification - -```bash -# 1. App connects as a capped non-superuser -cd backend && uv run python manage.py manage_rls --verify-user - -# 2. Connection count stays bounded under load (run as DB admin, watch over time) -watch -n 30 "psql -c \"SELECT count(*) FROM pg_stat_activity WHERE usename='bottlecrm'\"" - -# 3. SSE streams self-terminate: open the app, leave a tab on a page with the -# notifications stream, confirm the connection count does NOT climb with -# open-tab time (it should plateau, recycling ~every MAX_STREAM_SECONDS=300s). - -# 4. Rolling restart leaves no orphans -sudo systemctl restart bottlecrm-api -ps -ef | grep '[u]vicorn' # only the new master + 3 workers, no stragglers -``` - -If connections ever climb again: check `pg_stat_activity` for the `query` / -`state` / `backend_start` of the `bottlecrm` rows — long-idle (`idle` with old -`backend_start`) connections point back to a streaming/long-lived view holding a -connection; see `common/views/notification_views.py` for the pattern to follow -(close the request connection before a long-lived response, bound its lifetime).