Skip to content

Commit 74aab94

Browse files
weroperkingweroperking
authored
Merge pull request #67 from weroperking/docs/self-hosted-production-readiness-roadmap
docs: add self-hosted production readiness roadmap
2 parents 8406f5c + d0ab0e2 commit 74aab94

1 file changed

Lines changed: 332 additions & 0 deletions

File tree

‎docs/guides/production-readiness-roadmap.md‎

Lines changed: 332 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,332 @@
1+
# Self-Hosted Production Readiness Roadmap
2+
3+
A pragmatic technical audit scoped specifically for **self-hosted production** — teams running BetterBase on their own infrastructure (Docker Compose, VPS, or Kubernetes). This document strips away SaaS-scale distractions and focuses only on what is required to run a stable, secure, single-tenant instance that teams can confidently deploy and maintain.
4+
5+
## Executive Summary
6+
7+
BetterBase's self-hosted stack (Postgres, MinIO, Inngest, Nginx, server + dashboard) is already functional and deployable. The gaps between "it runs" and "a team can rely on it in production" are smaller than a generic SaaS audit would suggest.
8+
9+
| Pillar | Status | Real Blockers for Self-Hosting |
10+
|--------|--------|-------------------------------|
11+
| Security | Beta | Rate limiting, scope enforcement, HTML escaping |
12+
| Stability | Beta | Transactional migrations, deep health checks, graceful shutdown |
13+
| Operations | Alpha | Backups, Prometheus metrics, K8s manifests |
14+
| Multi-Instance Scaling | Alpha | Redis bridge for WS + rate limits across replicas |
15+
| Developer Experience | Good | React-only client is limiting but not production-blocking |
16+
17+
**Overall readiness: 80%.** The remaining 20% is operational hardening and multi-replica support. Several items from a generic enterprise audit are **not required** for self-hosting and have been intentionally excluded below.
18+
19+
---
20+
21+
## What We Are NOT Doing (Out of Scope)
22+
23+
These are valid features for a managed SaaS or unicorn-scale platform, but they are unnecessary overhead for teams self-hosting their own backend.
24+
25+
| Item | Why It's Excluded |
26+
|------|-------------------|
27+
| Read replica routing | Self-hosted teams scale vertically or use managed Postgres (Neon/RDS) which handles this |
28+
| Multi-language SDKs (Python, Go, etc.) | TypeScript client is sufficient for v1; OpenAPI export can come later |
29+
| React-free client SDK refactor | Important for DX, but not a production stability issue |
30+
| API versioning (`/v1/`) | Self-hosted teams control their own upgrade cadence |
31+
| OpenAPI/Swagger generation | Nice to have, but docs and examples are sufficient for self-hosted teams |
32+
| Chaos engineering / failure injection | Overkill for typical self-hosted deployments |
33+
| Circuit breakers | Single-tenant instances with local MinIO/Postgres do not need SaaS-style circuit breaking |
34+
| Terraform / Pulumi IaC | Teams bring their own infrastructure; Docker Compose and K8s are the interface |
35+
| One-click deploy scripts for Railway/Render/Fly | These platforms already support Docker; maintain official Compose instead |
36+
| MFA / TOTP for admin accounts | A single team's admin login can be protected by SSO proxy or VPN in practice |
37+
| IP allowlisting | Same as above — handled at the network/VPN layer for most self-hosted setups |
38+
| Application-level column encryption | Postgres at-rest encryption is sufficient for single-tenant self-hosting |
39+
| RS256 JWT / asymmetric signing | HS256 is fine when the secret is rotated and stored in container env/secrets |
40+
| Replacing Neon CDC polling | Self-hosted teams can use standard Postgres with `LISTEN/NOTIFY`, which already works |
41+
42+
---
43+
44+
## 1. Security (Must-Have for Self-Hosting)
45+
46+
### 1.1 Rate Limiting on Auth Endpoints
47+
48+
**Gap:** The `rate_limits` table exists (`packages/server/migrations/019_rate_limits.sql`) but no middleware uses it. Admin login and device verification are vulnerable to brute force and credential stuffing.
49+
50+
**Evidence:**
51+
52+
```typescript
53+
// packages/server/src/routes/admin/auth.ts
54+
authRoutes.post("/login", async (c) => {
55+
// No rate limiting applied
56+
});
57+
```
58+
59+
**Required:**
60+
61+
1. Implement a sliding-window rate limiter backed by `betterbase_meta.rate_limits`.
62+
2. Apply to `/admin/auth/login`, `/device/verify`, and `/device/token`.
63+
3. Add progressive delays after repeated failures (5 min, 15 min, 1 hour).
64+
4. Emit structured security events for audit logging.
65+
66+
### 1.2 API Key Scope Enforcement
67+
68+
**Gap:** API keys include `scopes` in the database (`008_api_keys.sql`), but `requireAdmin` middleware does not validate them. A read-only key currently grants full admin access.
69+
70+
**Evidence:**
71+
72+
```typescript
73+
// packages/server/src/lib/admin-middleware.ts
74+
if (keyRows.length === 0) return c.json({ error: "Invalid API key" }, 401);
75+
// Scope check is missing
76+
```
77+
78+
**Required:**
79+
80+
1. Add scope requirements to privileged routes.
81+
2. Reject requests where the API key lacks the required scope.
82+
3. Add tests proving scope-based denial.
83+
84+
### 1.3 HTML Injection in Device Verification
85+
86+
**Gap:** The device verification page interpolates `userCode` directly into HTML without escaping.
87+
88+
**Evidence:**
89+
90+
```typescript
91+
// packages/server/src/routes/device/index.ts
92+
`<input name="user_code" value="${userCode ?? ""}" required />`
93+
```
94+
95+
**Required:**
96+
97+
1. Escape HTML entities (`<`, `>`, `"`, `&`) before interpolation.
98+
2. Add `Content-Security-Policy: default-src 'self'` to the response.
99+
100+
### 1.4 Security Headers
101+
102+
**Gap:** No global security headers are set. Missing `X-Frame-Options`, `X-Content-Type-Options`, `Strict-Transport-Security`.
103+
104+
**Required:**
105+
106+
1. Add a Hono middleware that sets:
107+
108+
```typescript
109+
c.header("X-Content-Type-Options", "nosniff");
110+
c.header("X-Frame-Options", "DENY");
111+
c.header("Strict-Transport-Security", "max-age=31536000; includeSubDomains");
112+
```
113+
114+
### 1.5 Secret Management
115+
116+
**Gap:** `.env.self-hosted.example` uses placeholder secrets. No validation enforces strong secrets or rotation.
117+
118+
**Required:**
119+
120+
1. Add startup validation: `BETTERBASE_JWT_SECRET` must be >= 32 chars and not a known weak value.
121+
2. Document secret rotation procedure in [SELF_HOSTED.md](../../SELF_HOSTED.md).
122+
123+
---
124+
125+
## 2. Stability & Reliability
126+
127+
### 2.1 Transactional Migrations
128+
129+
**Gap:** The migration runner executes SQL without per-file transactions. A failed migration can leave the database in an inconsistent state.
130+
131+
**Evidence:**
132+
133+
```typescript
134+
// packages/server/src/lib/migrate.ts
135+
const sql = await readFile(join(MIGRATIONS_DIR, file), "utf-8");
136+
await pool.query(sql); // No BEGIN/COMMIT
137+
await pool.query("INSERT INTO betterbase_meta.migrations (filename) VALUES ($1)", [file]);
138+
```
139+
140+
**Required:**
141+
142+
1. Wrap each migration in `BEGIN; ... COMMIT;` with rollback on failure.
143+
2. Abort server startup if any migration fails.
144+
3. Add idempotency checks (`IF NOT EXISTS`) to all DDL.
145+
146+
### 2.2 Deep Health Checks
147+
148+
**Gap:** `/health` returns a static JSON object. It does not verify database connectivity, storage accessibility, or Inngest reachability.
149+
150+
**Required:**
151+
152+
1. Implement deep health checks:
153+
154+
```typescript
155+
app.get("/health", async (c) => {
156+
const db = await checkDatabase();
157+
const storage = await checkStorage();
158+
const inngest = await checkInngest();
159+
const ok = db && storage && inngest;
160+
return c.json({ db, storage, inngest }, ok ? 200 : 503);
161+
});
162+
```
163+
164+
2. Add `/ready` (dependencies up, migrations complete) and `/live` (process alive) for orchestrators.
165+
166+
### 2.3 Graceful Shutdown
167+
168+
**Gap:** The server does not handle `SIGTERM`. In-flight requests and database connections may be dropped during restarts or deploys.
169+
170+
**Required:**
171+
172+
1. On `SIGTERM`, stop accepting new connections.
173+
2. Wait for active requests to finish (with a timeout, e.g., 30s).
174+
3. Close the `pg` pool and WebSocket server cleanly.
175+
4. Exit with code 0.
176+
177+
### 2.4 E2E Test Suite
178+
179+
**Gap:** Extensive unit tests exist, but no end-to-end tests validate the full self-hosted stack (client → Nginx → server → Postgres → MinIO).
180+
181+
**Required:**
182+
183+
1. Add E2E tests for critical self-hosted flows:
184+
- Admin login → project creation → function call
185+
- Storage upload via signed URL
186+
- Realtime subscription over WebSocket
187+
2. Run E2E tests in CI against `docker-compose.self-hosted.yml`.
188+
189+
---
190+
191+
## 3. Operations (Running It Day-to-Day)
192+
193+
### 3.1 Backup & Recovery
194+
195+
**Gap:** No automated backup strategy exists for Postgres or MinIO. Self-hosted teams are responsible for their own data but have no guidance or tooling.
196+
197+
**Required:**
198+
199+
1. Add a Postgres backup sidecar to `docker-compose.self-hosted.yml` (e.g., `pg_dump` cron or `wal-g`).
200+
2. Document restore procedure in [SELF_HOSTED.md](../../SELF_HOSTED.md).
201+
3. Add MinIO bucket versioning and replication guidance.
202+
203+
### 3.2 Metrics & Monitoring
204+
205+
**Gap:** No Prometheus metrics endpoint exists. Operators cannot observe request rates, error rates, or database latency.
206+
207+
**Required:**
208+
209+
1. Add `/metrics` endpoint exporting Prometheus format:
210+
- `http_requests_total` (method, route, status)
211+
- `db_query_duration_seconds`
212+
- `ws_connections_active`
213+
2. Provide a Grafana dashboard JSON for self-hosted deployments.
214+
3. Optionally add `docker-compose.observability.yml` with Prometheus + Grafana.
215+
216+
### 3.3 Log Aggregation
217+
218+
**Gap:** Pino logs are structured but not shipped anywhere by default. Debugging a multi-container self-hosted deployment is painful.
219+
220+
**Required:**
221+
222+
1. Document how to forward logs to Loki, Datadog, or CloudWatch.
223+
2. Ensure `request_id` is present in every log line for traceability.
224+
225+
### 3.4 Upgrade Path
226+
227+
**Gap:** No documented procedure for upgrading a self-hosted instance to a new BetterBase release.
228+
229+
**Required:**
230+
231+
1. Document upgrade checklist:
232+
- Backup database
233+
- Pull new image / rebuild
234+
- Run migrations (automatic on startup)
235+
- Verify `/health`
236+
2. Provide a `docker-compose.self-hosted.yml` that pins image tags for reproducible upgrades.
237+
238+
---
239+
240+
## 4. Multi-Instance Scaling (For Teams Running Multiple Replicas)
241+
242+
### 4.1 Shared State for WebSockets
243+
244+
**Gap:** WebSocket tickets and subscriptions are stored in-memory. If two server containers run behind a load balancer, a client connected to instance A will miss invalidations triggered on instance B.
245+
246+
**Required:**
247+
248+
1. Add optional Redis integration.
249+
2. Use Redis for:
250+
- WebSocket ticket storage (instead of `Map`)
251+
- Cross-instance pub/sub for realtime invalidations
252+
3. Document: "Run a single server container, or add Redis and run multiple."
253+
254+
### 4.2 Shared Rate Limiting
255+
256+
**Gap:** The `rate_limits` table is Postgres-backed (good), but if implemented as in-memory caching it would break across replicas. Ensure the implementation queries the database or uses Redis.
257+
258+
**Required:**
259+
260+
1. Implement rate limiting against Postgres or Redis so it works consistently across replicas.
261+
262+
### 4.3 Kubernetes Manifests
263+
264+
**Gap:** Only Docker Compose exists. Teams running K8s must write their own manifests.
265+
266+
**Required:**
267+
268+
1. Provide basic K8s manifests:
269+
- `Deployment` for `betterbase-server`
270+
- `Deployment` for dashboard (nginx)
271+
- `Service` and `Ingress`
272+
- `StatefulSet` or external managed Postgres note
273+
2. Add a simple Helm chart with `values.yaml` for configuration.
274+
275+
---
276+
277+
## Implementation Roadmap
278+
279+
### Phase 1: Security & Stability (Weeks 1–3)
280+
281+
| Task | Severity | Deliverable |
282+
|------|----------|-------------|
283+
| Rate limiting middleware | Critical | `packages/server/src/middleware/rate-limit.ts` |
284+
| Enforce API key scopes | Critical | Updated `requireAdmin` + tests |
285+
| Transactional migrations | High | Updated `migrate.ts` with `BEGIN/COMMIT` |
286+
| Escape HTML in device verify | High | Updated `device/index.ts` + CSP |
287+
| Security headers middleware | Medium | Global Hono middleware |
288+
| Deep health checks | High | `/health`, `/ready`, `/live` |
289+
| Graceful shutdown | Medium | `SIGTERM` handler in server entry |
290+
291+
### Phase 2: Operations (Weeks 3–5)
292+
293+
| Task | Severity | Deliverable |
294+
|------|----------|-------------|
295+
| Prometheus `/metrics` | High | Metrics endpoint + Grafana dashboard |
296+
| Backup sidecar + docs | High | Compose service + restore guide |
297+
| E2E test suite | High | Tests against self-hosted Compose stack |
298+
| Upgrade documentation | Medium | Update [SELF_HOSTED.md](../../SELF_HOSTED.md) |
299+
| Log aggregation guide | Low | Docs for Loki/CloudWatch forwarding |
300+
301+
### Phase 3: Multi-Instance (Weeks 5–7)
302+
303+
| Task | Severity | Deliverable |
304+
|------|----------|-------------|
305+
| Redis integration for WS | Medium | Cross-instance pub/sub |
306+
| Redis-backed rate limits | Medium | Shared state across replicas |
307+
| Kubernetes manifests | Medium | `k8s/` directory + Helm chart |
308+
309+
---
310+
311+
## Acceptance Criteria for Self-Hosted Production
312+
313+
All of the following must be true to declare BetterBase **production-ready for self-hosting**:
314+
315+
1. **Auth abuse gate:** Login and device verification enforce rate limits and temporary lockouts.
316+
2. **Scope gate:** API key scopes are enforced on protected routes.
317+
3. **Migration gate:** Each migration executes atomically; startup aborts on failure.
318+
4. **Health gate:** `/health` checks database, storage, and Inngest; `/ready` and `/live` exist.
319+
5. **Shutdown gate:** Server handles `SIGTERM` gracefully without dropping in-flight requests.
320+
6. **E2E gate:** Critical flows (auth, CRUD, storage, realtime) pass E2E tests in CI against the self-hosted Compose stack.
321+
7. **Backup gate:** Automated Postgres backups are documented and runnable via Compose.
322+
8. **Metrics gate:** Prometheus metrics are available at `/metrics` with a reference Grafana dashboard.
323+
324+
---
325+
326+
## Related
327+
328+
- [SELF_HOSTED.md](../../SELF_HOSTED.md) - Self-hosted deployment guide
329+
- [Production Checklist](./production-checklist.md) - Pre-deployment checklist
330+
- [Deployment](./deployment.md) - Deployment platform guides
331+
- [Security Best Practices](./security-best-practices.md) - Security hardening
332+
- [Hardening Review v3](../core/hardening-review-v3.md) - Backend security audit

0 commit comments

Comments
 (0)