Skip to content

Commit 8de85b7

Browse files
committed
feat(websocket): implement real-time sync with HMAC authentication and fix folder moves
- Add complete WebSocket implementation for real-time note/folder synchronization - JWT authentication with 30-second timeout - Rate limiting (300 msg/min) and connection limits (20/user) - Connection management with automatic cleanup - Nonce-based replay attack prevention - Fix critical folder move sync issue - Add 'folderId' to allowed fields in note updates - Enable real-time sync of note folder changes across devices - Add comprehensive error logging for debugging - Enhance security infrastructure - Add security headers middleware (CSP, HSTS, XSS protection) - Implement enhanced rate limiting middleware - Fix TypeScript compilation issues - Resolve Map iterator compatibility with ES2022 target - Fix postgres module import issues - Add downlevelIteration support for future compatibility - Update comprehensive documentation - Add WEBSOCKET_INTEGRATION.md with complete protocol specification - Update README.md with current project structure and WebSocket features - Update SECURITY.md with latest security features and recommendations
1 parent 66b82e4 commit 8de85b7

24 files changed

+2478
-78
lines changed

.env.example

Lines changed: 175 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,190 @@
1-
# LOCAL DEVELOPMENT AND TESTING ENVIRONMENT VARIABLES
2-
# This file is ONLY for local development and testing
3-
# Copy this file to .env and update with your local testing values
1+
# ================================================================
2+
# TYPELETS API - ENVIRONMENT CONFIGURATION TEMPLATE
3+
# ================================================================
4+
# This file is for LOCAL DEVELOPMENT AND TESTING ONLY
5+
# Copy this file to `.env` and update with your actual values
6+
# Production deployments use AWS ECS task definitions, not .env files
47

5-
# Database connection for local testing
6-
# For Docker PostgreSQL (recommended for testing):
8+
# ================================================================
9+
# REQUIRED CONFIGURATION
10+
# ================================================================
11+
12+
# Database Connection (REQUIRED)
13+
# For Docker PostgreSQL (recommended for local development):
714
DATABASE_URL=postgresql://postgres:devpassword@localhost:5432/typelets_local
815
# For local PostgreSQL installation:
916
# DATABASE_URL=postgresql://postgres:your_password@localhost:5432/typelets_local
17+
# For production (example):
18+
# DATABASE_URL=postgresql://username:password@hostname:5432/database?sslmode=require
1019

11-
# Clerk authentication for local testing (get from https://dashboard.clerk.com)
20+
# Clerk Authentication (REQUIRED)
21+
# Get your secret key from: https://dashboard.clerk.com/
22+
# For development, use test keys (sk_test_...)
23+
# For production, use live keys (sk_live_...)
1224
CLERK_SECRET_KEY=sk_test_your_actual_clerk_secret_key_from_dashboard
1325

14-
# CORS configuration - REQUIRED
15-
# Comma-separated list of allowed origins
26+
# CORS Origins (REQUIRED)
27+
# Comma-separated list of allowed origins (no spaces after commas)
28+
# Include all frontend domains that will access this API
1629
CORS_ORIGINS=http://localhost:5173,http://localhost:3000
30+
# Production example:
31+
# CORS_ORIGINS=https://app.typelets.com,https://typelets.com
32+
33+
# ================================================================
34+
# SERVER CONFIGURATION
35+
# ================================================================
1736

18-
# Optional settings (have sensible defaults)
37+
# Server Port
1938
PORT=3000
39+
40+
# Environment
2041
NODE_ENV=development
2142

22-
# File upload configuration
23-
# MAX_FILE_SIZE_MB=50 # Maximum size per file in MB (default: 50)
24-
# MAX_NOTE_SIZE_MB=1024 # Maximum total attachments per note in MB (default: 1024MB = 1GB)
43+
# ================================================================
44+
# SECURITY CONFIGURATION
45+
# ================================================================
46+
47+
# WebSocket Message Authentication
48+
# Generate with: openssl rand -hex 32
49+
# CRITICAL: Use a strong random string (minimum 32 characters) in production
50+
MESSAGE_AUTH_SECRET=your-very-secure-random-string-here-min-32-chars
51+
52+
# ================================================================
53+
# RATE LIMITING CONFIGURATION
54+
# ================================================================
55+
# Current defaults are development-friendly
56+
# Adjust these values based on your usage patterns
57+
58+
# HTTP API Rate Limiting
59+
# HTTP_RATE_LIMIT_WINDOW_MS=900000 # 15 minutes in milliseconds
60+
# HTTP_RATE_LIMIT_MAX_REQUESTS=1000 # Max requests per window per user/IP
61+
# HTTP_FILE_RATE_LIMIT_MAX=100 # Max file operations per window
62+
63+
# WebSocket Rate Limiting
64+
WS_RATE_LIMIT_WINDOW_MS=60000 # Time window: 1 minute (in milliseconds)
65+
WS_RATE_LIMIT_MAX_MESSAGES=300 # Max messages per window per connection
66+
WS_MAX_CONNECTIONS_PER_USER=20 # Max concurrent connections per user
67+
WS_AUTH_TIMEOUT_MS=30000 # Authentication timeout: 30 seconds
68+
69+
# Production Rate Limiting Recommendations:
70+
# - HTTP: 500-1000 requests per 15 minutes per user
71+
# - WebSocket: 100-300 messages per minute per connection
72+
# - File uploads: 50-100 operations per 15 minutes per user
73+
# - Connections: 10-20 per user depending on multi-device usage
74+
75+
# ================================================================
76+
# FILE UPLOAD CONFIGURATION
77+
# ================================================================
78+
79+
# File Size Limits
80+
MAX_FILE_SIZE_MB=50 # Maximum size per file (default: 50MB)
81+
# MAX_NOTE_SIZE_MB=1024 # Maximum total attachments per note (default: 1GB)
82+
83+
# Allowed File Types (handled in code, documented here)
84+
# Images: JPEG, PNG, GIF, WebP
85+
# Documents: PDF, Plain Text, Markdown, JSON, CSV
86+
# Add new types in: src/lib/validation.ts
87+
88+
# ================================================================
89+
# BILLING & LIMITS CONFIGURATION
90+
# ================================================================
91+
92+
# Free Tier Limits
93+
FREE_TIER_STORAGE_GB=1 # Storage limit for free users (default: 1GB)
94+
FREE_TIER_NOTE_LIMIT=100 # Note count limit for free users (default: 100)
95+
96+
# Usage tracking for billing analytics
97+
# These limits trigger billing events logged to console
98+
# Upgrade prompts and enforcement handled in frontend
99+
100+
# ================================================================
101+
# LOGGING & MONITORING
102+
# ================================================================
103+
104+
# Debug Logging
105+
# DEBUG=false # Set to true for verbose logging (not recommended in production)
106+
107+
# Application Logging
108+
# Structured logs are automatically generated for:
109+
# - Authentication events
110+
# - Rate limiting violations
111+
# - Security events (failed auth, suspicious activity)
112+
# - Billing limit violations
113+
# - WebSocket connection events
114+
# - File upload events
115+
116+
# ================================================================
117+
# DEVELOPMENT HELPERS
118+
# ================================================================
119+
120+
# Database Development
121+
# Run with Docker: docker run --name typelets-postgres -e POSTGRES_PASSWORD=devpassword -p 5432:5432 -d postgres:15
122+
# Connect: psql postgresql://postgres:devpassword@localhost:5432/typelets_local
123+
124+
# Frontend Development URLs
125+
# Vite dev server: http://localhost:5173
126+
# Next.js dev server: http://localhost:3000
127+
# React dev server: http://localhost:3000
128+
129+
# API Testing
130+
# Health check: GET http://localhost:3000/health
131+
# WebSocket status: GET http://localhost:3000/websocket/status
132+
# API documentation: https://github.com/typelets/typelets-api
133+
134+
# ================================================================
135+
# SECURITY NOTES
136+
# ================================================================
137+
138+
# 🔒 NEVER commit actual secrets to version control
139+
# 🔒 Use different secrets for development and production
140+
# 🔒 Rotate secrets regularly in production
141+
# 🔒 MESSAGE_AUTH_SECRET is critical for WebSocket security
142+
# 🔒 CLERK_SECRET_KEY controls all authentication
143+
# 🔒 DATABASE_URL contains database credentials
144+
145+
# Generate secure secrets:
146+
# MESSAGE_AUTH_SECRET: openssl rand -hex 32
147+
# API Keys: Use your service provider's dashboard
148+
# Database passwords: Use password managers
149+
150+
# ================================================================
151+
# PRODUCTION DEPLOYMENT NOTES
152+
# ================================================================
153+
154+
# AWS ECS Deployment:
155+
# - Environment variables are set in ECS task definitions
156+
# - Secrets are managed via AWS Secrets Manager
157+
# - This .env file is NOT used in production
158+
# - SSL/TLS termination handled by Application Load Balancer
159+
# - Database connections use IAM authentication or managed secrets
160+
161+
# Docker Production:
162+
# - Pass environment variables via docker run -e or docker-compose
163+
# - Mount secrets as files or use Docker secrets
164+
# - Never build secrets into Docker images
165+
166+
# Kubernetes Deployment:
167+
# - Use ConfigMaps for non-sensitive configuration
168+
# - Use Secrets for sensitive data (base64 encoded)
169+
# - Consider external secret management (HashiCorp Vault, etc.)
170+
171+
# ================================================================
172+
# TROUBLESHOOTING
173+
# ================================================================
25174

26-
# Free tier limits for billing
27-
# FREE_TIER_STORAGE_GB=1 # Storage limit in GB for free tier (default: 1)
28-
# FREE_TIER_NOTE_LIMIT=100 # Note count limit for free tier (default: 100)
175+
# Common Issues:
176+
# 1. "Database connection failed" → Check DATABASE_URL and database status
177+
# 2. "Authentication failed" → Verify CLERK_SECRET_KEY is correct
178+
# 3. "CORS error" → Add your frontend URL to CORS_ORIGINS
179+
# 4. "Too many requests" → Increase rate limits or check for loops
180+
# 5. "WebSocket connection failed" → Check authentication and rate limits
29181

30-
# Debug logging
31-
# DEBUG=false # Set to true to enable debug logging in production
182+
# Useful Commands:
183+
# Check database connection: npx drizzle-kit studio
184+
# Test API endpoints: curl http://localhost:3000/health
185+
# Monitor logs: tail -f logs/app.log (if file logging enabled)
186+
# Generate new secrets: openssl rand -hex 32
32187

33-
# IMPORTANT: This .env file is for LOCAL TESTING ONLY
34-
# Production uses AWS ECS task definitions, not .env files
188+
# Support:
189+
# Documentation: https://github.com/typelets/typelets-api
190+
# Issues: https://github.com/typelets/typelets-api/issues

CHANGELOG.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4949

5050
### Added
5151

52+
- **Real-time WebSocket Sync**: Complete WebSocket implementation for real-time note and folder synchronization
53+
- JWT authentication for WebSocket connections
54+
- Optional HMAC-SHA256 message authentication for enhanced security
55+
- Rate limiting (300 messages/min) and connection limits (20 per user)
56+
- Connection management with automatic cleanup
57+
- Note and folder sync across multiple devices/sessions
58+
- Support for note folder moves via WebSocket
59+
- **Security Enhancements**:
60+
- Comprehensive security headers middleware (CSP, HSTS, XSS protection)
61+
- Enhanced rate limiting middleware
62+
- Production-ready security configuration
63+
- **TypeScript Improvements**: Fixed iterator compatibility and module import issues
64+
- **Documentation**: Complete WebSocket integration documentation and updated security policy
65+
66+
### Fixed
67+
68+
- WebSocket note sync issue where `folderId` changes weren't being broadcast
69+
- TypeScript compilation issues with Map iterators and postgres module imports
70+
- Memory leak prevention in nonce storage with automatic cleanup
71+
72+
### Previous Releases
73+
5274
- Initial open source release
5375
- TypeScript API with Hono framework
5476
- PostgreSQL database with Drizzle ORM

README.md

Lines changed: 68 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ The backend API for the [Typelets Application](https://github.com/typelets/typel
1818
- 📎 **File Attachments** with encrypted storage
1919
- 🏷️ **Tags & Search** for easy note discovery
2020
- 🗑️ **Trash & Archive** functionality
21+
- 🔄 **Real-time Sync** via WebSockets for multi-device support
2122
-**Fast & Type-Safe** with TypeScript and Hono
2223
- 🐘 **PostgreSQL** with Drizzle ORM
2324

@@ -91,6 +92,8 @@ pnpm run dev
9192

9293
🎉 **Your API is now running at `http://localhost:3000`**
9394

95+
**WebSocket connection available at: `ws://localhost:3000`**
96+
9497
The development server will automatically restart when you make changes to any TypeScript files.
9598

9699
### Why This Setup?
@@ -156,6 +159,7 @@ If you prefer to install PostgreSQL locally instead of Docker:
156159

157160
- `GET /` - API information and health status
158161
- `GET /health` - Health check endpoint
162+
- `GET /websocket/status` - WebSocket server status and statistics
159163

160164
### Authentication
161165

@@ -195,6 +199,30 @@ All `/api/*` endpoints require authentication via Bearer token in the Authorizat
195199
- `GET /api/files/:id` - Get file details
196200
- `DELETE /api/files/:id` - Delete file attachment
197201

202+
### WebSocket Real-time Sync
203+
204+
The API provides real-time synchronization via WebSocket connection at `ws://localhost:3000` (or your configured port).
205+
206+
**Connection Flow:**
207+
1. Connect to WebSocket endpoint
208+
2. Send authentication message with Clerk JWT token
209+
3. Join/leave specific notes for real-time updates
210+
4. Receive real-time sync messages for notes and folders
211+
212+
**Message Types:**
213+
- `auth` - Authenticate with JWT token
214+
- `ping`/`pong` - Heartbeat messages
215+
- `join_note`/`leave_note` - Subscribe/unsubscribe from note updates
216+
- `note_update` - Real-time note content changes and folder moves
217+
- `note_created`/`note_deleted` - Note lifecycle events
218+
- `folder_created`/`folder_updated`/`folder_deleted` - Folder events
219+
220+
**Security Features:**
221+
- JWT authentication required for all operations
222+
- Authorization checks ensure users only access their own notes/folders
223+
- Rate limiting (configurable, default: 300 messages per minute per connection)
224+
- Connection limits (configurable, default: 20 connections per user)
225+
198226
## Database Schema
199227

200228
The application uses the following main tables:
@@ -211,22 +239,28 @@ The application uses the following main tables:
211239
- **Input Validation**: Comprehensive Zod schemas for all inputs
212240
- **SQL Injection Protection**: Parameterized queries via Drizzle ORM
213241
- **CORS Configuration**: Configurable allowed origins
214-
- **Rate Limiting**: Configurable file size limits (default: 50MB per file, 1GB total per note)
242+
- **File Size Limits**: Configurable limits (default: 50MB per file, 1GB total per note)
243+
- **WebSocket Security**: JWT authentication, rate limiting, and connection limits
244+
- **Real-time Authorization**: Database-level ownership validation for all WebSocket operations
215245

216246
## Environment Variables
217247

218-
| Variable | Description | Required | Default |
219-
| --------------------- | -------------------------------------------- | -------- | ----------- |
220-
| `DATABASE_URL` | PostgreSQL connection string | Yes | - |
221-
| `CLERK_SECRET_KEY` | Clerk secret key for JWT verification | Yes | - |
222-
| `CORS_ORIGINS` | Comma-separated list of allowed CORS origins | Yes | - |
223-
| `PORT` | Server port | No | 3000 |
224-
| `NODE_ENV` | Environment (development/production) | No | development |
225-
| `MAX_FILE_SIZE_MB` | Maximum size per file in MB | No | 50 |
226-
| `MAX_NOTE_SIZE_MB` | Maximum total attachments per note in MB | No | 1024 (1GB) |
227-
| `FREE_TIER_STORAGE_GB`| Free tier storage limit in GB | No | 1 |
228-
| `FREE_TIER_NOTE_LIMIT`| Free tier note count limit | No | 100 |
229-
| `DEBUG` | Enable debug logging in production | No | false |
248+
| Variable | Description | Required | Default |
249+
| ---------------------------- | -------------------------------------------- | -------- | ----------- |
250+
| `DATABASE_URL` | PostgreSQL connection string | Yes | - |
251+
| `CLERK_SECRET_KEY` | Clerk secret key for JWT verification | Yes | - |
252+
| `CORS_ORIGINS` | Comma-separated list of allowed CORS origins | Yes | - |
253+
| `PORT` | Server port | No | 3000 |
254+
| `NODE_ENV` | Environment (development/production) | No | development |
255+
| `MAX_FILE_SIZE_MB` | Maximum size per file in MB | No | 50 |
256+
| `MAX_NOTE_SIZE_MB` | Maximum total attachments per note in MB | No | 1024 (1GB) |
257+
| `FREE_TIER_STORAGE_GB` | Free tier storage limit in GB | No | 1 |
258+
| `FREE_TIER_NOTE_LIMIT` | Free tier note count limit | No | 100 |
259+
| `DEBUG` | Enable debug logging in production | No | false |
260+
| `WS_RATE_LIMIT_WINDOW_MS` | WebSocket rate limit window in milliseconds | No | 60000 (1 min) |
261+
| `WS_RATE_LIMIT_MAX_MESSAGES` | Max WebSocket messages per window | No | 300 |
262+
| `WS_MAX_CONNECTIONS_PER_USER`| Max WebSocket connections per user | No | 20 |
263+
| `WS_AUTH_TIMEOUT_MS` | WebSocket authentication timeout in milliseconds | No | 30000 (30 sec) |
230264

231265
## Development
232266

@@ -240,15 +274,29 @@ src/
240274
├── lib/
241275
│ └── validation.ts # Zod validation schemas
242276
├── middleware/
243-
│ └── auth.ts # Authentication middleware
277+
│ ├── auth.ts # Authentication middleware
278+
│ ├── rate-limit.ts # Rate limiting middleware
279+
│ └── security.ts # Security headers middleware
244280
├── routes/
245-
│ ├── files.ts # File attachment routes
246-
│ ├── folders.ts # Folder management routes
247-
│ ├── notes.ts # Note management routes
248-
│ └── users.ts # User profile routes
281+
│ ├── files.ts # File attachment routes
282+
│ ├── folders.ts # Folder management routes
283+
│ ├── notes.ts # Note management routes
284+
│ └── users.ts # User profile routes
249285
├── types/
250-
│ └── index.ts # TypeScript type definitions
251-
└── server.ts # Application entry point
286+
│ └── index.ts # TypeScript type definitions
287+
├── websocket/
288+
│ ├── auth/
289+
│ │ └── handler.ts # JWT authentication and HMAC verification
290+
│ ├── handlers/
291+
│ │ ├── base.ts # Base handler for resource operations
292+
│ │ ├── notes.ts # Note sync operations
293+
│ │ └── folders.ts # Folder sync operations
294+
│ ├── middleware/
295+
│ │ ├── connection-manager.ts # Connection tracking and cleanup
296+
│ │ └── rate-limiter.ts # WebSocket rate limiting
297+
│ ├── types.ts # WebSocket message types
298+
│ └── index.ts # Main WebSocket server manager
299+
└── server.ts # Application entry point
252300
```
253301

254302
### Type Safety

0 commit comments

Comments
 (0)