feat: add comprehensive Sentry monitoring and fix 429 rate limiting error

Author: rui-typelets

Dockerfile

@@ -38,6 +38,13 @@ RUN pnpm install --frozen-lockfile --prod
 
 # Copy built application from builder stage
 COPY --from=builder /app/dist ./dist
+COPY newrelic.js ./
+
+# Set New Relic environment variables
+ENV NEW_RELIC_NO_CONFIG_FILE=false
+ENV NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
+ENV NEW_RELIC_LOG=stdout
+ENV NEW_RELIC_LOG_LEVEL=info
 
 # Change ownership to non-root user
 RUN chown -R typelets:nodejs /app

README.md

@@ -22,6 +22,7 @@ The backend API for the [Typelets Application](https://github.com/typelets/typel
 - ⚡ **Fast & Type-Safe** with TypeScript and Hono
 - 🐘 **PostgreSQL** with Drizzle ORM
 - 💻 **Code Execution** via secure Judge0 API proxy
+- 📊 **Performance Monitoring** with New Relic APM
 
 ## Tech Stack
 
@@ -30,6 +31,7 @@ The backend API for the [Typelets Application](https://github.com/typelets/typel
 - **Database**: PostgreSQL with [Drizzle ORM](https://orm.drizzle.team/)
 - **Authentication**: [Clerk](https://clerk.com/)
 - **Validation**: [Zod](https://zod.dev/)
+- **Monitoring**: [New Relic](https://newrelic.com/) for APM and error tracking
 - **TypeScript**: Strict mode enabled for type safety
 
 ## Prerequisites
@@ -39,6 +41,7 @@ The backend API for the [Typelets Application](https://github.com/typelets/typel
 - PostgreSQL database (local installation or Docker)
 - Clerk account for authentication ([sign up here](https://dashboard.clerk.com))
 - Judge0 API key for code execution (optional - [get from RapidAPI](https://rapidapi.com/judge0-official/api/judge0-ce))
+- New Relic account for monitoring (optional - [sign up here](https://newrelic.com/signup))
 
 ## Local Development Setup
 
@@ -74,13 +77,16 @@ cp .env.example .env
    - Create a free account at [Clerk Dashboard](https://dashboard.clerk.com)
    - Create a new application
    - (Optional) Get Judge0 API key from [RapidAPI](https://rapidapi.com/judge0-official/api/judge0-ce)
+   - (Optional) Get New Relic license key from [New Relic Dashboard](https://one.newrelic.com/launcher/api-keys-ui.api-keys-launcher)
    - Update `.env` with your settings:
 
    ```env
    CLERK_SECRET_KEY=sk_test_your_actual_clerk_secret_key_from_dashboard
    CORS_ORIGINS=http://localhost:5173,http://localhost:3000
    # Optional: For code execution features
    JUDGE0_API_KEY=your_judge0_rapidapi_key_here
+   # Optional: For performance monitoring and error tracking
+   NEW_RELIC_LICENSE_KEY=your_new_relic_license_key_here
    ```
 
 5. **Set up database schema:**
@@ -255,6 +261,54 @@ The application uses the following main tables:
 - **WebSocket Security**: JWT authentication, rate limiting, and connection limits
 - **Real-time Authorization**: Database-level ownership validation for all WebSocket operations
 
+## Performance Monitoring
+
+The API includes comprehensive monitoring via New Relic APM for production observability and performance optimization.
+
+### Monitoring Features
+
+- **Application Performance Monitoring (APM)**: Real-time performance metrics for HTTP requests, database queries, and external API calls
+- **Error Tracking**: Automatic error detection and alerting with detailed stack traces
+- **Custom Events**: Application startup, user authentication, and business logic events
+- **Transaction Tracing**: Detailed breakdown of slow requests and database operations
+- **Infrastructure Monitoring**: Server resources, memory usage, and system health
+
+### Environment-Specific Monitoring
+
+**Development Environment:**
+- App Name: `typelets-api-dev`
+- Debug-level logging with detailed instrumentation
+- Lower performance thresholds for early issue detection
+- Enhanced error collection and context
+
+**Production Environment:**
+- App Name: `typelets-api-prod`
+- Optimized logging for performance
+- High-security mode with sensitive data filtering
+- Production-tuned sampling rates
+
+### Monitored Operations
+
+- **HTTP Requests**: All API endpoints with response times and status codes
+- **Database Queries**: PostgreSQL operations with query performance tracking
+- **External APIs**: Judge0 API calls with latency and error monitoring
+- **WebSocket Operations**: Real-time connection management and message processing
+- **File Operations**: Upload/download performance and error tracking
+
+### New Relic Dashboard Access
+
+Once configured, monitor your application at:
+- Development: Search for `typelets-api-dev` in your New Relic dashboard
+- Production: Search for `typelets-api-prod` in your New Relic dashboard
+
+### Custom Monitoring
+
+The application sends custom events for business intelligence:
+- `ApplicationStartup`: Server initialization with configuration details
+- User authentication events with performance metrics
+- Slow operation alerts (configurable thresholds)
+- Resource usage patterns and scaling indicators
+
 ## Environment Variables
 
 | Variable                     | Description                                  | Required | Default     |
@@ -274,8 +328,13 @@ The application uses the following main tables:
 | `WS_MAX_CONNECTIONS_PER_USER`| Max WebSocket connections per user          | No       | 20          |
 | `WS_AUTH_TIMEOUT_MS`         | WebSocket authentication timeout in milliseconds | No   | 30000 (30 sec) |
 | `JUDGE0_API_KEY`             | Judge0 API key for code execution            | No*      | -           |
+| `NEW_RELIC_LICENSE_KEY`      | New Relic license key for APM monitoring     | No**     | -           |
+| `NEW_RELIC_APP_NAME`         | Custom New Relic application name            | No       | Auto-generated |
+| `NEW_RELIC_LOG`              | New Relic log output (stdout/stderr/filepath) | No      | stdout (dev) / file (prod) |
+| `NEW_RELIC_LOG_LEVEL`        | New Relic logging level (error/warn/info/debug/trace) | No | debug (dev) / info (prod) |
 
 *Required only for code execution features
+**Required only for performance monitoring and error tracking
 
 ## Development
 
@@ -334,6 +393,14 @@ docker build -t typelets-api .
 
 # 3. Run API container for local testing
 docker run -p 3000:3000 --env-file .env typelets-api
+
+# Or with explicit New Relic configuration
+docker run -p 3000:3000 \
+  -e NEW_RELIC_LICENSE_KEY=your_license_key_here \
+  -e NEW_RELIC_APP_NAME="typelets-api-dev" \
+  -e NODE_ENV=development \
+  --env-file .env \
+  typelets-api
 ```
 
 **This Docker setup is for local development and testing only.**
@@ -367,6 +434,25 @@ This application is designed for production deployment using AWS ECS (Elastic Co
 
 For production deployment, configure the same environment variables in your ECS task definition that you use locally in `.env`.
 
+### Production Monitoring Setup
+
+For production deployments, ensure New Relic monitoring is properly configured:
+
+```bash
+# Example production environment variables in ECS task definition
+NODE_ENV=production
+NEW_RELIC_LICENSE_KEY=your_production_license_key
+NEW_RELIC_APP_NAME=typelets-api-prod
+NEW_RELIC_LOG=stdout
+NEW_RELIC_LOG_LEVEL=info
+```
+
+This will automatically:
+- Enable production-optimized monitoring settings
+- Filter sensitive data from logs and traces
+- Set appropriate sampling rates for production scale
+- Create separate application monitoring (dev vs prod)
+
 ## Contributing
 
 We welcome contributions from the community!

newrelic.js

@@ -0,0 +1,104 @@
+'use strict';
+
+const isDevelopment = process.env.NODE_ENV === 'development';
+const isProduction = process.env.NODE_ENV === 'production';
+
+// Debug logging for configuration
+console.log('🔧 New Relic Config Loading...');
+console.log(`   Environment: ${process.env.NODE_ENV || 'development'}`);
+console.log(`   License Key: ${process.env.NEW_RELIC_LICENSE_KEY ? 'Set' : 'Using default'}`);
+console.log(`   Development Mode: ${isDevelopment}`);
+
+/**
+ * New Relic agent configuration.
+ */
+exports.config = {
+  // Environment-specific app naming
+  app_name: [
+    isDevelopment ? 'typelets-api-dev' :
+    isProduction ? 'typelets-api-prod' :
+    'typelets-api-staging'
+  ],
+
+  license_key: process.env.NEW_RELIC_LICENSE_KEY,
+
+  // Environment-specific logging configuration
+  logging: {
+    level: isDevelopment ? 'debug' : 'info',
+    filepath: process.env.NEW_RELIC_LOG || (isDevelopment ? 'stdout' : 'newrelic_agent.log'),
+    enabled: true
+  },
+
+  // Allow all data to be sent in high-security environments
+  allow_all_headers: true,
+
+  // Enable distributed tracing
+  distributed_tracing: {
+    enabled: true
+  },
+
+  // Application logging forwarding - more verbose in dev
+  application_logging: {
+    forwarding: {
+      enabled: true,
+      max_samples_stored: isDevelopment ? 10000 : 1000
+    },
+    metrics: {
+      enabled: true
+    },
+    local_decorating: {
+      enabled: isDevelopment // Enable log decoration in development
+    }
+  },
+
+  // Error collection
+  error_collector: {
+    enabled: true,
+    ignore_status_codes: [404],
+    capture_events: true,
+    max_event_samples_stored: isDevelopment ? 100 : 30
+  },
+
+  // Performance monitoring - more detailed in dev
+  slow_sql: {
+    enabled: true,
+    max_samples: isDevelopment ? 20 : 10
+  },
+
+  // Transaction tracer - more aggressive in development
+  transaction_tracer: {
+    enabled: true,
+    transaction_threshold: isDevelopment ? 0.1 : 'apdex_f', // 100ms in dev vs apdex_f in prod
+    record_sql: 'obfuscated',
+    explain_threshold: isDevelopment ? 100 : 500, // Lower threshold in dev
+    max_segments: isDevelopment ? 3000 : 2000
+  },
+
+  // Custom insights events
+  custom_insights_events: {
+    enabled: true,
+    max_samples_stored: isDevelopment ? 30000 : 1000
+  },
+
+  // Feature flags for development debugging
+  feature_flag: {
+    serverless_mode: false,
+    await_support: true,
+    promise_segments: true,
+    custom_metrics: true
+  },
+
+  // Environment-specific data collection
+  high_security: isProduction, // Enable high security mode in production
+
+  // Attributes - more permissive in development
+  attributes: {
+    enabled: true,
+    include_enabled: true,
+    exclude: isProduction ? [
+      'request.headers.cookie',
+      'request.headers.authorization',
+      'request.headers.x-api-key'
+    ] : []
+  }
+};