Merge branch 'codex/fix-remaining-issues-and-raise-pr' into codex/implement-kubeflow-for-ml-pipeline-orchestration

Author: fuzziecoder

backend/README.md

@@ -14,6 +14,17 @@ pip install -r backend/requirements.txt
 
 ## Running the Backend
 
+### Core API Layer
+
+- **FastAPI** - High-performance Python API framework
+- **Uvicorn** - ASGI server for FastAPI
+- **Gunicorn** - Production process manager
+- **NGINX** - Reverse proxy, load balancing, and rate limiting
+
+### Optional Alternative
+
+- **Django** - Use when a full admin experience and enterprise-grade auth system are required
+
 ### Option 1: CLI Testing (Phase 1)
 
 Test pipeline execution from command line:

pipeline/README.md

@@ -8,6 +8,9 @@ A production-ready pipeline automation system built with:
 - **PostgreSQL** - Persistence
 - **AI Safety Module** - Failure prediction & anomaly handling
 - **BentoML + Feast + Kubeflow** - End-to-end model infrastructure
+- **Prometheus + Grafana** - Metrics collection and dashboards
+- **ELK Stack (Elasticsearch, Logstash, Kibana)** - Centralized logging
+- **Sentry** - Error monitoring and tracing
 
 ## Architecture Overview
 
@@ -115,6 +118,10 @@ docker-compose up -d
 | FastAPI Docs | http://localhost:8000/api/docs | - |
 | PostgreSQL | localhost:5432 | airflow / airflow |
 | Redis | localhost:6379 | - |
+| Prometheus | http://localhost:9090 | - |
+| Grafana | http://localhost:3000 | admin / admin |
+| Kibana | http://localhost:5601 | - |
+| Elasticsearch | http://localhost:9200 | - |
 
 ### 4. Create Your First Pipeline
 
@@ -193,6 +200,26 @@ curl -X POST http://localhost:8000/api/executions/pipeline-xxx/execute
 | GET | `/metrics` | Dashboard metrics |
 | GET | `/insights` | AI insights |
 
+
+## Observability & Monitoring
+
+### Metrics (Prometheus)
+- Backend exposes Prometheus metrics at `/metrics` via `prometheus-fastapi-instrumentator`.
+- Prometheus scrapes `backend:8000/metrics` every 15 seconds.
+
+### Dashboards (Grafana)
+- Grafana runs on port `3000` and can connect to Prometheus (`http://prometheus:9090`) as a data source.
+- Default credentials are `admin/admin` (change in production).
+
+### Centralized Logging (ELK Stack)
+- Elasticsearch stores indexed logs.
+- Logstash listens on `5000` (TCP JSON) and `5044` (beats) and forwards to Elasticsearch.
+- Kibana provides visualization for indices like `flexiroaster-backend-*`.
+
+### Error Monitoring (Sentry)
+- Configure `SENTRY_DSN` to enable Sentry for FastAPI exception capture and tracing.
+- Optional tuning: `SENTRY_ENVIRONMENT`, `SENTRY_TRACES_SAMPLE_RATE`, and `SENTRY_PROFILES_SAMPLE_RATE`.
+
 ## Configuration
 
 ### Environment Variables
@@ -205,6 +232,9 @@ curl -X POST http://localhost:8000/api/executions/pipeline-xxx/execute
 | `EXECUTOR_STAGE_TIMEOUT` | `120` | Stage timeout in seconds |
 | `AI_BLOCK_HIGH_RISK` | `false` | Block high-risk executions |
 | `AI_RISK_THRESHOLD_HIGH` | `0.7` | High risk threshold |
+| `SENTRY_DSN` | `""` | Enables Sentry when set |
+| `SENTRY_ENVIRONMENT` | `development` | Sentry environment label |
+| `SENTRY_TRACES_SAMPLE_RATE` | `0.1` | Fraction of traced requests |
 
 ### Airflow Variables
 

pipeline/backend/config.py

@@ -96,6 +96,14 @@ class Settings(BaseSettings):
     LOG_FORMAT: str = "json"  # "json" or "text"
     LOG_FILE: Optional[str] = None
 
+    # ===================
+    # Observability Settings
+    # ===================
+    SENTRY_DSN: str = ""
+    SENTRY_ENVIRONMENT: str = "development"
+    SENTRY_TRACES_SAMPLE_RATE: float = 0.1
+    SENTRY_PROFILES_SAMPLE_RATE: float = 0.1
+
     # ===================
     # Airflow Integration
     # ===================

pipeline/backend/main.py

@@ -18,6 +18,8 @@
 from core.redis_state import redis_state_manager
 from core.executor import pipeline_executor
 from api.routes import ai_automation, executions, health, model_infra, monitoring, pipelines
+from api.routes import pipelines, executions, health, monitoring, ai_automation
+from observability import setup_observability
 
 
 # ===================
@@ -133,6 +135,9 @@ async def lifespan(app: FastAPI):
     allow_headers=["*"],
 )
 
+# Configure metrics and error monitoring
+setup_observability(app)
+
 
 # ===================
 # Exception Handlers

pipeline/backend/observability.py

@@ -0,0 +1,39 @@
+"""Observability setup for metrics and error monitoring."""
+import logging
+
+import sentry_sdk
+from fastapi import FastAPI
+from prometheus_fastapi_instrumentator import Instrumentator
+from sentry_sdk.integrations.asgi import SentryAsgiMiddleware
+
+from config import settings
+
+logger = logging.getLogger(__name__)
+
+
+def setup_observability(app: FastAPI) -> None:
+    """Configure Prometheus metrics and Sentry error monitoring."""
+    Instrumentator(
+        should_group_status_codes=False,
+        should_ignore_untemplated=True,
+        should_respect_env_var=False,
+        should_instrument_requests_inprogress=True,
+        excluded_handlers=["/metrics", "/health"],
+        env_var_name="ENABLE_METRICS",
+        inprogress_name="flexiroaster_inprogress",
+        inprogress_labels=True,
+    ).instrument(app).expose(app, include_in_schema=False)
+
+    if not settings.SENTRY_DSN:
+        logger.info("Sentry disabled - SENTRY_DSN not configured")
+        return
+
+    sentry_sdk.init(
+        dsn=settings.SENTRY_DSN,
+        environment=settings.SENTRY_ENVIRONMENT,
+        release=f"{settings.APP_NAME}@{settings.APP_VERSION}",
+        traces_sample_rate=settings.SENTRY_TRACES_SAMPLE_RATE,
+        profiles_sample_rate=settings.SENTRY_PROFILES_SAMPLE_RATE,
+    )
+    app.add_middleware(SentryAsgiMiddleware)
+    logger.info("Sentry error monitoring enabled")

pipeline/backend/requirements.txt

@@ -30,6 +30,8 @@ pyyaml==6.0.1
 httpx==0.25.2
 structlog==23.2.0
 tenacity==8.2.3
+sentry-sdk[fastapi]==2.19.2
+prometheus-fastapi-instrumentator==7.0.0
 
 # Async support
 anyio==4.1.0

pipeline/docker-compose.yml

@@ -94,6 +94,12 @@ services:
 
       # Logging
       LOG_LEVEL: INFO
+
+      # Error Monitoring
+      SENTRY_DSN: ${SENTRY_DSN:-}
+      SENTRY_ENVIRONMENT: ${SENTRY_ENVIRONMENT:-local}
+      SENTRY_TRACES_SAMPLE_RATE: ${SENTRY_TRACES_SAMPLE_RATE:-0.1}
+      SENTRY_PROFILES_SAMPLE_RATE: ${SENTRY_PROFILES_SAMPLE_RATE:-0.1}
     ports:
       - "8000:8000"
     depends_on:
@@ -198,9 +204,91 @@ services:
       - -c
       - airflow
 
+  # Prometheus metrics store
+  prometheus:
+    image: prom/prometheus:v2.54.1
+    container_name: flexiroaster-prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+    volumes:
+      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    depends_on:
+      backend:
+        condition: service_healthy
+    restart: always
+
+  # Grafana dashboards
+  grafana:
+    image: grafana/grafana:11.2.2
+    container_name: flexiroaster-grafana
+    ports:
+      - "3000:3000"
+    environment:
+      GF_SECURITY_ADMIN_USER: admin
+      GF_SECURITY_ADMIN_PASSWORD: admin
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana-data:/var/lib/grafana
+    depends_on:
+      - prometheus
+    restart: always
+
+  # ELK Stack - Elasticsearch
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.1
+    container_name: flexiroaster-elasticsearch
+    environment:
+      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
+      discovery.type: single-node
+      xpack.security.enabled: "false"
+    volumes:
+      - ./monitoring/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
+      - elasticsearch-data:/usr/share/elasticsearch/data
+    ports:
+      - "9200:9200"
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://localhost:9200/_cluster/health || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 10
+    restart: always
+
+  # ELK Stack - Logstash
+  logstash:
+    image: docker.elastic.co/logstash/logstash:8.15.1
+    container_name: flexiroaster-logstash
+    volumes:
+      - ./monitoring/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
+    ports:
+      - "5000:5000"
+      - "5044:5044"
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+    restart: always
+
+  # ELK Stack - Kibana
+  kibana:
+    image: docker.elastic.co/kibana/kibana:8.15.1
+    container_name: flexiroaster-kibana
+    environment:
+      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
+    ports:
+      - "5601:5601"
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+    restart: always
+
 volumes:
   postgres-data:
   redis-data:
+  prometheus-data:
+  grafana-data:
+  elasticsearch-data:
 
 networks:
   default:

pipeline/monitoring/elasticsearch/elasticsearch.yml

@@ -0,0 +1,5 @@
+cluster.name: flexiroaster-observability
+node.name: flexiroaster-es01
+network.host: 0.0.0.0
+discovery.type: single-node
+xpack.security.enabled: false

pipeline/monitoring/logstash/logstash.conf

@@ -0,0 +1,31 @@
+input {
+  beats {
+    port => 5044
+  }
+
+  tcp {
+    port => 5000
+    codec => json
+  }
+}
+
+filter {
+  if [service] == "flexiroaster-backend" {
+    mutate {
+      add_field => { "[@metadata][target_index]" => "flexiroaster-backend-%{+YYYY.MM.dd}" }
+    }
+  } else {
+    mutate {
+      add_field => { "[@metadata][target_index]" => "flexiroaster-logs-%{+YYYY.MM.dd}" }
+    }
+  }
+}
+
+output {
+  elasticsearch {
+    hosts => ["http://elasticsearch:9200"]
+    index => "%{[@metadata][target_index]}"
+  }
+
+  stdout { codec => rubydebug }
+}

pipeline/monitoring/prometheus/prometheus.yml

@@ -0,0 +1,13 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'flexiroaster-backend'
+    metrics_path: /metrics
+    static_configs:
+      - targets: ['backend:8000']
+
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['prometheus:9090']