Merge branch 'codex/fix-remaining-issues-and-raise-pr' into codex/implement-ray-for-ml-orchestration

Author: fuzziecoder

backend/.env.example

@@ -23,3 +23,13 @@ SENSITIVE_FIELDS=password,token,secret,api_key,authorization,credit_card,ssn
 # Development default: http://localhost:3000,http://localhost:5173,http://127.0.0.1:5173
 # Production example: https://yourdomain.com,https://app.yourdomain.com
 CORS_ORIGINS=http://localhost:3000,http://localhost:5173,http://127.0.0.1:5173
+
+# Event streaming (Kafka-compatible, supports Apache Kafka and Redpanda)
+ENABLE_EVENT_STREAMING=False
+EVENT_STREAM_BACKEND=kafka
+KAFKA_BOOTSTRAP_SERVERS=localhost:9092
+KAFKA_CLIENT_ID=flexiroaster-backend
+TOPIC_PIPELINE_CREATED=pipeline.created
+TOPIC_EXECUTION_STARTED=execution.started
+TOPIC_EXECUTION_FAILED=execution.failed
+TOPIC_EXECUTION_COMPLETED=execution.completed

backend/README.md

@@ -163,19 +163,35 @@ cp backend/.env.example backend/.env
 
 ## Event-Driven Architecture (Advanced Setup)
 
-FlexiRoaster supports Kafka-backed domain events for loose coupling, high scalability, audit-friendly workflows, and real-time analytics.
+FlexiRoaster supports Kafka-compatible domain events (Apache Kafka or Redpanda) for loose coupling, high scalability, audit-friendly workflows, and real-time analytics.
+
+
+#### Apache Kafka
+- Event-driven triggers
+- High-throughput ingestion
+- Real-time pipeline activation
+
+#### Redpanda
+- Kafka-compatible
+- Lower operational complexity
+
+Use this layer when:
+- Pipelines should trigger from events
+- You need real-time monitoring
+- You process millions of records
 
 ### Published topics
 - `pipeline.created`
 - `execution.started`
 - `execution.failed`
 - `execution.completed`
 
-### Enable Kafka publishing
+### Enable streaming publishing
 Set the following environment variables in `backend/.env`:
 
 ```env
 ENABLE_EVENT_STREAMING=true
+EVENT_STREAM_BACKEND=kafka
 KAFKA_BOOTSTRAP_SERVERS=localhost:9092
 KAFKA_CLIENT_ID=flexiroaster-backend
 TOPIC_PIPELINE_CREATED=pipeline.created
@@ -184,7 +200,7 @@ TOPIC_EXECUTION_FAILED=execution.failed
 TOPIC_EXECUTION_COMPLETED=execution.completed
 ```
 
-If Kafka is unavailable, the backend falls back to structured application logs for events so local development continues to work.
+If Kafka/Redpanda is unavailable, the backend falls back to structured application logs for events so local development continues to work.
 
 ## Next Steps
 

backend/api/routes/airflow.py

@@ -48,8 +48,11 @@ async def trigger_from_airflow(
             detail=f"Pipeline not found: {trigger_data.pipeline_id}",
         )
 
+    pipeline = pipelines_db[trigger_data.pipeline_id]
+
     execution = initialize_execution(
         trigger_data.pipeline_id,
+        user_id=pipeline.user_id,
         context={
             "triggered_by": "airflow",
             "airflow": {