Skip to content

WORKFLOWS.md

Latest commit

 

History

History
273 lines (210 loc) · 14.6 KB

WORKFLOWS.md

File metadata and controls

273 lines (210 loc) · 14.6 KB

Workflow Implementations

📋 Document Summary

What This Document Covers:

  • Overview of 8 production workflows in LaunchAgencyBot
  • InputSource → Processor → Executor architecture pattern
  • Real-time monitoring, AI generation, and curation workflows
  • Workflow categories (detection, generation, processing, editing, trends)
  • Links to detailed individual workflow documentation

Sections in This Document:

Related Documentation:

Context Tags: #workflows #architecture #domain-pattern #real-time #ai-generation

This directory contains complete workflow implementations demonstrating the full InputSource → Processor → Executor domain architecture pattern. The workflows provide production-ready systems for real-time token launch monitoring and AI-powered memecoin generation with advanced CLIP embeddings, RAG context retrieval, and professional curation workflows.

🚀 System Overview

The workflow system implements a sophisticated modular architecture with:

  • Real-time Event Processing: WebSocket-based monitoring with automatic reconnection
  • Multi-Stage Processing Pipelines: 5-7 stage validation with early termination
  • AI-Enhanced Content Generation: RAG-powered creation using curated examples with weighted multi-embedding search
  • CLIP Multimodal Integration: Text and image embeddings across 5 collections for semantic similarity
  • Professional Curation Workflow: 3-tier storage (pending/approved filesystem + vector database)
  • Complete Data Quality Control: Zero-tolerance policy for incomplete records

📋 Available Workflows (8 Total)

Each workflow has its own comprehensive documentation with architecture details, usage examples, and configuration options. All workflows follow the established InputSource → Processor → Executor architecture pattern.

🎯 1. Launch Detection Workflow

File: live_launch_detection_workflow.py Purpose: Real-time monitoring and processing of new token launches with complete data validation Architecture: 6-stage pipeline with WebSocket monitoring, AI classification, and CLIP embeddings Pattern Compliance: ✅ Complete

📖 View Detailed Documentation →

Key Features:

  • Real-time WebSocket token launch detection
  • AI-powered multimodal tag classification
  • CLIP embeddings for semantic similarity
  • Dual console + database output
  • Early termination for data quality control

📊 2. CSV Launch Detection Workflow

File: csv_launch_detection_workflow.py Purpose: Batch processing of historical token launches from CSV files Architecture: 5-stage pipeline with CSV batch reading, validation, and file storage Pattern Compliance: ✅ Complete

📖 View Detailed Documentation →

Key Features:

  • Batch CSV processing with configurable batch size
  • Same validation pipeline as live detection
  • Automatic row deletion after processing
  • Optimized for historical data analysis
  • No AI processing (delegated to RAG workflow)

🤖 3. RAG v2 Generation Workflow Service

File: rag_memecoin_generation_workflow.py Purpose: AI-powered memecoin creation with retrieval-augmented generation using dependency injection architecture Architecture: Dual-mode workflow supporting both standalone FastAPI service and orchestrator-managed operation Pattern Compliance: ✅ Complete

📖 View Detailed Documentation →

Key Features:

  • Dual-mode operation: Standalone (auto-init services) vs Orchestrator mode (injected services)
  • Dependency injection: Services passed via constructor for orchestrator integration
  • Callback pattern: Executors emit SessionStatusUpdateAction for real-time progress
  • Side-effect executors: Non-blocking incremental status updates to orchestrator
  • 6-stage pipeline: Query enhancement → RAG retrieval → LLM generation → Image gen → Assembly → Output
  • Multi-embedding RAG: Weighted search across 5 collections (1 image + 4 caption types)
  • Real-time progress: Stage-specific data attached to each status update (45%, 65%, 85%, 100%)
  • Session-based tracking: Immediate status availability via orchestrator memory
  • Complete file output: JSON + PNG with full generation metadata

🔄 4. RAG Memecoin Insertion Workflow

File: rag_memecoin_insertion_workflow.py Purpose: Processing memecoin JSON files from approved directory and inserting directly to confirmed collection Architecture: 8-stage enhanced pipeline with validation, tagging, captions, and vector DB insertion Pattern Compliance: ✅ Complete

📖 View Detailed Documentation →

Key Features:

  • Batch processing of JSON+PNG file pairs
  • Enhanced validation and data completeness checks
  • AI tag classification with main category mapping
  • Structured caption generation ([What] → [References] → [Visual Style] → [Emotions])
  • Direct confirmed collection insertion
  • Progress tracking and file cleanup

🔍 5. Trend Detection Workflow

File: trend_detection_workflow.py Purpose: Real-time AI-powered trend classification system for token launches with multimodal analysis Architecture: 4-stage pipeline with WebSocket monitoring, LLM classification, and Firebase trend management Pattern Compliance: ✅ Complete

📖 View Detailed Documentation →

Key Features:

  • Real-time WebSocket token launch monitoring
  • AI-powered trend classification using Gemini Flash 2.5
  • Multimodal trend creation (text + image analysis)
  • Firebase trend storage with automatic promotion (≥5 memecoins from potential to confirmed)
  • TrendOrchestrator integration for Web UI
  • Prominence-based trend ranking

🔧 6. RAG Memecoin Update Workflow

File: rag_memecoin_update_workflow.py Purpose: Complete vector database update pipeline with caption re-embedding and atomic 5-collection updates Architecture: 2-stage pipeline (validation + re-embedding) → atomic executor Pattern Compliance: ✅ Complete

📖 View Detailed Documentation →

Key Features:

  • Caption re-embedding with CLIP (entity, context, visual, emotions)
  • Image embedding preservation (image unchanged)
  • Atomic 5-collection ChromaDB updates
  • Edit proposal acceptance workflow
  • Complete validation and error handling
  • Orchestrator integration for accept proposal

✏️ 7. RAG Memecoin Edit Analysis Workflow

File: rag_memecoin_edit_analysis_workflow.py Purpose: AI-powered memecoin editing with LLM-based feedback analysis and targeted field regeneration Architecture: Single-stage workflow with EditProposalExecutor for proposal caching Pattern Compliance: ✅ Complete (refactored to use proper executor)

📖 View Detailed Documentation →

Key Features:

  • LLM-powered user feedback analysis
  • Intelligent field detection from natural language
  • Targeted regeneration of affected metadata only
  • EditProposalExecutor caches proposals for approval
  • Orchestrator integration via dependency injection
  • MemeStorageOrchestrator integration for database edits

🎨 8. Generated Meme Edit Analysis Workflow

File: generated_meme_edit_analysis_workflow.py Purpose: Intelligent editing workflow for generated memecoins with context-enhanced regeneration Architecture: 4-stage conditional pipeline with GeneratedEditProposalExecutor Pattern Compliance: ✅ Complete (refactored to use proper executor)

📖 View Detailed Documentation →

Key Features:

  • LLM-powered edit strategy selection (metadata-only vs. full regeneration)
  • Context retrieval from vector DB for problematic entities
  • Selective field regeneration (saves time and API costs)
  • Conditional image regeneration (only when needed)
  • GeneratedEditProposalExecutor caches proposals
  • GenerationOrchestrator integration via dependency injection

🏗️ Architecture Overview

All workflows follow the consistent InputSource → Processor → Executor domain architecture with advanced patterns for production use:

                  events                actions
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   InputSource   │───▶│   Processor     │───▶│    Executor     │
│                 │    │                 │    │                 │
│ • WebSocket     │    │ • Multi-Stage   │    │ • Console       │
│ • Files         │    │ • StageGraph    │    │ • Database      │
│ • HTTP API      │    │ • Early Term.   │    │ • File Output   │
│ • Web Request   │    │ • Side-Effects  │    │ • Side-Effects  │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Core Components

  • InputSources: WebSocket connections, file readers, HTTP API integrations that generate Events
  • Events: Structured domain objects representing workflow inputs (TokenLaunchEvent, GenerationRequestEvent)
  • Processors: Multi-stage validation and enrichment pipelines using StageGraph with early termination
  • Actions: Validated output models from processing (MemecoinAction, GeneratedTokenAction) that flow to Executors
  • Executors: Final execution layer (console display, database storage, file creation) with ExecutorRouter for multi-action workflows
  • Side-Effect Actions: Special actions that update external state (SessionStatusUpdateAction for progress tracking)

Shared Services

  • LLM Service: AI-powered text generation and classification via LiteLLM
  • CLIP Service: Multimodal embeddings for text and images via Replicate API
  • Vector Store: ChromaDB 5-collection multi-embedding database with weighted semantic search
  • Progress Tracking: Real-time incremental updates via side-effect executors with orchestrator callbacks

Production Patterns

  • Dependency Injection: Services can be injected via constructor for orchestrator-managed mode, or auto-initialized for standalone mode (RAG v2 Generation Workflow)
  • Callback Pattern: Executors emit side-effect actions to update orchestrator state via injected references (ExecutorRouter → Orchestrator)
  • Side-Effect Executors: Special executors that update external state without producing files/database entries (SessionStatusUpdateExecutor)
  • Multi-Action Processing: ExecutorRouter routes different Action types to appropriate Executors
  • Incremental Progress Updates: Each stage executor emits progress updates with stage-specific data (RAG v2 Workflow: 45% RAG, 65% LLM, 85% Image, 100% Output)
  • Dual-Mode Operation: Workflows detect service injection and adapt initialization (Standalone vs Orchestrator)
  • StageGraph: Directed graph execution of processing stages with dependency management
  • Early Termination: Zero-tolerance validation that immediately stops processing on incomplete data
  • Dual Output: Most workflows support both console output and database storage simultaneously

🎯 Workflow Selection Guide

Workflow Primary Use Case Key Feature
Live Launch Detection Real-time token launch monitoring WebSocket monitoring with AI classification and CLIP embeddings
CSV Launch Detection Batch processing of historical launches CSV batch reading with automatic row deletion
RAG v2 Generation Service AI-generated memecoins via web UI/API Microservice architecture with session-based real-time status tracking
RAG Memecoin Insertion Process approved files with AI enhancement 8-stage pipeline with direct confirmed collection insertion
Trend Detection Real-time trend discovery and tracking AI-powered multimodal trend classification with Firebase promotion
RAG Memecoin Edit Analysis AI-powered metadata editing from user feedback LLM-based field detection with targeted regeneration and proposal caching
RAG Memecoin Update Vector database metadata updates Caption re-embedding with atomic 5-collection updates
Generated Meme Edit Analysis Intelligent editing with context-enhanced regeneration LLM-powered edit strategy selection (metadata-only vs. full regen)

🔧 Configuration

Each workflow uses specialized configuration files in res/config/:

  • litellm.yaml - Main LLM service configuration
  • clip_litellm.yaml - CLIP embedding service settings
  • generate_meme_llm.yaml - RAG generation LLM config
  • image_generation_llm.yaml - AI image generation settings

See individual workflow documentation for detailed configuration options.

📚 Additional Resources

For detailed workflow-specific documentation, architecture diagrams, usage examples, and configuration options, please refer to each workflow's individual README file linked above.