Skip to content

0019-python-only-architecture.md

Latest commit

 

History

History
491 lines (380 loc) · 17.8 KB

0019-python-only-architecture.md

File metadata and controls

491 lines (380 loc) · 17.8 KB
Raw

19. Python-Only Architecture Decision

Status

Accepted (Supersedes ADR-0001)

Context

While ADR-0001 proposed a hybrid C#/Python architecture, the project has been successfully implemented using Python exclusively. This decision reflects the actual implementation and provides clarity on the technology stack in use.

Decision

We will use a Python-only architecture for the entire codeflow-engine project with the following stack:

  • Python 3.12+ for all application code
  • FastAPI for REST API and server components
  • Flask-SocketIO for real-time WebSocket communication
  • PostgreSQL with SQLAlchemy and Alembic for data persistence
  • Redis for caching and queue management
  • Poetry for dependency management
  • Docker for containerization and deployment

Key Technologies

Core Framework

  • Python 3.12+ with type hints for type safety
  • Pydantic v2 for data validation and settings management
  • Structlog for structured JSON logging

AI/ML Stack

  • OpenAI GPT models (GPT-4, GPT-3.5)
  • Anthropic Claude models
  • Mistral AI and Groq for alternative providers
  • AutoGen for multi-agent orchestration

Integrations

  • PyGithub for GitHub API
  • GitPython for Git operations
  • aiohttp for async HTTP requests
  • websockets for real-time communication

Database & Persistence

  • PostgreSQL (via psycopg2-binary)
  • SQLAlchemy for ORM
  • Alembic for migrations
  • Redis for caching

Observability

  • OpenTelemetry SDK for distributed tracing
  • Prometheus metrics (via prometheus-client)
  • Sentry for error tracking
  • DataDog for monitoring

Alternatives Considered

A thorough evaluation was conducted comparing multiple architectural approaches and technology stacks.

Option 1: Python-Only (Chosen)

Technology: Python 3.12+ with FastAPI, async/await, Pydantic

Pros:

  • Rich AI/ML ecosystem (OpenAI, Anthropic, LangChain, AutoGen)
  • Rapid development and iteration
  • Single language reduces complexity
  • Strong async capabilities (asyncio, aiohttp)
  • Excellent type safety with Pydantic v2 and type hints
  • Mature deployment tooling (Docker, K8s)
  • Lower learning curve for data science teams

Cons:

  • Lower raw CPU performance vs compiled languages
  • Global Interpreter Lock (GIL) limits CPU-bound parallelism
  • Higher memory footprint
  • Slower cold start times

Separation of Concerns:

  • Modular package structure (codeflow_engine.*)
  • Plugin system for extensibility
  • Service layer pattern for business logic
  • Repository pattern for data access
  • Clear API boundaries with FastAPI routers

Option 2: Rust for Performance-Critical Components

Technology: Rust + Python hybrid via PyO3/maturin

Pros:

  • Exceptional performance (comparable to C++)
  • Memory safety without garbage collection
  • Zero-cost abstractions
  • Growing ecosystem (Tokio, Actix, Rocket)
  • Excellent concurrency model
  • Python integration via PyO3

Cons:

  • Steeper learning curve (ownership/borrowing)
  • Smaller AI/ML ecosystem
  • Longer compile times
  • More complex debugging across language boundary
  • Limited team expertise in Rust
  • Overhead of FFI (Foreign Function Interface)

Separation of Concerns:

  • Could isolate performance-critical paths (e.g., data processing, parsing)
  • Python for orchestration, Rust for computation
  • Clear FFI boundaries

Decision: Not chosen for initial implementation due to added complexity and lack of immediate performance requirements. Remains viable for future optimization.

Option 3: C# (.NET 8+) Hybrid Architecture

Technology: C# for services + Python for AI/ML

Pros:

  • Excellent performance (JIT + AOT compilation)
  • Strong type system
  • Mature ecosystem (.NET, ASP.NET Core)
  • Good async/await support
  • Native cloud support (Azure)
  • Cross-platform (.NET 8+)

Cons:

  • Weaker AI/ML ecosystem vs Python
  • Requires gRPC or REST for inter-service communication
  • Increased deployment complexity
  • Split team expertise required
  • More complex build pipelines
  • Limited direct access to Python AI libraries

Separation of Concerns:

  • C# for core engine, APIs, infrastructure
  • Python for AI/ML processing
  • gRPC for communication (ADR-0002)
  • Clear service boundaries

Decision: Proposed in ADR-0001 but not implemented. The communication overhead and complexity outweighed benefits given Python's adequate performance for our workload.

Option 4: Go for Microservices

Technology: Go for services + Python for AI/ML

Pros:

  • Fast compilation and execution
  • Excellent concurrency (goroutines)
  • Simple deployment (single binary)
  • Good performance
  • Strong standard library
  • Lower memory usage than Python

Cons:

  • Limited AI/ML ecosystem
  • Less mature web framework ecosystem vs Python/C#
  • Weaker type system vs Rust/C#
  • No native async/await (goroutines are different model)
  • Would still need Python for AI features

Separation of Concerns:

  • Go for high-throughput APIs
  • Python for AI processing
  • REST/gRPC communication

Decision: Not chosen. Go doesn't provide sufficient advantage over Python for our use case, and still requires Python for AI capabilities.

Option 5: Polyglot Microservices (Multiple Languages)

Technology: Python (AI) + Rust (performance) + C#/Go (services)

Pros:

  • Best tool for each job
  • Maximum performance potential
  • Clear separation by language boundaries

Cons:

  • Extreme complexity
  • Multiple deployment pipelines
  • Cross-language debugging difficulties
  • Team fragmentation
  • Increased operational overhead
  • Version management nightmare

Decision: Rejected as over-engineering for current scale. May reconsider at massive scale (10x+ current load).

Rationale for Python-Only

Primary Factors

  1. AI/ML Ecosystem Dominance: Python is the undisputed leader for AI/ML with direct access to:

    • LLM providers (OpenAI, Anthropic, Mistral)
    • Multi-agent frameworks (AutoGen, LangChain, CrewAI)
    • Data processing (NumPy, Pandas)
    • ML libraries (scikit-learn, transformers)

  2. Development Velocity: Single language stack accelerates:

    • Feature development (no cross-language coordination)
    • Debugging (single toolchain)
    • Testing (unified test framework)
    • Onboarding (one language to learn)

  3. Sufficient Performance: For our workload characteristics:

    • I/O-bound operations (API calls, database queries)
    • Async Python handles I/O efficiently
    • CPU-intensive work is minimal (done by LLM APIs)
    • Redis caching reduces computation needs

  4. Separation of Concerns via Python: Python supports clean architecture through:

    • Domain Layer: Core business logic in engine/codeflow_engine/engine.py
    • Application Layer: Use cases in engine/codeflow_engine/actions/
    • Infrastructure Layer: Integrations in engine/codeflow_engine/integrations/
    • Presentation Layer: APIs in engine/codeflow_engine/server.py
    • Plugin System: Extensibility via codeflow_engine.actions namespace
    • Type Safety: Pydantic models enforce contracts

  5. Operational Simplicity: Single runtime reduces:

    • Container image size
    • Deployment complexity
    • Monitoring overhead
    • Security surface area

When Python-Only May Not Be Sufficient

We acknowledge Python-only has limitations. Future scenarios that might require multi-language:

  1. Performance Bottlenecks: If profiling reveals CPU-bound hotspots consuming >30% resources
  2. Real-time Requirements: If latency requirements drop below 50ms p99
  3. Memory Constraints: If memory usage becomes problematic at scale
  4. Concurrency Needs: If need for true parallel CPU work (not I/O) emerges

In these cases, hybrid approaches (Python + Rust/C#) remain viable migration paths.

Consequences

Positive

  • Simplified Architecture: Single codebase, single deployment pipeline
  • Faster Iteration: No gRPC communication layer needed
  • Better Type Safety: Python 3.12+ with Pydantic v2 provides strong typing
  • Rich AI Ecosystem: Direct access to all Python AI/ML libraries
  • Easier Testing: Single language testing framework
  • Lower Maintenance: Fewer moving parts, simpler debugging

Negative

  • CPU Performance: Lower raw CPU performance compared to C#
  • Memory Usage: Python's memory footprint is higher
  • GIL Limitations: Global Interpreter Lock affects multi-threaded CPU work
  • Startup Time: Slower cold starts compared to compiled languages

Mitigations

  • Use async I/O extensively to avoid GIL bottlenecks
  • Leverage Redis for caching to reduce CPU load
  • Scale horizontally with multiple worker processes
  • Use PyPy or Cython for performance-critical sections if needed

Separation of Concerns in Python Implementation

While using a single language, the architecture maintains clear separation of concerns through:

Layered Architecture

engine/codeflow_engine/
├── engine.py         # Domain Layer (core business logic)
├── actions/          # Application Layer (use cases, workflows)
├── integrations/     # Infrastructure Layer (external services)
├── ai/               # AI/ML Layer (LLM providers, agents)
├── database/         # Data Layer (models, repositories)
├── config/           # Configuration Layer (settings, validation)
├── security/         # Security Layer (auth, validation)
└── server.py         # Presentation Layer (API endpoints)

Domain-Driven Design Principles

  1. Bounded Contexts: Clear module boundaries

    • actions.* - Workflow automation context
    • integrations.* - External service context
    • ai.* - AI/ML processing context
    • security.* - Security and auth context

  2. Dependency Inversion: Abstractions over implementations

    # Abstract base in domain layer
class LLMProvider(ABC):
    @abstractmethod
    async def generate(self, prompt: str) -> str: ...

# Concrete implementations in infrastructure layer
class OpenAIProvider(LLMProvider): ...
class AnthropicProvider(LLMProvider): ...

  3. Plugin Architecture: Extensibility without core changes

    [tool.poetry.plugins."codeflow.actions"]
"platform_detector" = "codeflow_engine.actions.platform_detector:PlatformDetector"

  4. Repository Pattern: Data access abstraction

    class WorkflowRepository:
    async def save(self, workflow: Workflow) -> None: ...
    async def find_by_id(self, id: str) -> Workflow: ...

  5. Service Layer: Business logic orchestration

    class WorkflowService:
    def __init__(self, repo: WorkflowRepository, llm: LLMProvider):
        self._repo = repo
        self._llm = llm

Module Independence

  • Each module can be tested independently
  • Clear interfaces (Pydantic models) between layers
  • Minimal coupling through dependency injection
  • Type safety enforced at module boundaries

Microservices-Ready

While monolithic, the architecture supports future microservices extraction:

  • Each bounded context could become a service
  • FastAPI routers already provide API boundaries
  • Database schema separates concerns (workflow, auth, audit)
  • Event-driven patterns in place for async processing

This demonstrates that separation of concerns is achieved through software design patterns, not just language choice.

Migration Summary from Proposed Architecture

What Changed from ADR-0001

Proposed (ADR-0001) Actual Implementation Rationale
C# + Python hybrid Python-only Simpler, faster development
gRPC communication Direct Python calls No cross-language overhead
.NET 6+ for core FastAPI + Flask Python web frameworks sufficient
Separate C# service Monolithic Python app Easier deployment and debugging

Migration Path (If Needed)

If performance becomes a critical issue, migration options include:

Option A: Gradual Optimization (Recommended First Step)

  1. Profile First: Use cProfile, py-spy to identify actual bottlenecks
  2. Optimize Hot Paths:
    • Cython for CPU-intensive Python code
    • PyPy for JIT compilation benefits
    • Numba for numerical computations
  3. Strategic Caching: Add Redis caching to frequently accessed data
  4. Database Optimization: Query optimization, indexing, connection pooling
  5. Async Improvements: Ensure proper async/await usage throughout

Expected Gains: 2-5x performance improvement with minimal architectural changes

Option B: Rust Extensions via PyO3 (Recommended for CPU-bound Work)

  1. Identify Candidates: Profile to find CPU-bound bottlenecks
  2. Extract Module: Create isolated Rust crate for specific function
  3. PyO3 Integration: Build Python bindings using PyO3/maturin
  4. Incremental Migration: Replace Python module with Rust extension
  5. Examples of Good Candidates:
    • Large file parsing (YAML, JSON processing)
    • Data transformation pipelines
    • Text processing/tokenization
    • Compression/decompression
    • Cryptographic operations

Example Structure:

engine/codeflow_engine/
├── core/                    # Python code
├── extensions/
│   └── fast_parser/        # Rust extension
│       ├── Cargo.toml
│       ├── src/lib.rs      # Rust code
│       └── pyproject.toml  # maturin config

Expected Gains: 10-50x on CPU-bound operations, maintains Python ecosystem benefits

Why Rust over C#/Go:

  • Zero-cost abstractions: Performance matches C/C++
  • Memory safety: No garbage collection, no null pointers
  • Python integration: PyO3 provides excellent ergonomics
  • Growing adoption: Used by Polars, Pydantic v2 core, Ruff linter
  • Future-proof: Mozilla, AWS, Microsoft investing heavily

Option C: C# Microservices (For Complex Business Logic)

  1. Extract Service: Identify service boundary (e.g., workflow engine)
  2. Define API Contract: gRPC or REST interface
  3. Implement in C#: ASP.NET Core service
  4. Dual Deployment: Run alongside Python services
  5. Gradual Migration: Move components incrementally

Best For: When you need entire service rewrite, not just performance optimization

Expected Gains: 5-10x performance, but adds operational complexity

Option D: Full Microservices (Multiple Languages)

  1. Service Extraction: Break monolith into 3-5 core services
  2. Language Selection: Choose best tool per service
    • Python: AI/ML processing, orchestration
    • Rust: High-throughput data processing
    • C#/Go: Business logic APIs
  3. Communication: gRPC for inter-service, REST for external
  4. Independent Deployment: Each service scales independently

Best For: 10x+ scale, clear team specialization

Expected Gains: Horizontal scalability, technology flexibility, but significant complexity

Decision Tree for Migration

Is performance actually a problem? (Profile first!)
├── No → Stay with Python-only
└── Yes → Is it CPU-bound or I/O-bound?
    ├── I/O-bound → Optimize async, add caching, horizontal scaling
    └── CPU-bound → Profile specific functions
        ├── <5% of codebase hot → Use Rust via PyO3
        ├── Entire module/service → Consider C# microservice
        └── Multiple services → Evaluate polyglot architecture

Rust Migration Example

For context, here's how a Rust extension might work:

Python interface (engine/codeflow_engine/parsers/fast.py):

from .extensions.fast_parser import parse_yaml_fast

def parse_workflow(content: str) -> dict:
    """Parse workflow YAML with Rust performance."""
    return parse_yaml_fast(content)

Rust implementation (extensions/fast_parser/src/lib.rs):

use pyo3::prelude::*;
use serde_yaml;

#[pyfunction]
fn parse_yaml_fast(content: &str) -> PyResult<PyObject> {
    let parsed: serde_yaml::Value = serde_yaml::from_str(content)?;
    // Convert to Python dict
    Ok(parsed.into_py(py))
}

#[pymodule]
fn fast_parser(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_yaml_fast, m)?)?;
    Ok(())
}

This approach maintains Python's developer experience while adding Rust's performance where needed.

Related Decisions

References

Python Resources

Alternative Language Resources

Architecture Patterns

Real-World Examples