Merge pull request #25 from nadeem4/feature/enh-001-sandboxed-execution

feat(core): Implement Sandboxed Execution (ENH-001)

Author: nadeem4

README.md

@@ -1,101 +1,100 @@
-# NL2SQL Platform
+# Enterprise NL2SQL Engine
 
-An enterprise-grade **Natural Language to SQL** engine built on an Agentic Graph Architecture.
+> **A Production-Grade Natural Language to SQL Engine built on the principles of Zero Trust and Deterministic Execution.**
 
-## 🚀 Overview
+This platform treats "Text-to-SQL" not as a prompt engineering problem, but as a **Distributed Systems** problem. It replaces fragile one-shot generation with a robust, compiled pipeline that bridges the gap between Unstructured Intention (User Language) and Structured Execution (SQL Databases).
 
-This platform transforms complex natural language questions into safe, optimized, and executable SQL queries across multiple database engines (PostgreSQL, MySQL, MSSQL, SQLite). It uses a **Directed Cyclic Graph** (LangGraph) to orchestrate planning, validation, generation, and self-correction.
+---
 
-### Key Features
+## 🏗️ System Topology
 
-* **🛡️ Security First**: Strict AST Validation, **Intent Analysis** (Jailbreak Detection), RBAC Policies, and Read-Only enforcement.
-* **🧠 Agentic Reasoning**: Self-correcting nodes that fix SQL errors automatically.
-* **🔌 Polyglot**: First-class support for Postgres, MySQL, MSSQL, and SQLite.
-* **⚡ Smart Routing**: Decomposes complex queries into sub-queries for multi-datasource environments.
-* **🔄 Reliability**: Built-in **Exponential Backoff** and **Circuit Breakers** to handle transient failures gracefully.
+The architecture is composed of three distinct planes, ensuring separation of concerns and failure isolation.
 
-## 🏁 Quick Demo
+### 1. The Control Plane (The Graph)
 
-Explore the platform's capabilities with our interactive setup wizard. You can choose between **Lite Mode** (in-memory, no deps) or **Docker Mode** (real databases).
+**Responsibility**: Reasoning, Planning, and Orchestration.
 
-### 1. Lite Mode (Fastest) uses SQLite
+* **Agentic Graph**: Implemented as a Directed Cyclic Graph (LangGraph) to enable "Refinement Loops". If a plan fails validation, the system self-corrects.
+* **State Management**: Deterministic state transitions ensure auditability and reproducibility of every decision.
 
-Perfect for a standardized, local environment without needing Docker.
+### 2. The Security Plane (The Firewall)
 
-```bash
-nl2sql setup --demo
-```
+**Responsibility**: Invariants Enforcement.
 
-### 2. Docker Mode (Full Fidelity) uses Postgres
+* **Valid-by-Construction**: The LLM *never* executes SQL directly. It generates an **Abstract Syntax Tree (AST)**.
+* **Static Analysis**: The [Validator Node](docs/core/nodes.md#4-logical-validator) enforces **Row-Level Security (RLS)** and type safety on the AST *before* compilation.
+* **Intent Classification**: Upstream detection of adversarial prompts (Jailbreaks/Injections).
 
-Spins up real orchestrator and database containers for a production-like test.
+### 3. The Data Plane (The Sandbox)
 
-```bash
-nl2sql setup --demo --docker
-```
+**Responsibility**: Semantic Search and Execution.
 
-## 🛠️ Installation
+* **Blast Radius Isolation**: SQL Drivers (ODBC/C-Ext) run in a dedicated **[Sandboxed Process Pool](docs/architecture/decisions/ADR-001_sandboxed_execution.md)**. A segfault in a driver kills a disposable worker, not the Agent.
+* **Partitioned Retrieval**: The [Orchestrator](docs/core/indexing.md) uses Partitioned MMR to inject only relevant schema context, preventing context window overflow.
 
-This is a monorepo. To develop or run the platform from source:
+---
 
-### Prerequisites
-
-* Python 3.10+
-* Docker & Docker Compose (optional, for Integration Tests)
+## 📐 Architectural Invariants
 
-### Setup
+| Invariant | Rationale | Mechanism |
+| :--- | :--- | :--- |
+| **No Unvalidated SQL** | Prevent Hullucinations & Data Leaks | All plans pass through `LogicalValidator` (AST) + `PhysicalValidator` (Dry Run) before execution. |
+| **Zero Shared State** | Crash Safety | Execution happens in isolated processes; no shared memory with the Control Plane. |
+| **Fail-Fast** | Reliability | Circuit Breakers and Strict Timeouts prevent cascading failures (Retry Storms). |
+| **Determinism** | Debuggability | Temperature-0 generation + Strict Typing (Pydantic) for all LLM outputs. |
 
-1. **Clone and Install**:
+---
 
-    ```bash
-    git clone https://github.com/nadeem4/nl2sql.git
-    cd nl2sql
-    
-    # Create virtual environment
-    python -m venv venv
-    source venv/bin/activate  # or .\venv\Scripts\activate on Windows
-    
-    # Install Core and CLI
-    pip install -e packages/core
-    pip install -e packages/adapter-sdk
-    pip install -e packages/cli
-    pip install -e packages/adapters/postgres  # Install specific adapters as needed
-    ```
+## 🚀 Quick Start
 
-2. **Verify Installation**:
+### Prerequisites
 
-    ```bash
-    nl2sql --help
-    ```
+* Python 3.10+
+* Docker (Optional, for full integration environment)
 
-## 🏗️ Architecture
+### 1. Installation
 
-The system is composed of specialized Neural Nodes:
+```bash
+git clone https://github.com/nadeem4/nl2sql.git
+cd nl2sql
 
-1. **Semantic Analysis**: Intent classification and entity extraction.
-2. **Decomposer (Router)**: Splits complex queries and routes them to the correct datasource.
-3. **Planner**: Generates a database-agnostic Abstract Syntax Tree (AST).
-4. **Validator**: Enforces security policies and logical correctness on the AST.
-5. **Generator**: Compiles AST to dialect-specific SQL.
-6. **Executor**: Runs the query in a sandboxed environment.
-7. **Refiner**: Self-corrects errors by analyzing stack traces and feedback.
-8. **Aggregator**: Synthesizes results from multiple sub-queries.
+# Set up environment
+python -m venv venv
+source venv/bin/activate
 
-See [Architecture Documentation](docs/core/architecture.md) for details.
+# Install Core Engine & CLI
+pip install -e packages/core
+pip install -e packages/cli
+pip install -e packages/adapter-sdk
+```
 
-## 📚 Documentation
+### 2. Run Demo (Lite Mode)
 
-Full documentation is available in the `docs/` directory.
+Boot the engine with an in-memory SQLite database (No Docker required).
 
 ```bash
-pip install -r requirements-docs.txt
-mkdocs serve
+nl2sql setup --demo
 ```
 
-## 📂 Repository Structure
+---
+
+## 📚 Technical Documentation
 
-* `packages/core`: The core graph engine, nodes, and state management.
-* `packages/cli`: Command-line interface tool.
-* `packages/adapter-sdk`: SDK for building custom database adapters.
-* `configs/`: Configuration files (Policies, Datasources).
-* `docs/`: MkDocs source files.
+* **[System Architecture](docs/core/architecture.md)**: Deep dive into the Control, Security, and Data planes.
+* **[Component Reference](docs/core/nodes.md)**: Detailed specs for Planner, Validator, Executor, etc.
+* **[Security Model](docs/safety/security.md)**: Defense-in-depth strategy against prompt injection and unauthorized access.
+* **[ADR-001: Sandboxed Execution](docs/architecture/decisions/ADR-001_sandboxed_execution.md)**: Decision record for the Process Pool architecture.
+
+---
+
+## 📦 Repository Structure
+
+```text
+packages/
+├── core/               # The Engine (Graph, State, Logic)
+├── cli/                # Terminal Interface & Ops Tools
+├── adapter-sdk/        # Interface Contract for new Databases
+└── adapters/           # Official Dialects (Postgres, MSSQL, MySQL)
+configs/                # Runtime Configuration (Policies, Prompts)
+docs/                   # Architecture & Operations Manual
+```

docs/architecture/decisions/ADR-001_sandboxed_execution.md

@@ -98,12 +98,12 @@ class ExecutionResponse(BaseModel):
 
 ## 6. Recommendation
 
-**Phase 1: Option A (Multiprocessing)**
+### Phase 1: Option A (Multiprocessing)
 
 * It solves the critical **Blast Radius** immediately.
 * It separates **Indexing** from **Execution** via distinct pools.
 * It is "Service-Ready": The `ExecutionRequest` protocol makes migrating to Option B trivial later.
 
-**Phase 2: Option B (Dedicated Service)**
+### Phase 2: Option B (Dedicated Service)
 
 * Migrate only when we need to scale Execution independently of the Agent or require strict network segmentation.

docs/core/architecture.md

@@ -74,3 +74,4 @@ The platform is designed to be cloud-agnostic and operationally robust.
 
 * **[Environments](environment.md)**: Configuration management via `.env` files.
 * **[Adapters](../adapters/index.md)**: The connectivity layer that abstracts specific database dialects (Postgres, MSSQL, etc.).
+* **[Sandboxed Execution](../architecture/decisions/ADR-001_sandboxed_execution.md)**: Isolated process pools for executing untrusted SQL and performing heavy schema introspection.

docs/core/indexing.md

@@ -10,6 +10,7 @@ We index two main types of documents: **Schema** and **Examples**.
 
 Table schemas are "flattened" into a rich text representation before embedding.
 
+* **Sandboxed Introspection**: Schema fetching is offloaded to the **Indexing Process Pool** ensuring that connecting to slow/unstable databases does not block the Orchestrator's main event loop.
 * **Format**: `Table: {name} (Alias: {alias}). Columns: {col_desc}. Primary Key: {pk}. Foreign Keys: {fk}.`
 * **Offline Aliasing**: To prevent conflicts, tables are assigned canonical aliases (e.g., `sales_db_t1`) during indexing.
 * **Rich Metadata (via Adapter)**:

docs/core/nodes.md

@@ -45,14 +45,14 @@ The pipeline is composed of specialized **Nodes**. Each node performs a single r
 
 **Responsibility**: Execution Readiness.
 
-* **Dry Run**: Executes `EXPLAIN` or equivalent to verify syntax validity without running the query.
-* **Performance**: Checks row cost estimates against `row_limit`.
+* **Dry Run (Sandboxed)**: Executes `EXPLAIN` or equivalent inside the **Execution Sandbox** to verify syntax validity without crashing the main process.
+* **Performance (Sandboxed)**: Checks row cost estimates against `row_limit` in the sandbox.
 
 ## 7. Executor Node
 
 **Responsibility**: Sandboxed Execution.
 
-* **Logic**: Connects to the database using the specific Adapter and executes the query.
+* **Logic**: Offloads the query execution to the **Execution Process Pool** (`nl2sql.common.sandbox`). This isolates the main application from driver segfaults (e.g., OOMs, C-extension panics).
 * **Safety**: Read-only connection enforcement.
 
 ## 8. Refiner Node

packages/adapter-sqlalchemy/src/nl2sql_sqlalchemy_adapter/adapter.py

@@ -40,6 +40,7 @@ def __init__(self, datasource_id: str = None, datasource_engine_type: str = None
         if self.statement_timeout_ms:
             self.execution_options["timeout"] = self.statement_timeout_ms / 1000.0
 
+        self.connection_args = connection_args
         self.connection_string = self.construct_uri(connection_args)
 
         self.engine: Engine = None

packages/core/src/nl2sql/common/sandbox.py

@@ -0,0 +1,92 @@
+"""
+Sandbox Manager for isolating unsafe operations (SQL Execution, Indexing).
+
+This module manages two separate ProcessPoolExecutors:
+1. Execution Pool: Low-latency, for user-facing query execution.
+2. Indexing Pool: High-throughput, for background schema fetching.
+
+It implements the Singleton pattern to ensure pools are reused across requests.
+"""
+from __future__ import annotations
+
+import atexit
+from concurrent.futures import ProcessPoolExecutor
+from typing import Optional
+
+from nl2sql.common.logger import get_logger
+from nl2sql.common.settings import settings
+
+logger = get_logger("sandbox_manager")
+
+class SandboxManager:
+    """Manages global process pools for sandboxed execution."""
+    
+    _exec_pool: Optional[ProcessPoolExecutor] = None
+    _index_pool: Optional[ProcessPoolExecutor] = None
+
+    @classmethod
+    def get_execution_pool(cls) -> ProcessPoolExecutor:
+        """Returns the process pool for latency-sensitive execution tasks."""
+        if cls._exec_pool is None:
+            workers = settings.sandbox_exec_workers
+            logger.info(f"Initializing Execution Sandbox with {workers} workers.")
+            cls._exec_pool = ProcessPoolExecutor(
+                max_workers=workers,
+                initializer=cls._init_worker,
+                initargs=("execution",)
+            )
+        return cls._exec_pool
+
+    @classmethod
+    def get_indexing_pool(cls) -> ProcessPoolExecutor:
+        """Returns the process pool for throughput-heavy indexing tasks."""
+        if cls._index_pool is None:
+            workers = settings.sandbox_index_workers
+            logger.info(f"Initializing Indexing Sandbox with {workers} workers.")
+            cls._index_pool = ProcessPoolExecutor(
+                max_workers=workers,
+                initializer=cls._init_worker,
+                initargs=("indexing",)
+            )
+        return cls._index_pool
+
+    @staticmethod
+    def _init_worker(pool_type: str):
+        """Initializer run once per worker process startup.
+        
+        Sets a distinct process name for debugging and prepares the worker execution environment.
+        """
+        import multiprocessing
+        p = multiprocessing.current_process()
+        p.name = f"SandboxWorker-{pool_type}-{p.name}"
+
+    @classmethod
+    def shutdown(cls):
+        """Shuts down all sandbox pools waiting for pending tasks to complete."""
+        if cls._exec_pool:
+            logger.info("Shutting down Execution Sandbox...")
+            cls._exec_pool.shutdown(wait=True)
+            cls._exec_pool = None
+            
+        if cls._index_pool:
+            logger.info("Shutting down Indexing Sandbox...")
+            cls._index_pool.shutdown(wait=True)
+            cls._index_pool = None
+
+atexit.register(SandboxManager.shutdown)
+
+def get_execution_pool() -> ProcessPoolExecutor:
+    """Helper accessor for execution pool.
+
+    Returns:
+        ProcessPoolExecutor: The shared execution pool instance.
+    """
+    return SandboxManager.get_execution_pool()
+
+def get_indexing_pool() -> ProcessPoolExecutor:
+    """Helper accessor for indexing pool.
+
+    Returns:
+        ProcessPoolExecutor: The shared indexing pool instance.
+    """
+    return SandboxManager.get_indexing_pool()

packages/core/src/nl2sql/common/settings.py

@@ -45,6 +45,18 @@ class Settings(BaseSettings):
         description="Global timeout in seconds for pipeline execution."
     )
 
+    sandbox_exec_workers: int = Field(
+        default=4,
+        validation_alias="SANDBOX_EXEC_WORKERS",
+        description="Max workers for latency-sensitive execution pool."
+    )
+
+    sandbox_index_workers: int = Field(
+        default=2,
+        validation_alias="SANDBOX_INDEX_WORKERS",
+        description="Max workers for throughput-heavy indexing pool."
+    )
+
     model_config = SettingsConfigDict(
         env_file=".env", 
         env_file_encoding="utf-8",