feat: Introduce Query, Result, and Settings APIs for NL2SQL

- Added QueryAPI for executing natural language queries against databases.
- Implemented ResultAPI for managing and storing query results.
- Created SettingsAPI for configuration and settings management.
- Removed obsolete ResultStore and SandboxManager classes.
- Refactored PhysicalValidatorNode to streamline validation processes.
- Established a public API for NL2SQL core functionalities, enhancing modular access.
- Updated tests to reflect changes in the API structure and removed deprecated tests.

Author: nadeem4

docs/architecture/nodes/physical_validator_node.md

@@ -1,160 +0,0 @@
-# PhysicalValidatorNode
-
-## Overview
-
-- Performs dry‑run and cost‑estimate checks on generated SQL using sandboxed execution.
-- Exists to validate executability and performance limits before execution.
-- Not wired in the default SQL agent graph.
-- Class: `PhysicalValidatorNode`
-- Source: `packages/core/src/nl2sql/pipeline/nodes/validator/physical_node.py`
-
----
-
-## Responsibilities
-
-- Run dry‑run validation via adapter `dry_run()`.
-- Run cost estimation via adapter `cost_estimate()`.
-- Use sandboxed execution pool and `DB_BREAKER` for isolation and resilience.
-
----
-
-## Position in Execution Graph
-
-Upstream:
-- None (not connected in current graph).
-
-Downstream:
-- None (not connected in current graph).
-
-Trigger conditions:
-- Not executed in current pipeline; requires graph wiring.
-
-```mermaid
-flowchart LR
-    PhysicalValidator[PhysicalValidatorNode]
-```
-
----
-
-## Inputs
-
-From `SubgraphExecutionState`:
-
-- `generator_response.sql_draft` (required)
-- `sub_query.datasource_id` (required)
-
-From `NL2SQLContext`:
-
-- `ds_registry` (adapter resolution)
-
-Validation performed:
-
-- If `sql` is missing, returns empty response.
-- If `datasource_id` missing, emits `MISSING_DATASOURCE_ID`.
-
----
-
-## Outputs
-
-Mutations to `SubgraphExecutionState`:
-
-- `physical_validator_response` (`PhysicalValidatorResponse`)
-- `errors` and `reasoning`
-
-Side effects:
-
-- Sandbox execution pool usage via `execute_in_sandbox`.
-- Adapter calls to `dry_run()` and `cost_estimate()`.
-
----
-
-## Internal Flow (Step-by-Step)
-
-1. If SQL is missing, return empty response.
-2. Resolve datasource adapter via registry.
-3. `_validate_semantic()` runs dry‑run in sandbox under `DB_BREAKER`.
-4. `_validate_performance()` runs cost estimate in sandbox under `DB_BREAKER`.
-5. Collect errors/warnings and return `PhysicalValidatorResponse`.
-6. On exceptions, emit `PHYSICAL_VALIDATOR_FAILED`.
-
----
-
-## Contracts & Interfaces
-
-Implements a LangGraph node callable:
-
-```
-def __call__(self, state: SubgraphExecutionState) -> Dict[str, Any]
-```
-
-Key contracts:
-
-- `PhysicalValidatorResponse`
-- `ExecutionRequest` / `ExecutionResult`
-
----
-
-## Determinism Guarantees
-
-- Deterministic for a fixed SQL and adapter behavior.
-- External DB behavior can vary across runs.
-
----
-
-## Error Handling
-
-Emits `PipelineError` with:
-
-- `MISSING_DATASOURCE_ID`
-- `EXECUTION_ERROR`
-- `EXECUTOR_CRASH`
-- `PERFORMANCE_WARNING`
-- `SERVICE_UNAVAILABLE`
-- `PHYSICAL_VALIDATOR_FAILED`
-
----
-
-## Retry + Idempotency
-
-- No internal retry logic beyond circuit breaker behavior.
-- Idempotency depends on adapter dry‑run semantics.
-
----
-
-## Performance Characteristics
-
-- Uses process pool execution (sandbox).
-- Cost estimation and dry‑run are external DB calls.
-
----
-
-## Observability
-
-- Logger: `physical_validator`
-- Uses `DB_BREAKER` which logs breaker state changes.
-
----
-
-## Configuration
-
-- None directly; uses sandbox pool sizing from settings and adapter config.
-
----
-
-## Extension Points
-
-- Wire into `build_sql_agent_graph()` between generator and executor.
-- Extend dry‑run and cost estimation logic per adapter.
-
----
-
-## Known Limitations
-
-- Not connected to the default SQL agent graph.
-- Behavior depends on adapter support for dry‑run and cost estimation.
-
----
-
-## Related Code
-
-- `packages/core/src/nl2sql/pipeline/nodes/validator/physical_node.py`

docs/execution/sandbox.md

@@ -1,53 +0,0 @@
-# Execution Sandbox Architecture
-
-NL2SQL includes a sandbox subsystem that provides **process-level isolation** for execution and indexing tasks. The sandbox is implemented via `ProcessPoolExecutor` pools managed by `SandboxManager`.
-
-## Sandbox structure (available)
-
-```mermaid
-flowchart TD
-    Caller[Execution/Indexing Task] --> Sandbox[get_execution_pool()/get_indexing_pool()]
-    Sandbox --> Worker[ProcessPool Worker]
-    Worker --> Adapter[DatasourceAdapterProtocol.execute()]
-    Adapter --> Database[(Database)]
-```
-
-## Current wiring (as implemented)
-
-- `SandboxManager` and `execute_in_sandbox()` exist and are production-ready.
-- The default `SqlExecutorService` currently executes **in-process** and does not call `execute_in_sandbox()`.
-- Circuit breakers are available in `nl2sql.common.resilience`, but `SqlExecutorService` does not wrap calls with `DB_BREAKER` today.
-
-This means execution isolation is **available but not enforced** by default in SQL execution.
-
-## Concurrency model
-
-- `run_with_graph()` executes the control graph within a `ThreadPoolExecutor`.
-- Sandbox pools are **process-based** and designed for isolation and crash containment.
-- Execution and indexing pools are configured via `Settings.sandbox_exec_workers` and `Settings.sandbox_index_workers`.
-
-## Sandbox APIs
-
-- `SandboxManager.get_execution_pool()` for latency-sensitive execution.
-- `SandboxManager.get_indexing_pool()` for throughput-heavy indexing.
-- `execute_in_sandbox()` for timeouts and crash handling.
-
-## Failure handling semantics
-
-`execute_in_sandbox()` returns an `ExecutionResult` that captures:
-
-- timeouts (worker hung)
-- worker crashes (segfault/OOM)
-- serialization or runtime errors
-
-## Retry behavior
-
-Retry behavior is not defined at the sandbox layer. Retries are controlled by:
-
-- SQL agent retry loop (`sql_agent_max_retries`)
-- Circuit breaker configuration (fail-fast)
-
-## Source references
-
-- Sandbox manager: `packages/core/src/nl2sql/common/sandbox.py`
-- SQL executor: `packages/core/src/nl2sql/execution/executor/sql_executor.py`

packages/adapter-sdk/src/nl2sql_adapter_sdk/protocols.py

@@ -34,3 +34,8 @@ def execute(self, request: AdapterRequest) -> ResultFrame:
     def get_dialect(self) -> str:
         """Return the normalized dialect string (SQL adapters)."""
         ...
+
+
+    def test_connection(self) -> bool:
+        """Test if the connection to the datasource can be established."""
+        ...

packages/adapter-sqlalchemy/src/nl2sql_sqlalchemy_adapter/adapter.py

@@ -460,3 +460,16 @@ def get_dialect(self) -> str:
 
     def cost_estimate(self, sql: str) -> CostEstimate:
         raise NotImplementedError(f"Adapter {self.__class__.__name__} must implement cost_estimate")
+    
+
+    def test_connection(self) -> bool:
+        """
+        Tests the database connection by executing a simple query.
+        """
+        try:
+            with self.engine.connect() as conn:
+                conn.execute(text("SELECT 1"))
+            return True
+        except Exception as e:
+            logger.error(f"Connection test failed for {self}: {e}")
+            return False

packages/api/API_DOCS.md

@@ -0,0 +1,99 @@
+# NL2SQL API Documentation
+
+The NL2SQL API provides a REST interface to the NL2SQL engine, allowing external clients (such as the TypeScript CLI) to interact with the system.
+
+## Available Endpoints
+
+### Query Endpoint
+- **URL**: `/api/v1/query`
+- **Method**: `POST`
+- **Description**: Execute a natural language query against a database
+- **Request Body**:
+  ```json
+  {
+    "natural_language": "Show top 10 customers by revenue",
+    "datasource_id": "optional_datasource_id",
+    "execute": true,
+    "user_context": {
+      "user_id": "user123",
+      "permissions": ["read_customers", "read_orders"]
+    }
+  }
+  ```
+- **Response**:
+  ```json
+  {
+    "sql": "SELECT customer_name, revenue FROM customers ORDER BY revenue DESC LIMIT 10",
+    "results": [...],
+    "final_answer": "Here are the top 10 customers by revenue...",
+    "errors": [],
+    "trace_id": "unique_trace_id",
+    "reasoning": [...],
+    "warnings": [...]
+  }
+  ```
+
+### Schema Endpoints
+- **URL**: `/api/v1/schema/{datasource_id}`
+- **Method**: `GET`
+- **Description**: Get schema information for a specific datasource
+- **Response**:
+  ```json
+  {
+    "datasource_id": "my_database",
+    "tables": [...],
+    "relationships": [...],
+    "metadata": {...}
+  }
+  ```
+
+- **URL**: `/api/v1/schema`
+- **Method**: `GET`
+- **Description**: List all available datasources
+- **Response**: `["datasource1", "datasource2", ...]`
+
+### Health Check Endpoints
+- **URL**: `/api/v1/health`
+- **Method**: `GET`
+- **Description**: Check if the API is running
+- **Response**:
+  ```json
+  {
+    "success": true,
+    "message": "NL2SQL API is running"
+  }
+  ```
+
+- **URL**: `/api/v1/ready`
+- **Method**: `GET`
+- **Description**: Check if the API is ready to serve requests
+- **Response**:
+  ```json
+  {
+    "success": true,
+    "message": "NL2SQL API is ready"
+  }
+  ```
+
+## Running the API Server
+
+To start the API server:
+
+```bash
+nl2sql-api --host 0.0.0.0 --port 8000 --reload
+```
+
+Or using uvicorn directly:
+
+```bash
+uvicorn nl2sql_api.main:app --host 0.0.0.0 --port 8000 --reload
+```
+
+## Configuration
+
+The API relies on the same configuration files as the core NL2SQL engine:
+- `configs/datasources.yaml` - Database connection configurations
+- `configs/llm.yaml` - LLM provider configurations
+- `configs/secrets.yaml` - Secret management configurations
+
+Make sure these files are properly configured before starting the API server.

packages/api/README.md

@@ -0,0 +1,51 @@
+# NL2SQL API
+
+API layer for the NL2SQL engine that provides a REST interface to the core functionality.
+
+## Overview
+
+This package provides a FastAPI-based REST API that uses the NL2SQL core's public API to interact with the NL2SQL engine over HTTP. It serves as a bridge between external clients (such as the TypeScript CLI) and the core engine functionality.
+
+## Architecture
+
+The API package leverages the NL2SQL core's public API layer (`NL2SQL` class), ensuring clean separation between the API service and the core engine implementation. The service layer uses the core's public methods like `run_query()`, `list_datasources()`, and schema access through `engine.context.schema_store`.
+
+## Features
+
+- RESTful API endpoints for natural language to SQL conversion
+- Schema introspection capabilities
+- Health and readiness checks
+- Proper error handling and response formatting
+- Lazy initialization to avoid configuration issues during import
+- Integration with the core's public API layer
+
+## Endpoints
+
+- `POST /api/v1/query` - Execute a natural language query
+- `GET /api/v1/schema/{datasource_id}` - Get schema for a specific datasource
+- `GET /api/v1/schema` - List available datasources
+- `GET /api/v1/health` - Health check endpoint
+- `GET /api/v1/ready` - Readiness check endpoint
+
+## Running the API
+
+```bash
+pip install -e .
+nl2sql-api --host 0.0.0.0 --port 8000 --reload
+```
+
+Or using uvicorn directly:
+
+```bash
+uvicorn nl2sql_api.main:app --host 0.0.0.0 --port 8000 --reload
+```
+
+## Development
+
+Install in development mode:
+
+```bash
+pip install -e .
+```
+
+For detailed API documentation, see [API_DOCS.md](API_DOCS.md).