Merge pull request #24 from nadeem4/doc

feat: enhance docs

Author: nadeem4

docs/adapters/development.md

@@ -1,120 +1,72 @@
-# Building Adapters
+# Building Adapters Guide
 
-The **Adapter SDK** (`nl2sql-adapter-sdk`) allows you to extend the platform to support new databases or APIs.
+The NL2SQL Platform is designed to be extensible. You can build adapters for any datasource, from SQL databases to REST APIs.
 
-## Implementing an Adapter
+## Implementation Path
 
-You must implement the `DatasourceAdapter` interface.
+There are two primary ways to build an adapter. Choose the one that fits your target:
 
-### Mandatory Properties
-
-* `datasource_id`: Unique identifier (e.g. "postgres_prod").
-* `row_limit`: **Safety Breaker**. Must return `1000` (or config value) to prevent massive result sets.
-* `max_bytes`: **Safety Breaker**. limit result size at the network/driver level if possible.
-
-### Mandatory Methods
-
-* `fetch_schema()`: Must return `SchemaMetadata` with `tables`, `columns`, `pks`, `fks`. *Crucially, it should also populate `col.statistics` (samples, min/max) for Indexing.*
-* `execute(sql)`: Returns `QueryResult`.
-* `dry_run(sql)`: Returns validity checks.
-
-### Optional Optimization
-
-* `explain(sql)`: Returns query plan.
-* `cost_estimate(sql)`: Returns estimated rows/time. used by PhysicalValidator.
+| If you are checking... | Use... | Reference |
+| :--- | :--- | :--- |
+| A standard SQL Database (Postgres, Oracle, Snowflake) | `nl2sql-adapter-sqlalchemy` | **[SQLAlchemy Adapter Reference](sqlalchemy.md)** |
+| A NoSQL DB, REST API, or custom driver | `nl2sql-adapter-sdk` | **[Adapter SDK Reference](sdk.md)** |
 
-::: nl2sql_adapter_sdk.interfaces.DatasourceAdapter
+## Option 1: The "Fast Lane" (SQLAlchemy)
 
-## Compliance Testing
+For 95% of use cases, you are connecting to a SQL database that already has a Python SQLAlchemy dialect.
 
-The SDK provides a compliance test suite. **All Adapters MUST pass this suite.**
+**Use `BaseSQLAlchemyAdapter`**. It handles:
 
-It verifies:
+* Automatic Schema Introspection (Tables, PKs, FKs)
+* Connection Pooling
+* Statistic Gathering
+* Transaction-based Dry Runs
 
-* Schema Introspection (PKs/FKs detected?)
-* Type Mapping (Date -> Python Date, Numeric -> Python Float)
-* Error Handling (Bad SQL -> AdapterError)
+### Example
 
 ```python
-# tests/test_my_adapter.py
-from nl2sql_adapter_sdk.testing import BaseAdapterTest
-from my_adapter import MyAdapter
+from nl2sql_sqlalchemy_adapter import BaseSQLAlchemyAdapter
 
-class TestMyAdapter(BaseAdapterTest):
-    @pytest.fixture
-    def adapter(self):
-        return MyAdapter(...)
+class PostgresAdapter(BaseSQLAlchemyAdapter):
+    def construct_uri(self, args: Dict[str, Any]) -> str:
+        # Convert args to connection string
+        return f"postgresql://{args['user']}:{args['password']}@{args['host']}/{args['database']}"
 ```
 
-## Choosing a Base Class
-
-The platform provides two ways to build adapters. Choose the one that fits your target datasource.
-
-| Feature | `DatasourceAdapter` (Base Interface) | `BaseSQLAlchemyAdapter` (Helper Class) |
-| :--- | :--- | :--- |
-| **Package** | `nl2sql-adapter-sdk` | `nl2sql-adapter-sqlalchemy` |
-| **Best For** | REST APIs, NoSQL, GraphQL, Manual SQL Drivers. | SQL Databases with SQLAlchemy dialects (Postgres, Oracle, Snowflake). |
-| **Schema Fetching** | **Manual Implementation Required**. You must map metadata to `SchemaMetadata`. | **Automatic**. Uses `sqlalchemy.inspect` to reflect tables/FKs. |
-| **Execution** | **Manual Implementation Required**. You handle connections, cursors, and types. | **Automatic**. Handles pooling, transactions, and result formatting. |
-| **Stats Gathering** | **Manual**. You write queries to fetch min/max/nulls. | **Automatic**. Runs optimized generic queries for stats. |
-| **Dry Run** | **Manual**. | **Automatic**. Uses transaction rollback pattern. |
-
-### When to use `DatasourceAdapter`?
+> See the **[SQLAlchemy Adapter Reference](sqlalchemy.md)** for full API details.
 
-Use the raw interface when:
+## Option 2: The "Custom" Path (SDK)
 
-1. You are connecting to a non-SQL source (e.g., Elasticsearch, HubSpot API).
-2. You are using a customized internal SQL driver that is not compatible with SQLAlchemy.
-3. You need complete control over the execution lifecycle (e.g. async-only drivers).
+If you need to connect to something else (e.g., ElasticSearch, a CRM API, or a raw SQL driver), you must implement the raw interface.
 
-### When to use `BaseSQLAlchemyAdapter`?
+**Implement `DatasourceAdapter`**. You must manually handle:
 
-Use this helper class when:
+* Fetching and normalizing schema metadata.
+* Executing queries and formatting results.
+* Implementing safety breakers (`row_limit`).
 
-1. There is an existing SQLAlchemy dialect for your database (this covers 95% of SQL databases).
-2. You want to save time on boilerplate (connection pooling, schema reflection).
-3. You want consistent behavior with the core supported adapters.
+### Example
 
-## Building SQL Adapters (The Fast Way)
-
-For SQL databases supported by SQLAlchemy, you should use the `nl2sql-adapter-sqlalchemy` package as described in the comparison above.
+```python
+from nl2sql_adapter_sdk import DatasourceAdapter
 
-### `BaseSQLAlchemyAdapter` Features
+class MyRestAdapter(DatasourceAdapter):
+    def fetch_schema(self) -> SchemaMetadata:
+        # call API, return schema
+        pass
 
-This base class implements ~90% of the required functionality for you:
+    def execute(self, query: str) -> QueryResult:
+        # run query, return rows
+        pass
+```
 
-* **Automatic Schema Fetching**: Uses `sqlalchemy.inspect` to get tables, columns, PKs.
-* **Automatic Statistics**: Runs optimized queries to fetch `min/max`, `null_percentage`, `distinct_count`, and `sample_values` for text columns.
-* **Generic Execution**: Handles connection pooling and result formatting.
-* **Safety**: Built-in generic `dry_run` using transaction rollbacks.
+> See the **[Adapter SDK Reference](sdk.md)** for the mandatory method signatures and compliance testing guide.
 
-### Example Implementation
+## Compliance Testing
 
-See `packages/adapters/postgres` for a reference implementation.
+Regardless of which path you choose, your adapter **MUST** pass the compliance test suite to ensuring it handles types and errors correctly.
 
 ```python
-from nl2sql_sqlalchemy_adapter import BaseSQLAlchemyAdapter
-
-class PostgresAdapter(BaseSQLAlchemyAdapter):
-    def construct_uri(self, args: Dict[str, Any]) -> str:
-        return f"postgresql://{args.get('user')}:{args.get('password')}@{args.get('host')}/{args.get('database')}"
-    
-    # Optional: Override dry_run for better performance using EXPLAIN
-    def dry_run(self, sql: str):
-        self.execute(f"EXPLAIN {sql}")
-        return DryRunResult(is_valid=True)
+from nl2sql_adapter_sdk.testing import BaseAdapterTest
+# ... see SDK Reference for test setup
 ```
-
-## Reference Adapters
-
-For detailed usage configurations of our supported adapters, please see the **[Supported Adapters](index.md)** section.
-
-Explore the `packages/adapters/` directory for examples:
-
-* `postgres`: Standard implementation using `sqlalchemy`.
-* `sqlite`: Simple, file-based.
-* `mssql` / `mysql`: Standard enterprise drivers.
-
-## Next Steps
-
-Check out the [Postgres Adapter Source Code](https://github.com/nadeem4/nl2sql/tree/main/packages/adapters/postgres) for a complete, production-grade example.

docs/adapters/index.md

@@ -13,6 +13,13 @@ We provide first-class support for the following SQL databases via SQLAlchemy.
 | **[Microsoft SQL Server](mssql.md)** | Enterprise support via `pyodbc` and `T-SQL` dialect. | 🟡 Beta |
 | **[SQLite](sqlite.md)** | File-based local development. | 🟢 Stable |
 
+## Core Libraries
+
+For developers building their own adapters, we provide detailed reference documentation for our core SDKs.
+
+* **[Adapter SDK Reference](sdk.md)**: The core interface (`DatasourceAdapter`) that all adapters must implement.
+* **[SQLAlchemy Adapter Reference](sqlalchemy.md)**: The helper base class (`BaseSQLAlchemyAdapter`) for building SQL-based adapters.
+
 ## Missing your database?
 
 Can't find what you need? Check out the **[Building Adapters](development.md)** guide to see how to implement your own.

docs/adapters/mssql.md

@@ -2,8 +2,7 @@
 
 Support for SQL Server 2017+ and Azure SQL.
 
-!!! info "Implementation"
-    This adapter extends `BaseSQLAlchemyAdapter` but provides specialized `dry_run` logic using `SET NOEXEC ON` to safely validate T-SQL.
+This adapter extends `BaseSQLAlchemyAdapter` but provides specialized `dry_run` logic using `SET NOEXEC ON` to safely validate T-SQL.
 
 ## Configuration
 
@@ -35,6 +34,12 @@ connection:
 | **Dry Run** | `SET NOEXEC ON` | Validates syntax without execution. |
 | **Costing** | `SET SHOWPLAN_XML ON` | Parses XML for `StatementSubTreeCost`. |
 
+### Optimization Details
+
+* **Dry Run**: Uses `SET NOEXEC ON`. This is a native T-SQL session setting that compiles the query but ensures it is **not executed**. This is extremely safe and accurate for validation.
+* **Explain**: Uses `SET SHOWPLAN_XML ON` to retrieve the execution plan in XML format.
+* **Cost Estimate**: Parses the XML plan to find `StatementSubTreeCost` (estimated cost) and `StatementEstRows` (estimated rows).
+
 ## Requirements
 
 You must have the MS ODBC Driver installed in your Docker image or local environment.

docs/adapters/mysql.md

@@ -36,6 +36,12 @@ connection:
 | **Costing** | `EXPLAIN FORMAT=JSON` | Extracts `query_cost`. |
 | **Stats** | `SELECT count(*), min(), max()` | Standard aggregation. |
 
+### Optimization Details
+
+* **Dry Run**: Uses a Transaction Rollback strategy. It starts a transaction (`BEGIN`), executes the query, and immediately rolls back (`ROLLBACK`). **Note**: This means the query *is* technically executed, but its effects are reversed.
+* **Explain**: Uses `EXPLAIN FORMAT=JSON {sql}` to get the execution plan.
+* **Cost Estimate**: Parses the JSON output to extract `query_cost`. MySQL does not reliably provide a global "estimated rows" count for complex queries, so this is often returned as 0.
+
 ## Limitations
 
 * **Row Estimation**: MySQL's `EXPLAIN` does not always provide a reliable "Total Rows" estimate for complex joins compared to Postgres.

docs/adapters/postgres.md

@@ -2,8 +2,7 @@
 
 The Postgres adapter is the **Gold Standard** adapter for the platform. It supports the full set of optimization features including `EXPLAIN`-based dry runs and cost estimation.
 
-!!! info "Implementation"
-    This adapter extends `BaseSQLAlchemyAdapter`, leveraging automatic schema reflection and statistics gathering.
+This adapter extends `BaseSQLAlchemyAdapter`, leveraging automatic schema reflection and statistics gathering.
 
 ## Configuration
 
@@ -36,6 +35,16 @@ connection:
 | **Costing** | `EXPLAIN (FORMAT JSON) {sql}` | Returns "Total Cost" and "Plan Rows". |
 | **Stats** | Optimized Queries | Fetches `null_perc`, `distinct`, `min/max`. |
 
+### Optimization Details
+
+The Postgres adapter leverages native `EXPLAIN` capabilities for robust validation and estimation:
+
+* **Dry Run**: Implemented via `EXPLAIN {sql}`. This validates the SQL syntax and ensures that all tables/columns exist without actually executing the query.
+* **Explain**: Uses `EXPLAIN (FORMAT JSON) {sql}` to retrieve the full query execution plan in structured JSON format.
+* **Cost Estimate**: Uses the same `EXPLAIN (FORMAT JSON) {sql}` command. It parses the root `Plan` object to extract:
+  * `Total Cost`: Used as the query cost proxy.
+  * `Plan Rows`: Used as the estimated result size.
+
 ## Troubleshooting
 
 ### SSL Verification

docs/adapters/sdk.md

@@ -0,0 +1,82 @@
+# Adapter SDK Reference
+
+The **Adapter SDK** (`nl2sql-adapter-sdk`) defines the core contract that all datasources must implement.
+
+## Interface: `DatasourceAdapter`
+
+All adapters must inherit from `nl2sql_adapter_sdk.interfaces.DatasourceAdapter`.
+
+```python
+from nl2sql_adapter_sdk import DatasourceAdapter
+```
+
+### Mandatory Properties
+
+| Property | Type | Description |
+| :--- | :--- | :--- |
+| `datasource_id` | `str` | Unique identifier (e.g., "production_db"). |
+| `row_limit` | `int` | **Safety Breaker**. Must return a safe limit (e.g., 1000) to prevent OOM errors. |
+| `max_bytes` | `int` | **Safety Breaker**. Recommended limit for network payloads. |
+
+### Mandatory Methods
+
+#### `fetch_schema()`
+
+Returns `SchemaMetadata`.
+
+* **Returns**: `SchemaMetadata` containing tables, columns, PKs, FKs.
+* **Requirement**: Must populate `col.statistics` (samples, min/max) for the validation logic to work effectively.
+
+#### `execute(sql: str)`
+
+Executes a query and returns results.
+
+* **Args**: `sql` (str) - The SQL query to run.
+* **Returns**: `QueryResult` with `rows` (list of dicts) and `columns` (list of names).
+
+#### `dry_run(sql: str)`
+
+Validates SQL without executing it (or safely rolling back).
+
+* **Args**: `sql` (str)
+* **Returns**: `DryRunResult(is_valid=bool, error_message=str)`
+
+### Optional Methods
+
+#### `explain(sql: str)`
+
+Returns the execution plan.
+
+* **Returns**: `QueryPlan(plan_text=str)`
+
+#### `cost_estimate(sql: str)`
+
+Returns cost/row estimates for the Physical Validator.
+
+* **Returns**: `CostEstimate(estimated_cost=float, estimated_rows=int)`
+
+---
+
+## Compliance Testing
+
+The SDK provides a compliance test suite. **All Adapters MUST pass this suite.**
+
+It verifies:
+
+* Schema Introspection (PKs/FKs detected?)
+* Type Mapping (Date -> Python Date, Numeric -> Python Float)
+* Error Handling (Bad SQL -> AdapterError)
+
+### Running Tests
+
+```python
+# tests/test_my_adapter.py
+from nl2sql_adapter_sdk.testing import BaseAdapterTest
+from my_adapter import MyAdapter
+import pytest
+
+class TestMyAdapter(BaseAdapterTest):
+    @pytest.fixture
+    def adapter(self):
+        return MyAdapter(...)
+```

docs/adapters/sqlalchemy.md

@@ -0,0 +1,43 @@
+# SQLAlchemy Adapter Reference
+
+The **SQLAlchemy Adapter** (`nl2sql-adapter-sqlalchemy`) provides a helper base class for building adapters for any SQL database supported by SQLAlchemy.
+
+## Base Class: `BaseSQLAlchemyAdapter`
+
+Constructs a robust adapter by wrapping standard SQLAlchemy components.
+
+```python
+from nl2sql_sqlalchemy_adapter import BaseSQLAlchemyAdapter
+```
+
+### Features
+
+| Feature | Description |
+| :--- | :--- |
+| **Automatic Schema** | Uses `sqlalchemy.inspect` to reflect tables, columns, and foreign keys automatically. |
+| **Automatic Stats** | Runs optimized generic SQL queries to fetch `min`, `max`, `null_percentage`, and `distinct_count`. |
+| **Connection Pooling** | Manages engine lifecycle and connection pools. |
+| **Transaction Safety** | Implements generic `dry_run` using transaction rollbacks. |
+
+### Required Overrides
+
+#### `construct_uri(args: Dict[str, Any]) -> str`
+
+Converts a configuration dictionary into a SQLAlchemy connection string.
+
+* **Args**: `args` - The `connection` dictionary from `datasources.yaml`.
+* **Returns**: A valid URL (e.g., `postgresql://...`).
+
+### Optional Overrides
+
+#### `connect()`
+
+Override to provide custom connection arguments (e.g., timeouts, isolation levels).
+
+#### `get_dialect() -> str`
+
+Returns the logical dialect name. Defaults to the engine driver name.
+
+#### `explain(sql: str)` / `cost_estimate(sql: str)`
+
+The base class provides stubs. Override these to implement database-specific optimization logic (e.g., `EXPLAIN ANALYZE`).

docs/adapters/sqlite.md

@@ -32,6 +32,12 @@ connection:
 | **Dry Run** | `EXPLAIN QUERY PLAN` | Validates parsing (rudimentary). |
 | **Costing** | Stubbed | Returns default cost=1.0. |
 
+### Optimization Details
+
+* **Dry Run**: Uses `EXPLAIN QUERY PLAN {sql}`. If this command succeeds, the SQL syntax is valid.
+* **Explain**: Currently stubbed (returns a simple message) as SQLite's explain output is not in a standardized, easily parsable format like JSON or XML.
+* **Cost Estimate**: Stubbed. Returns a fixed cost of `1.0` and `10` estimated rows, as SQLite does not expose cost metrics comfortably.
+
 ## Hints
 
 * **Concurrency**: SQLite is poor at high concurrency. Use for **Lite Mode** or single-user testing only.