feat: expand documentation and architecture details for NL2SQL

- Updated mkdocs.yml to enhance navigation, adding new sections for Indexing and Extensions, and reorganizing existing content for clarity.
- Introduced a new glossary.md file to define core concepts and terminology used throughout the documentation.
- Enhanced getting-started.md with instructions for indexing datasource schemas before query execution.
- Added detailed architecture documents for pipeline, indexing, and various nodes, improving understanding of system components and their interactions.
- Included multiple ADRs to capture architectural decisions related to chunking strategy, schema store design, adapter abstraction, deterministic planning, and artifact storage.

Author: nadeem4

docs/adapters/architecture.md

@@ -1,6 +1,6 @@
 # Plugin / Adapter Architecture
 
-Adapters are discovered via Python entry points (`nl2sql.adapters`) and registered in `DatasourceRegistry`. All adapters implement the `DatasourceAdapterProtocol` and return a standardized `ResultFrame`.
+Adapters integrate NL2SQL with external datasources. Each adapter implements a **protocol contract**, is discovered via **Python entry points**, and is registered in the `DatasourceRegistry`.
 
 ## Discovery and registration
 
@@ -13,7 +13,7 @@ flowchart TD
     AdapterClass --> AdapterInstance[DatasourceAdapterProtocol instance]
 ```
 
-## Core contracts
+## Core adapter contract
 
 ```mermaid
 classDiagram
@@ -27,9 +27,7 @@ classDiagram
     class AdapterRequest {
         +plan_type
         +payload
-        +parameters
         +limits
-        +trace_id
     }
     class ResultFrame {
         +success
@@ -40,13 +38,37 @@ classDiagram
     }
 ```
 
-## Executor integration
+## Capability-driven routing
 
-Execution nodes resolve the executor via `ExecutorRegistry`, which maps datasource capabilities to executor implementations (e.g., `SqlExecutorService` for SQL).
+Adapters expose capabilities (e.g., `supports_sql`, `supports_schema_introspection`). These capabilities drive:
+
+- **Subgraph selection** (`resolve_subgraph()` in routing).
+- **Executor selection** (`ExecutorRegistry.get_executor()`).
+
+```mermaid
+flowchart TD
+    Adapter[DatasourceAdapterProtocol] --> Caps[capabilities()]
+    Caps --> Exec[ExecutorRegistry]
+    Caps --> Subgraph[resolve_subgraph()]
+    Exec --> Service[Executor Service]
+    Subgraph --> Graph[Subgraph Selection]
+```
+
+## Multi-datasource routing
+
+The control graph can resolve multiple datasources for a single user query. `DecomposerNode` produces sub-queries scoped to individual datasources. Each sub-query is then routed to a subgraph that matches its adapter capabilities.
+
+## Extensibility model
+
+To add a new adapter:
+
+1. Implement `DatasourceAdapterProtocol` (or extend a base adapter).
+2. Publish the adapter class as an `nl2sql.adapters` entry point.
+3. Configure the datasource in `configs/datasources.yaml`.
 
 ## Source references
 
-- Adapter protocol and contracts: `packages/adapter-sdk/src/nl2sql_adapter_sdk/protocols.py`, `packages/adapter-sdk/src/nl2sql_adapter_sdk/contracts.py`
+- Adapter protocol: `packages/adapter-sdk/src/nl2sql_adapter_sdk/protocols.py`
 - Adapter discovery: `packages/core/src/nl2sql/datasources/discovery.py`
 - Datasource registry: `packages/core/src/nl2sql/datasources/registry.py`
-- Executor registry: `packages/core/src/nl2sql/execution/executor/registry.py`
+- Example adapter: `packages/adapter-sqlalchemy/src/nl2sql_sqlalchemy_adapter/adapter.py`

docs/adr/adr-003-chunking-strategy.md

@@ -0,0 +1,41 @@
+# ADR-003: Schema Chunking Strategy
+
+## Status
+
+Accepted (implemented in `SchemaChunkBuilder` and `VectorStore`).
+
+## Context
+
+Full-schema injection into LLM prompts is brittle and expensive. Retrieval needs to be **semantically structured** so that:
+
+- Datasource routing is reliable.
+- Schema grounding is precise.
+- Planning context is scoped to relevant tables and columns.
+
+## Decision
+
+Use **typed schema chunks** with staged retrieval:
+
+- `schema.datasource` for datasource routing and grounding.
+- `schema.table` for table-level context and primary keys.
+- `schema.column` for column semantics and statistics.
+- `schema.relationship` for explicit join hints.
+
+Retrieval is staged in `SchemaRetrieverNode`:
+
+1. `retrieve_schema_context()` (tables/metrics)
+2. fallback to `retrieve_column_candidates()` if no tables found
+3. `retrieve_planning_context()` for columns/relationships of selected tables
+
+## Consequences
+
+- Reduces LLM context to schema slices relevant to the query.
+- Preserves authoritative schema by resolving final context from `SchemaStore`.
+- Enables deterministic and explainable retrieval behavior.
+
+## Source references
+
+- Chunk models: `packages/core/src/nl2sql/indexing/models.py`
+- Chunk builder: `packages/core/src/nl2sql/indexing/chunk_builder.py`
+- Retrieval: `packages/core/src/nl2sql/indexing/vector_store.py`
+- Schema retriever: `packages/core/src/nl2sql/pipeline/nodes/schema_retriever/node.py`

docs/adr/adr-004-schema-store-design.md

@@ -0,0 +1,31 @@
+# ADR-004: Schema Store Design and Fingerprinting
+
+## Status
+
+Accepted (implemented in `SqliteSchemaStore` and `InMemorySchemaStore`).
+
+## Context
+
+The system needs an authoritative, versioned view of each datasource schema. Vector indexes may drift or be stale, so planning must reference a canonical schema snapshot.
+
+## Decision
+
+Store schema snapshots with **deterministic fingerprints**:
+
+- `SchemaContract` content is hashed to produce a stable fingerprint.
+- Snapshots are versioned using timestamp + fingerprint prefix.
+- Older versions are evicted beyond a configurable maximum.
+
+Persistent storage is provided by a SQLite-backed schema store, with an in-memory alternative for testing.
+
+## Consequences
+
+- Schema versions are stable and deduplicated.
+- Retrieval uses authoritative snapshots even if vector chunks drift.
+- The system can enforce version mismatch policies.
+
+## Source references
+
+- Fingerprinting: `packages/core/src/nl2sql/schema/protocol.py`
+- Sqlite store: `packages/core/src/nl2sql/schema/sqlite_store.py`
+- In-memory store: `packages/core/src/nl2sql/schema/in_memory_store.py`

docs/adr/adr-005-adapter-abstraction.md

@@ -0,0 +1,32 @@
+# ADR-005: Adapter Abstraction and Capability Routing
+
+## Status
+
+Accepted (implemented via `DatasourceAdapterProtocol`, registries, and routing).
+
+## Context
+
+NL2SQL must support heterogeneous datasources (SQL, REST, GraphQL, etc.) while keeping orchestration stable and deterministic.
+
+## Decision
+
+Adopt a **capability-driven adapter abstraction**:
+
+- Adapters implement `DatasourceAdapterProtocol`.
+- Capabilities are declared via `capabilities()`.
+- Routing and execution select services/subgraphs based on capability subsets.
+
+Adapters are discovered via Python entry points and registered at runtime based on configuration.
+
+## Consequences
+
+- New datasources can be integrated without changing core orchestration.
+- Subgraphs and executors remain decoupled and capability-focused.
+- Capability mismatches fail fast with clear errors.
+
+## Source references
+
+- Adapter protocol: `packages/adapter-sdk/src/nl2sql_adapter_sdk/protocols.py`
+- Adapter discovery: `packages/core/src/nl2sql/datasources/discovery.py`
+- Datasource registry: `packages/core/src/nl2sql/datasources/registry.py`
+- Routing: `packages/core/src/nl2sql/pipeline/routes.py`

docs/adr/adr-006-deterministic-planning.md

@@ -0,0 +1,29 @@
+# ADR-006: Deterministic Planning and Stable IDs
+
+## Status
+
+Accepted (implemented in decomposer and DAG models).
+
+## Context
+
+Enterprise workflows require repeatable orchestration to enable reliable caching, debugging, and audit trails. Non-deterministic planning introduces unstable IDs and inconsistent execution paths.
+
+## Decision
+
+Use **stable hashes and deterministic layering**:
+
+- `DecomposerNode` generates stable sub-query and post-op IDs by hashing content.
+- `ExecutionDAG._layered_toposort()` deterministically computes execution layers.
+- Aggregation processes layers in deterministic order.
+
+## Consequences
+
+- Artifact keys and execution node IDs are stable across runs.
+- Deterministic routing and aggregation simplify debugging and auditing.
+- Planning remains reproducible even when subgraphs run in parallel.
+
+## Source references
+
+- Decomposer: `packages/core/src/nl2sql/pipeline/nodes/decomposer/node.py`
+- Execution DAG: `packages/core/src/nl2sql/pipeline/nodes/global_planner/schemas.py`
+- Aggregation: `packages/core/src/nl2sql/aggregation/aggregator.py`

docs/adr/adr-007-artifact-storage.md

@@ -0,0 +1,31 @@
+# ADR-007: Artifact Storage for Execution Results
+
+## Status
+
+Accepted (implemented in `ArtifactStore` and executor services).
+
+## Context
+
+Query execution results need to be persisted for aggregation and downstream usage. Persisting raw results in memory would be expensive and non-durable for multi-step DAGs.
+
+## Decision
+
+Persist execution results as Parquet artifacts:
+
+- Adapters return `ResultFrame` objects.
+- `SqlExecutorService` writes results to an `ArtifactStore`.
+- Aggregation reads artifacts and applies combine/post operations.
+
+Backends are pluggable (`local`, `s3`, `adls`).
+
+## Consequences
+
+- Results are durable across pipeline stages.
+- Aggregation operates on persisted artifacts, reducing memory pressure.
+- Backends can be swapped without changing executor logic.
+
+## Source references
+
+- Artifact store base: `packages/core/src/nl2sql/execution/artifacts/base.py`
+- Local store: `packages/core/src/nl2sql/execution/artifacts/local_store.py`
+- Executor service: `packages/core/src/nl2sql/execution/executor/sql_executor.py`

docs/adr/index.md

@@ -4,3 +4,8 @@ This section captures architectural decisions that are reflected in the current
 
 - `adr-001-sandboxed-execution.md`
 - `adr-002-circuit-breakers.md`
+- `adr-003-chunking-strategy.md`
+- `adr-004-schema-store-design.md`
+- `adr-005-adapter-abstraction.md`
+- `adr-006-deterministic-planning.md`
+- `adr-007-artifact-storage.md`

docs/agents/architecture.md

@@ -153,3 +153,24 @@ flowchart LR
   - Adds a `PLAN_FEEDBACK` warning to drive retry logic.
 - **Errors**: `MISSING_LLM`, `REFINER_FAILED`
 - **Source**: `packages/core/src/nl2sql/pipeline/nodes/refiner/node.py`
+
+## Shared state model (subgraph)
+
+`SubgraphExecutionState` carries:
+
+- `sub_query`, `user_context`, `relevant_tables`
+- `ast_planner_response`, `logical_validator_response`, `generator_response`, `executor_response`
+- `retry_count`, `errors`, `reasoning`, `warnings`
+
+The subgraph state is merged back into `GraphState` via `wrap_subgraph()`, which extracts artifacts and diagnostics into `SubgraphOutput`.
+
+## Retry and failure semantics
+
+- Planner/validator failures trigger the `retry_handler` path if errors are retryable and `retry_count < sql_agent_max_retries`.
+- Critical failures (e.g., RBAC violations, missing plan) are non-retryable.
+- Physical validation exists but is currently not wired in the default subgraph.
+
+## Deterministic behavior notes
+
+- Sub-query IDs are deterministic hashes, ensuring stable artifact keys.
+- `ExecutionDAG` layers are deterministic, so subgraph invocation order is stable.