feat(api): Implement datasource and LLM management endpoints

- Added new endpoints for managing datasources including adding, listing, retrieving, and removing datasources.
- Introduced LLM management endpoints for configuring, listing, and retrieving LLM details.
- Enhanced health and readiness checks with service integration.
- Refactored query execution to utilize the new service architecture.
- Updated API documentation to reflect the new two-tier architecture and added endpoint descriptions.
- Introduced new service classes for datasource, LLM, health, and indexing management.
- Removed deprecated schema endpoints and integrated schema retrieval into the new structure.

Author: nadeem4

docs/api/index.md

@@ -1,12 +1,30 @@
 # API Overview
 
-The primary Python API surface for running NL2SQL is small and centered on:
+NL2SQL provides a two-tier API architecture for flexible integration:
+
+## Two-Tier API Architecture
+
+### 1. Core API (Python)
+- **Location**: Core package (`nl2sql-core`)
+- **Interface**: Direct Python class interface (`NL2SQL` class and low-level functions)
+- **Use Case**: Direct Python integration, embedded applications
+- **Access**: Import and use directly in Python code
+
+### 2. REST API (HTTP)
+- **Location**: API package (`nl2sql-api`)
+- **Interface**: HTTP REST endpoints
+- **Use Case**: Remote clients, web applications, TypeScript CLI
+- **Access**: HTTP requests to API endpoints
+
+Both APIs provide access to the same underlying NL2SQL engine functionality, allowing flexible integration options.
+
+## Core API (Low-Level)
+
+The low-level Python API surface is centered on:
 
 - `NL2SQLContext` (initialization)
 - `run_with_graph()` (pipeline execution)
 
-## Core API
-
 ```python
 from nl2sql.context import NL2SQLContext
 from nl2sql.pipeline.runtime import run_with_graph
@@ -15,6 +33,27 @@ ctx = NL2SQLContext()
 result = run_with_graph(ctx, "Example query")
 ```
 
+## High-Level Core API
+
+The high-level Python API provides a cleaner interface through the `NL2SQL` class:
+
+```python
+from nl2sql import NL2SQL
+
+engine = NL2SQL()
+result = engine.run_query("Example query")
+```
+
+## REST API
+
+The REST API provides HTTP endpoints for remote access:
+
+- `POST /api/v1/query` - Execute natural language queries
+- `GET /api/v1/schema/{datasource_id}` - Get schema information
+- `GET /api/v1/schema` - List available datasources
+- `GET /api/v1/health` - Health check
+- `GET /api/v1/ready` - Readiness check
+
 ## Supporting contracts
 
 - `GraphState` (`nl2sql.pipeline.state.GraphState`)
@@ -25,3 +64,5 @@ result = run_with_graph(ctx, "Example query")
 
 - Context: `packages/core/src/nl2sql/context.py`
 - Runtime: `packages/core/src/nl2sql/pipeline/runtime.py`
+- Public API: `packages/core/src/nl2sql/public_api.py`
+- API Package: `packages/api/src/nl2sql_api/`

docs/getting-started.md

@@ -1,14 +1,28 @@
 # Getting Started
 
-This guide walks through the minimal steps to run the NL2SQL pipeline from Python code. The runtime is configured via environment variables and the YAML/JSON files under `configs/`.
+This guide walks through the minimal steps to run the NL2SQL pipeline. NL2SQL provides a **two-tier API architecture** for flexible integration:
+
+1. **Core API (Python)**: Direct Python integration using the `NL2SQL` class
+2. **REST API (HTTP)**: HTTP endpoints for remote access and web applications
+
+Choose the option that best fits your use case.
 
 ## Prerequisites
 
-- Python 3.10+
+- Python 3.10+ (for Python integration)
+- Docker (for API container deployment, optional)
 - A configured datasource in `configs/datasources.yaml`
 - An LLM configuration in `configs/llm.yaml`
 
-## Install the core package
+## Installation Options
+
+### Option 1: Install Core Package from PyPI
+
+```bash
+pip install nl2sql-core
+```
+
+### Option 2: Install from Local Source (Development)
 
 ```bash
 pip install -e packages/core
@@ -21,7 +35,13 @@ If you are using the SQLAlchemy adapter, install it as well:
 pip install -e packages/adapter-sqlalchemy
 ```
 
-## Configure the system
+### Option 3: Install API Package
+
+```bash
+pip install -e packages/api
+```
+
+## Configuration
 
 `NL2SQLContext` reads its configuration from these settings (see `nl2sql.common.settings.Settings`):
 
@@ -32,11 +52,41 @@ pip install -e packages/adapter-sqlalchemy
 
 Template examples exist in `configs/*.example.yaml` and `configs/*.example.json`.
 
-## Index schema (required for retrieval)
+## Two Integration Approaches
+
+### Approach 1: Core API (Python) - Direct Integration
+
+Use the Core API for direct Python integration in your applications.
+
+#### Initialize the Engine
+
+```python
+from nl2sql import NL2SQL
+
+# Initialize with configuration files
+engine = NL2SQL(
+    ds_config_path="configs/datasources.yaml",
+    llm_config_path="configs/llm.yaml",
+    secrets_config_path="configs/secrets.yaml",
+    policies_config_path="configs/policies.json"
+)
+
+# Or initialize with default settings
+engine = NL2SQL()
+```
+
+#### Index Schema (Required for Retrieval)
 
 Before running queries, index your datasource schema:
 
 ```python
+# Index a specific datasource
+engine.index_datasource("your_datasource_id")
+
+# Index all registered datasources
+engine.index_all_datasources()
+
+# Or use the lower-level approach
 from nl2sql.context import NL2SQLContext
 from nl2sql.indexing.orchestrator import IndexingOrchestrator
 
@@ -47,24 +97,184 @@ for adapter in ctx.ds_registry.list_adapters():
     orchestrator.index_datasource(adapter)
 ```
 
-## Run a query
+#### Run Queries
 
 ```python
-from nl2sql.context import NL2SQLContext
-from nl2sql.pipeline.runtime import run_with_graph
+from nl2sql.auth.models import UserContext
 
-ctx = NL2SQLContext()
-result = run_with_graph(ctx, "Top 5 customers by revenue last quarter?")
+# Create a user context (optional)
+user_ctx = UserContext(
+    user_id="user123",
+    roles=["admin"]
+)
+
+# Run a natural language query
+result = engine.run_query(
+    "Top 5 customers by revenue last quarter?",
+    user_context=user_ctx
+)
+
+print(result.final_answer)
+print(result.sql)
+print(result.errors)
+```
+
+#### Manage Datasources Programmatically
+
+```python
+# Add a datasource programmatically
+engine.add_datasource({
+    "id": "my_postgres_db",
+    "description": "My PostgreSQL database",
+    "connection": {
+        "type": "postgres",
+        "host": "localhost",
+        "port": 5432,
+        "database": "mydb",
+        "user": "${SECRET_POSTGRES_USER}",
+        "password": "${SECRET_POSTGRES_PASSWORD}"
+    }
+})
+
+# List all datasources
+datasources = engine.list_datasources()
+print(datasources)
+```
+
+#### Configure LLMs Programmatically
 
+```python
+# Configure an LLM programmatically
+engine.configure_llm({
+    "name": "my_openai_model",
+    "provider": "openai",
+    "model": "gpt-4o",
+    "api_key": "${SECRET_OPENAI_API_KEY}",
+    "temperature": 0.0
+})
+```
+
+### Approach 2: REST API (HTTP) - Remote Access
+
+Use the REST API for remote access, web applications, or when you need HTTP-based integration.
+
+#### Start the API Server
+
+```bash
+# Using the CLI command
+nl2sql-api --host 0.0.0.0 --port 8000 --reload
+
+# Or using uvicorn directly
+uvicorn nl2sql_api.main:app --host 0.0.0.0 --port 8000 --reload
+```
+
+#### Use the REST API
+
+Once the server is running, you can make HTTP requests:
+
+```bash
+# Execute a query
+curl -X POST http://localhost:8000/api/v1/query \
+  -H "Content-Type: application/json" \
+  -d '{
+    "natural_language": "Top 5 customers by revenue last quarter?",
+    "datasource_id": "your_datasource_id",
+    "execute": true
+  }'
+
+# Get schema for a specific datasource
+curl http://localhost:8000/api/v1/schema/your_datasource_id
+
+# List all datasources
+curl http://localhost:8000/api/v1/schema
+
+# Add a new datasource
+curl -X POST http://localhost:8000/api/v1/datasource \
+  -H "Content-Type: application/json" \
+  -d '{
+    "config": {
+      "id": "my_postgres_db",
+      "description": "My PostgreSQL database",
+      "connection": {
+        "type": "postgres",
+        "host": "localhost",
+        "port": 5432,
+        "database": "mydb",
+        "user": "${SECRET_POSTGRES_USER}",
+        "password": "${SECRET_POSTGRES_PASSWORD}"
+      }
+    }
+  }'
+
+# List all registered datasources
+curl http://localhost:8000/api/v1/datasource
+
+# Configure an LLM
+curl -X POST http://localhost:8000/api/v1/llm \
+  -H "Content-Type: application/json" \
+  -d '{
+    "config": {
+      "name": "my_openai_model",
+      "provider": "openai",
+      "model": "gpt-4o",
+      "api_key": "${SECRET_OPENAI_API_KEY}",
+      "temperature": 0.0
+    }
+  }'
+
+# Index a specific datasource
+curl -X POST http://localhost:8000/api/v1/index/my_postgres_db
+
+# Index all datasources
+curl -X POST http://localhost:8000/api/v1/index-all
+
+# Get index status
+curl http://localhost:8000/api/v1/index/status
+
+# Health check
+curl http://localhost:8000/api/v1/health
+```
+
+#### Python Client for REST API
+
+You can also use Python to interact with the REST API:
+
+```python
+import requests
+
+# Execute a query via REST API
+response = requests.post("http://localhost:8000/api/v1/query", json={
+    "natural_language": "Top 5 customers by revenue last quarter?",
+    "datasource_id": "your_datasource_id",
+    "execute": True
+})
+
+result = response.json()
 print(result.get("final_answer"))
-print(result.get("errors"))
 ```
 
-## Execution flag
+## API Comparison
+
+| Feature | Core API (Python) | REST API (HTTP) |
+|---------|------------------|-----------------|
+| **Latency** | Lower (direct call) | Higher (network round-trip) |
+| **Deployment** | Embedded in your app | Separate service |
+| **Scaling** | Scale with your app | Independent scaling |
+| **Security** | App-level security | Network-level security |
+| **Use Cases** | Embedded, internal tools | Web apps, remote clients |
+| **Configuration** | Python code | HTTP requests |
+
+## Execution Flag
+
+Both APIs accept an `execute` flag that determines whether to execute the generated SQL against the database:
 
-`run_with_graph()` accepts an `execute` flag and passes it to `build_graph()`. The current graph builder does not branch on this flag, so any execution gating must be implemented inside nodes or executors.
+- `execute=True` (default): Execute the SQL and return results
+- `execute=False`: Return the generated SQL without execution
 
-## Next steps
+## Next Steps
 
 - See `configuration/system.md` for configuration details and environment variable mapping.
 - See `deployment/architecture.md` for production deployment guidance.
+- See `api/index.md` for detailed API documentation.
+- See `packages/core/PUBLIC_API_DOCS.md` for Core API reference.
+- See `packages/api/API_DOCS.md` for REST API reference.

packages/adapter-sdk/src/nl2sql_adapter_sdk/protocols.py

@@ -39,3 +39,7 @@ def get_dialect(self) -> str:
     def test_connection(self) -> bool:
         """Test if the connection to the datasource can be established."""
         ...
+
+    def get_details(self) -> Dict[str, Any]:
+        """Return detailed information about the datasource."""
+        ...

packages/adapter-sqlalchemy/src/nl2sql_sqlalchemy_adapter/adapter.py

@@ -473,3 +473,4 @@ def test_connection(self) -> bool:
         except Exception as e:
             logger.error(f"Connection test failed for {self}: {e}")
             return False
+