diff --git a/docs/integrations/app-integrations/rest-api.mdx b/docs/integrations/app-integrations/rest-api.mdx new file mode 100644 index 00000000000..1ead2f519ae --- /dev/null +++ b/docs/integrations/app-integrations/rest-api.mdx @@ -0,0 +1,148 @@ +--- +title: REST API +sidebarTitle: REST API +--- + +In this section, we present how to connect any REST API to MindsDB using bearer-token authentication. + +The REST API handler is a generic integration that lets you forward HTTP requests to any API through MindsDB using stored credentials. Unlike named integrations (HubSpot, Shopify, etc.), it requires no handler-specific knowledge — just a base URL and a bearer token. + +This is useful for APIs that MindsDB doesn't have a dedicated handler for, or when you only need direct HTTP access without SQL table mapping. + +## Connection + +The required arguments to establish a connection are as follows: + +- `base_url`: the base URL of the REST API (e.g. `https://api.example.com`). All request paths are appended to this URL. +- `bearer_token`: the token used for authentication. Injected as `Authorization: Bearer ` on every request. + +Optional arguments: + +- `default_headers`: a JSON object of static headers added to every request (e.g. `{"Accept": "application/json"}`). +- `allowed_hosts`: a list of allowed hostnames for requests. Defaults to the hostname of `base_url`. Use `["*"]` to disable host containment. +- `test_path`: the path used by the test endpoint to verify connectivity. Defaults to `/`. + +To connect a REST API to MindsDB, create a new database: + +```sql +CREATE DATABASE my_api +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://api.example.com", + "bearer_token": "your_token_here" +}; +``` + +### Example: Connect to HubSpot + +```sql +CREATE DATABASE my_hubspot +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://api.hubapi.com", + "bearer_token": "pat-eu1-..." +}; +``` + +### Example: Connect with default headers and a custom test path + +```sql +CREATE DATABASE my_internal_api +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://internal.example.com/api/v2", + "bearer_token": "sk-...", + "default_headers": {"Accept": "application/json"}, + "test_path": "/health" +}; +``` + +### Example: Multiple allowed hosts + +```sql +CREATE DATABASE my_multi_region_api +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://api.example.com", + "bearer_token": "your_token", + "allowed_hosts": ["api.example.com", "api.eu.example.com"] +}; +``` + +## Usage + +This handler is **passthrough-only** — it does not expose SQL tables. All interaction is through the REST passthrough endpoint. + +### Sending requests + +Forward HTTP requests to the upstream API: + +``` +POST /api/integrations/my_api/passthrough +``` + +```json +{ + "method": "GET", + "path": "/v1/users", + "query": {"limit": "10"}, + "headers": {"Accept": "application/json"} +} +``` + +The response wraps the upstream HTTP response: + +```json +{ + "status_code": 200, + "headers": {"content-type": "application/json"}, + "body": {"results": [...]}, + "content_type": "application/json" +} +``` + +Supported HTTP methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`. + +### Testing the connection + +Verify that the base URL, token, and host allowlist are configured correctly: + +``` +POST /api/integrations/my_api/passthrough/test +``` + +A successful response: + +```json +{"ok": true, "status_code": 200, "host": "api.example.com", "latency_ms": 140} +``` + +A failed response: + +```json +{"ok": false, "error_code": "auth_failed", "message": "upstream rejected credentials; base URL and allowlist look correct"} +``` + +## Security + +- Credentials are stored in MindsDB and never exposed to the caller. +- Requests are restricted to hostnames in the allowlist. Private and loopback IP addresses are rejected by default. +- Callers cannot override `Authorization`, `Host`, `Cookie`, or `Proxy-*` headers. +- If the upstream API echoes the token in responses, it is replaced with `[REDACTED_API_KEY]`. +- Request bodies are capped at 1 MB, response bodies at 10 MB. + + +**`host 'X' is not in the datasource allowlist`** + +The request path resolved to a different hostname than `base_url`. Add the hostname to `allowed_hosts`, or use `["*"]` to disable host containment (not recommended for production). + + + +**`upstream rejected credentials (401/403)`** + +The token is invalid, expired, or missing required scopes. Verify the token with the upstream API provider. + + + +For more information about available actions and development plans, visit [this page](https://github.com/mindsdb/mindsdb/blob/main/mindsdb/integrations/handlers/rest_api_handler/README.md). + diff --git a/mindsdb/integrations/handlers/rest_api_handler/README.md b/mindsdb/integrations/handlers/rest_api_handler/README.md new file mode 100644 index 00000000000..037d6ff429a --- /dev/null +++ b/mindsdb/integrations/handlers/rest_api_handler/README.md @@ -0,0 +1,168 @@ +--- +title: REST API +sidebarTitle: REST API +--- + +This documentation describes the integration of MindsDB with generic REST APIs using bearer-token authentication. +The integration allows MindsDB to forward HTTP requests to any REST API using stored credentials via the passthrough endpoint — no SQL table mapping required. + +### Prerequisites + +Before proceeding, ensure the following prerequisites are met: + +1. Install MindsDB locally via [Docker](https://docs.mindsdb.com/setup/self-hosted/docker) or [Docker Desktop](https://docs.mindsdb.com/setup/self-hosted/docker-desktop). +2. Obtain a bearer token (API key, personal access token, etc.) for the target REST API. + +## Connection + +Establish a connection to a REST API from MindsDB by executing the following SQL command: + +```sql +CREATE DATABASE my_api +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://api.example.com", + "bearer_token": "your_token_here" +}; +``` + +Required connection parameters include the following: + +* `base_url`: The base URL of the REST API (e.g. `https://api.example.com`). All passthrough request paths are appended to this URL. +* `bearer_token`: The bearer token used for authentication. Injected as `Authorization: Bearer ` on every request. + +Optional connection parameters include the following: + +* `default_headers`: A JSON object of static headers added to every request (e.g. `{"Accept": "application/json"}`). +* `allowed_hosts`: A list of allowed hostnames for passthrough requests. Defaults to the hostname of `base_url`. Use `["*"]` to disable host containment. +* `test_path`: The path used by the `/passthrough/test` endpoint to verify connectivity. Defaults to `/`. + +### Examples + +Connect to the HubSpot API: + +```sql +CREATE DATABASE my_hubspot +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://api.hubapi.com", + "bearer_token": "pat-eu1-..." +}; +``` + +Connect to a custom internal API with default headers: + +```sql +CREATE DATABASE my_internal_api +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://internal.example.com/api/v2", + "bearer_token": "sk-...", + "default_headers": {"Accept": "application/json", "X-Team": "data"}, + "test_path": "/health" +}; +``` + +Connect to an API with multiple allowed hosts: + +```sql +CREATE DATABASE my_multi_region_api +WITH ENGINE = 'rest_api', +PARAMETERS = { + "base_url": "https://api.example.com", + "bearer_token": "your_token", + "allowed_hosts": ["api.example.com", "api.eu.example.com"] +}; +``` + +## Usage + +This handler is **passthrough-only** — it does not expose SQL tables. All interaction is through the REST passthrough endpoint. + +### Passthrough Requests + +Send HTTP requests to the upstream API through MindsDB: + +``` +POST /api/integrations/my_api/passthrough +``` + +```json +{ + "method": "GET", + "path": "/v1/users", + "query": {"limit": "10"}, + "headers": {"Accept": "application/json"} +} +``` + +The response wraps the upstream HTTP response: + +```json +{ + "status_code": 200, + "headers": {"content-type": "application/json"}, + "body": {"results": [...]}, + "content_type": "application/json" +} +``` + +Supported HTTP methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`. + +### Testing the Connection + +Verify that the base URL, token, and host allowlist are configured correctly: + +``` +POST /api/integrations/my_api/passthrough/test +``` + +Returns: + +```json +{"ok": true, "status_code": 200, "host": "api.example.com", "latency_ms": 140} +``` + +Or on failure: + +```json +{"ok": false, "error_code": "auth_failed", "message": "upstream rejected credentials; base URL and allowlist look correct"} +``` + +## Security + +- **Credentials are never exposed.** The bearer token is stored in MindsDB and injected at request time. It is never returned to the caller. +- **Host containment.** Requests are restricted to hostnames in the allowlist (defaults to the `base_url` host). Private/loopback IP addresses are rejected by default. +- **Header filtering.** Callers cannot override `Authorization`, `Host`, `Cookie`, or `Proxy-*` headers. +- **Response scrubbing.** If the upstream API echoes the token in responses, it is replaced with `[REDACTED_API_KEY]` before returning to the caller. +- **Size limits.** Request bodies are capped at 1 MB, response bodies at 10 MB (configurable via environment variables). + +## Troubleshooting + + +`base_url is not configured` + +* **Symptoms**: Passthrough requests fail with a configuration error. +* **Checklist**: + 1. Ensure `base_url` is provided in the connection parameters. + 2. The URL must include the scheme (`https://`). + + + +`host 'X' is not in the datasource allowlist` + +* **Symptoms**: Passthrough requests to a valid URL are rejected. +* **Checklist**: + 1. The request path may resolve to a different hostname than `base_url`. + 2. Add the hostname to `allowed_hosts` in the connection parameters. + 3. Use `["*"]` to disable host containment (not recommended for production). + + + +`upstream rejected credentials (401/403)` + +* **Symptoms**: The `/passthrough/test` endpoint returns `error_code: "auth_failed"`. +* **Checklist**: + 1. Verify the bearer token is valid and not expired. + 2. Check that the token has the required scopes/permissions for the API endpoints you are calling. + diff --git a/mindsdb/integrations/handlers/rest_api_handler/__about__.py b/mindsdb/integrations/handlers/rest_api_handler/__about__.py new file mode 100644 index 00000000000..b7f131f401c --- /dev/null +++ b/mindsdb/integrations/handlers/rest_api_handler/__about__.py @@ -0,0 +1,9 @@ +__title__ = "MindsDB REST API handler" +__package_name__ = "mindsdb_rest_api_handler" +__version__ = "0.0.1" +__description__ = "MindsDB handler for generic REST APIs with bearer-token passthrough" +__author__ = "MindsDB Inc" +__github__ = "https://github.com/mindsdb/mindsdb" +__pypi__ = "https://pypi.org/project/mindsdb/" +__license__ = "MIT" +__copyright__ = "Copyright 2026 - mindsdb" diff --git a/mindsdb/integrations/handlers/rest_api_handler/__init__.py b/mindsdb/integrations/handlers/rest_api_handler/__init__.py new file mode 100644 index 00000000000..d9f8fcf24eb --- /dev/null +++ b/mindsdb/integrations/handlers/rest_api_handler/__init__.py @@ -0,0 +1,32 @@ +from mindsdb.integrations.libs.const import HANDLER_TYPE, HANDLER_SUPPORT_LEVEL + +from .__about__ import __version__ as version, __description__ as description +from .connection_args import connection_args, connection_args_example + +try: + from .rest_api_handler import RestApiHandler as Handler + + import_error = None +except Exception as e: + Handler = None + import_error = e + +title = "REST API" +name = "rest_api" +type = HANDLER_TYPE.DATA +icon_path = "icon.svg" +support_level = HANDLER_SUPPORT_LEVEL.MINDSDB + +__all__ = [ + "Handler", + "version", + "name", + "type", + "support_level", + "title", + "description", + "import_error", + "icon_path", + "connection_args", + "connection_args_example", +] diff --git a/mindsdb/integrations/handlers/rest_api_handler/connection_args.py b/mindsdb/integrations/handlers/rest_api_handler/connection_args.py new file mode 100644 index 00000000000..bba20202ba0 --- /dev/null +++ b/mindsdb/integrations/handlers/rest_api_handler/connection_args.py @@ -0,0 +1,44 @@ +from collections import OrderedDict + +from mindsdb.integrations.libs.const import HANDLER_CONNECTION_ARG_TYPE as ARG_TYPE + +connection_args = OrderedDict( + base_url={ + "type": ARG_TYPE.STR, + "description": "Base URL of the REST API (e.g. https://api.example.com)", + "required": True, + "label": "Base URL", + }, + bearer_token={ + "type": ARG_TYPE.PWD, + "description": "Bearer token injected as Authorization: Bearer ", + "required": True, + "label": "Bearer Token", + "secret": True, + }, + default_headers={ + "type": ARG_TYPE.DICT, + "description": 'Static headers added to every request (e.g. {"Accept": "application/json"})', + "required": False, + "label": "Default Headers", + }, + allowed_hosts={ + "type": ARG_TYPE.LIST, + "description": 'Allowed hostnames for passthrough requests. Defaults to the base_url host. Use ["*"] to disable containment.', + "required": False, + "label": "Allowed Hosts", + }, + test_path={ + "type": ARG_TYPE.STR, + "description": "Path used by the /passthrough/test endpoint. Defaults to /", + "required": False, + "label": "Test Path", + }, +) + +connection_args_example = OrderedDict( + base_url="https://api.example.com", + bearer_token="your_token_here", + default_headers={"Accept": "application/json"}, + allowed_hosts=["api.example.com"], +) diff --git a/mindsdb/integrations/handlers/rest_api_handler/icon.svg b/mindsdb/integrations/handlers/rest_api_handler/icon.svg new file mode 100644 index 00000000000..2346f8d4d3e --- /dev/null +++ b/mindsdb/integrations/handlers/rest_api_handler/icon.svg @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/mindsdb/integrations/handlers/rest_api_handler/rest_api_handler.py b/mindsdb/integrations/handlers/rest_api_handler/rest_api_handler.py new file mode 100644 index 00000000000..a55ad7c1f4d --- /dev/null +++ b/mindsdb/integrations/handlers/rest_api_handler/rest_api_handler.py @@ -0,0 +1,87 @@ +from typing import Any + +from mindsdb.integrations.libs.api_handler import APIHandler +from mindsdb.integrations.libs.passthrough import PassthroughMixin +from mindsdb.integrations.libs.passthrough_types import PassthroughRequest +from mindsdb.integrations.libs.response import ( + HandlerStatusResponse as StatusResponse, + HandlerResponse as Response, + RESPONSE_TYPE, +) +from mindsdb.utilities import log + +logger = log.getLogger(__name__) + + +class RestApiHandler(APIHandler, PassthroughMixin): + """Generic REST API handler — passthrough only, no SQL tables. + + This is the "bring your own URL" escape hatch for any bearer-token API + that mindsdb doesn't have a named handler for. Users supply a base_url + and a bearer_token and get full passthrough access. + """ + + name = "rest_api" + + def __init__(self, name: str, **kwargs: Any) -> None: + super().__init__(name) + self.connection_data = kwargs.get("connection_data") or {} + self.kwargs = kwargs + self.is_connected = False + + # PassthroughMixin reads these instance attributes at runtime. + self._bearer_token_arg = "bearer_token" + self._base_url_default = None # user must supply base_url + + # Build the test request from connection_data. Default to GET / + # unless the user provided a custom test_path. + test_path = self.connection_data.get("test_path", "/") + if not test_path.startswith("/"): + test_path = f"/{test_path}" + self._test_request = PassthroughRequest(method="GET", path=test_path) + + def connect(self) -> None: + """No persistent connection needed — passthrough is stateless. + + Validation happens in check_connection(), which we + call separately during CREATE DATABASE. + """ + self.is_connected = True + + def check_connection(self) -> StatusResponse: + """Validate that base_url and bearer_token are present.""" + response = StatusResponse(False) + try: + base_url = self._build_base_url() + if not base_url: + response.error_message = "base_url is required" + return response + token = self.connection_data.get(self._bearer_token_arg) + if not token: + response.error_message = "bearer_token is required" + return response + response.success = True + self.is_connected = True + except Exception as e: + response.error_message = str(e) + return response + + def native_query(self, query: str) -> Response: + """Not supported — use passthrough instead.""" + return Response( + RESPONSE_TYPE.ERROR, + error_message="rest_api handler is passthrough-only. Use the /passthrough endpoint.", + ) + + def get_tables(self) -> Response: + """No SQL tables — passthrough only.""" + import pandas as pd + + return Response(RESPONSE_TYPE.TABLE, data_frame=pd.DataFrame()) + + def get_columns(self, table_name: str) -> Response: + """No SQL tables — passthrough only.""" + return Response( + RESPONSE_TYPE.ERROR, + error_message="rest_api handler is passthrough-only. No tables available.", + ) diff --git a/tests/unit/handlers/test_rest_api.py b/tests/unit/handlers/test_rest_api.py new file mode 100644 index 00000000000..ff10b02d67f --- /dev/null +++ b/tests/unit/handlers/test_rest_api.py @@ -0,0 +1,165 @@ +"""Unit tests for the generic REST API passthrough handler.""" + +from unittest.mock import patch, MagicMock + +from mindsdb.integrations.handlers.rest_api_handler.rest_api_handler import RestApiHandler +from mindsdb.integrations.libs.passthrough import PassthroughProtocol +from mindsdb.integrations.libs.passthrough_types import PassthroughRequest, PassthroughResponse +from mindsdb.integrations.libs.response import ( + HandlerStatusResponse as StatusResponse, +) + + +VALID_DATA = { + "base_url": "https://api.example.com", + "bearer_token": "test-token-123", +} + + +def _make_handler(connection_data=None): + if connection_data is None: + connection_data = dict(VALID_DATA) + return RestApiHandler("test_rest", connection_data=connection_data) + + +class TestRestApiHandlerInit: + def test_satisfies_passthrough_protocol(self): + assert issubclass(RestApiHandler, PassthroughProtocol) + + def test_stores_connection_data(self): + data = {"base_url": "https://x.com", "bearer_token": "tok"} + handler = _make_handler(data) + assert handler.connection_data == data + + def test_default_test_request_path(self): + handler = _make_handler() + assert handler._test_request.method == "GET" + assert handler._test_request.path == "/" + + def test_custom_test_path(self): + handler = _make_handler( + { + "base_url": "https://api.example.com", + "bearer_token": "tok", + "test_path": "/health", + } + ) + assert handler._test_request.path == "/health" + + def test_custom_test_path_without_slash(self): + handler = _make_handler( + { + "base_url": "https://api.example.com", + "bearer_token": "tok", + "test_path": "status", + } + ) + assert handler._test_request.path == "/status" + + +class TestCheckConnection: + def test_success(self): + handler = _make_handler() + response = handler.check_connection() + assert isinstance(response, StatusResponse) + assert response.success is True + assert not response.error_message + + def test_missing_base_url(self): + handler = _make_handler({"bearer_token": "tok"}) + response = handler.check_connection() + assert response.success is False + assert "base_url" in response.error_message + + def test_missing_bearer_token(self): + handler = _make_handler({"base_url": "https://api.example.com"}) + response = handler.check_connection() + assert response.success is False + assert "bearer_token" in response.error_message + + def test_empty_connection_data(self): + handler = _make_handler({}) + response = handler.check_connection() + assert response.success is False + + +class TestPassthroughIntegration: + """Test that the mixin methods work correctly on RestApiHandler.""" + + @patch("mindsdb.integrations.libs.passthrough.requests.request") + def test_api_passthrough_injects_bearer(self, mock_request): + mock_resp = MagicMock() + mock_resp.status_code = 200 + mock_resp.headers = {"Content-Type": "application/json"} + mock_resp.iter_content.return_value = [b'{"ok": true}'] + mock_resp.close = MagicMock() + mock_request.return_value = mock_resp + + handler = _make_handler() + result = handler.api_passthrough(PassthroughRequest(method="GET", path="/v1/users")) + + assert isinstance(result, PassthroughResponse) + assert result.status_code == 200 + headers = mock_request.call_args.kwargs["headers"] + assert headers["Authorization"] == "Bearer test-token-123" + + @patch("mindsdb.integrations.libs.passthrough.requests.request") + def test_api_passthrough_uses_base_url(self, mock_request): + mock_resp = MagicMock() + mock_resp.status_code = 200 + mock_resp.headers = {} + mock_resp.iter_content.return_value = [b""] + mock_resp.close = MagicMock() + mock_request.return_value = mock_resp + + handler = _make_handler() + handler.api_passthrough(PassthroughRequest(method="GET", path="/foo")) + + called_url = mock_request.call_args.args[1] + assert called_url == "https://api.example.com/foo" + + @patch("mindsdb.integrations.libs.passthrough.requests.request") + def test_api_passthrough_includes_default_headers(self, mock_request): + mock_resp = MagicMock() + mock_resp.status_code = 200 + mock_resp.headers = {} + mock_resp.iter_content.return_value = [b""] + mock_resp.close = MagicMock() + mock_request.return_value = mock_resp + + handler = _make_handler( + { + "base_url": "https://api.example.com", + "bearer_token": "tok", + "default_headers": {"Accept": "application/json", "X-Team": "data"}, + } + ) + handler.api_passthrough(PassthroughRequest(method="GET", path="/")) + + headers = mock_request.call_args.kwargs["headers"] + assert headers["Accept"] == "application/json" + assert headers["X-Team"] == "data" + + @patch("mindsdb.integrations.libs.passthrough.requests.request") + def test_test_passthrough_success(self, mock_request): + mock_resp = MagicMock() + mock_resp.status_code = 200 + mock_resp.headers = {"Content-Type": "application/json"} + mock_resp.iter_content.return_value = [b'{"ok": true}'] + mock_resp.close = MagicMock() + mock_request.return_value = mock_resp + + handler = _make_handler() + result = handler.test_passthrough() + + assert isinstance(result, dict) + assert result["ok"] is True + assert result["status_code"] == 200 + + def test_test_passthrough_with_no_network(self): + """test_passthrough catches connection errors gracefully.""" + handler = _make_handler() + result = handler.test_passthrough() + assert isinstance(result, dict) + assert result["ok"] is False + assert result["error_code"] in ("network", "unknown")