feat(sqlserver): add optional mssql-python connection backend and docs (#681)

Add support for selecting either the existing pyodbc backend or the new
mssql-python backend for SQL Server connections.

Keep pyodbc as the default backend for backward compatibility, while allowing
users to opt into mssql-python through the new backend configuration and
optional dependency extra.

Changes include:
- Add backend selection using a SQL Server backend enum
- Add lazy imports for optional backend dependencies
- Keep pyodbc as the default backend
- Add optional mssql-python dependency support
- Update pyodbc validation for missing, empty, and whitespace-only driver values
- Add authentication normalization and validation for SQL Server profile fields
- Add mssql-python connection-string handling for MSI and integrated authentication
- Add support for Active Directory access-token authentication
- Improve connection-string sanitization for logging
- Validate connection requirements and query timeout settings
- Expand unit tests for backend selection, authentication handling, and connection behavior
- Refactor integration tests to cover both pyodbc and mssql-python backends
- Add a dedicated mssql CI image and backend requirements table
- Update README and contributing docs for backend installation and development setup
- Add workflow_dispatch support for CI/CD Docker publishing

Author: axellpadilla

.devcontainer/setup_env.sh

@@ -2,6 +2,9 @@ set -euo pipefail
 
 cp test.env.sample test.env
 
+sudo apt-get update
+sudo apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2
+
 docker compose build
 docker compose up -d
 
@@ -12,5 +15,6 @@ command -v uv >/dev/null 2>&1 || pip install uv
 [ -d .venv ] || uv venv
 source .venv/bin/activate
 
-uv sync --group dev --extra pyodbc
+# Install both backend extras so the devcontainer can exercise either connection path.
+uv sync --group dev --extra pyodbc --extra mssql
 pre-commit install

.github/workflows/integration-tests-sqlserver.yml

@@ -35,17 +35,72 @@ on:  # yamllint disable-line rule:truthy
 
 jobs:
   integration-tests-sql-server:
-    name: Regular
+    name: ${{ matrix.backend }} / Py${{ matrix.python_version }} / SQL${{ matrix.sqlserver_version }} / ODBC${{ matrix.msodbc_version }} / ${{ matrix.collation }}
     if: github.actor != 'dependabot[bot]'
     strategy:
+      # Smaller matrix allows this to check if a failure is specific.
+      fail-fast: false
       matrix:
         python_version: ["3.10", "3.11", "3.12", "3.13"]
-        msodbc_version: ["17", "18"]
-        sqlserver_version: ["2017", "2019", "2022"]
-        collation: ["SQL_Latin1_General_CP1_CS_AS", "SQL_Latin1_General_CP1_CI_AS"]
+        backend: [pyodbc, mssql-python]
+        # Baseline on 2022
+        sqlserver_version: ["2022"]
+        msodbc_version: ["18"]
+        collation: [SQL_Latin1_General_CP1_CI_AS]
+        exclude:
+          - backend: mssql-python
+            python_version: "3.10"
+            sqlserver_version: "2022"
+          - backend: mssql-python
+            python_version: "3.11"
+            sqlserver_version: "2022"
+          - backend: mssql-python
+            python_version: "3.12"
+            sqlserver_version: "2022"
+        include:
+          # Keep pyodbc on every supported Python version, but retain
+          # SQL Server ODBC 17 coverage for the oldest and newest Python.
+          - backend: pyodbc
+            python_version: "3.10"
+            sqlserver_version: "2022"
+            msodbc_version: "17"
+            collation: SQL_Latin1_General_CP1_CI_AS
+          - backend: pyodbc
+            python_version: "3.13"
+            sqlserver_version: "2022"
+            msodbc_version: "17"
+            collation: SQL_Latin1_General_CP1_CI_AS
+          # Older SQL Server versions stay on pyodbc only, with a single
+          # collation so the matrix stays small and readable.
+          - backend: pyodbc
+            python_version: "3.10"
+            sqlserver_version: "2017"
+            msodbc_version: "17"
+            collation: SQL_Latin1_General_CP1_CI_AS
+          - backend: pyodbc
+            python_version: "3.13"
+            sqlserver_version: "2019"
+            msodbc_version: "17"
+            collation: SQL_Latin1_General_CP1_CI_AS
+          # Add the case-sensitive collation only on the latest SQL Server
+          # and latest Python/backend rows.
+          - backend: pyodbc
+            python_version: "3.13"
+            sqlserver_version: "2022"
+            msodbc_version: "17"
+            collation: SQL_Latin1_General_CP1_CS_AS
+          # mssql-python stays on the latest Python only.
+          - backend: mssql-python
+            python_version: "3.13"
+            sqlserver_version: "2022"
+            msodbc_version: "18"
+            collation: SQL_Latin1_General_CP1_CS_AS
     runs-on: ubuntu-latest
     container:
-      image: ghcr.io/${{ github.repository }}:CI-${{ matrix.python_version }}-msodbc${{ matrix.msodbc_version }}
+      image: ghcr.io/${{ github.repository }}:CI-${{ matrix.python_version }}-${{ matrix.backend == 'pyodbc' && format('msodbc{0}', matrix.msodbc_version) || 'mssql' }}
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
     services:
       sqlserver:
         image: ghcr.io/${{ github.repository }}:server-${{ matrix.sqlserver_version }}
@@ -63,12 +118,15 @@ jobs:
         run: pip install uv
 
       - name: Install dependencies
-        run: uv pip install --system -e ".[pyodbc]" --group dev
+        env:
+          INSTALL_EXTRA: ${{ matrix.backend == 'pyodbc' && 'pyodbc' || 'mssql' }}
+        run: uv pip install --system -e ".[$INSTALL_EXTRA]" --group dev
 
       - name: Run functional tests
         run: pytest -ra -v tests/functional --profile "ci_sql_server"
         env:
           DBT_TEST_USER_1: DBT_TEST_USER_1
           DBT_TEST_USER_2: DBT_TEST_USER_2
           DBT_TEST_USER_3: DBT_TEST_USER_3
-          SQLSERVER_TEST_DRIVER: 'ODBC Driver ${{ matrix.msodbc_version }} for SQL Server'
+          SQLSERVER_TEST_DRIVER: "ODBC Driver ${{ matrix.msodbc_version }} for SQL Server"
+          SQLSERVER_TEST_BACKEND: ${{ matrix.backend }}

.github/workflows/publish-docker.yml

@@ -1,6 +1,7 @@
 ---
 name: Publish Docker images for CI/CD
 on:  # yamllint disable-line rule:truthy
+  workflow_dispatch:
   push:
     paths:
       - 'devops/**'
@@ -13,7 +14,7 @@ jobs:
     strategy:
       matrix:
         python_version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
-        docker_target: ["msodbc17", "msodbc18"]
+        docker_target: ["msodbc17", "msodbc18", "mssql"]
     runs-on: ubuntu-latest
     permissions:
       contents: read

CONTRIBUTING.md

@@ -32,15 +32,52 @@ The functional tests require a running SQL Server instance. You can easily spin
 make server
 ```
 
+### Backend requirements at a glance
+
+| Backend | Python package | Debian/Ubuntu system packages |
+|---|---|---|
+| `pyodbc` | `dbt-sqlserver[pyodbc]` or `pyodbc` | `unixodbc-dev` plus the Microsoft ODBC Driver for SQL Server |
+| `mssql-python` | `dbt-sqlserver[mssql]` or `mssql-python` | `libltdl7`, `libkrb5-3`, `libgssapi-krb5-2` |
+
+The default development flow uses the ODBC-based path, but the ODBC driver itself is now an optional dependency. If you want to develop or test that backend, install either the adapter extra or the driver itself before running tests.
+
+```shell
+pip install -U "dbt-sqlserver[pyodbc]"
+# or
+pip install -U pyodbc
+```
+
+If you want to develop or test the optional `mssql-python` backend instead, install either the adapter extra or the driver itself before running tests.
+
+```shell
+pip install -U "dbt-sqlserver[mssql]"
+# or
+pip install -U mssql-python
+```
+
+On Debian/Ubuntu-based environments, `mssql-python` may also require these system libraries:
+
+```shell
+sudo apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2
+```
+
 This will use Docker Compose to spin up a local instance of SQL Server. Docker Compose is now bundled with Docker, so make sure to [install the latest version of Docker](https://docs.docker.com/get-docker/).
 
 Next, tell our tests how they should connect to the local instance by creating a file called `test.env` in the root of the project.
 You can use the provided `test.env.sample` as a base and if you started the server with `make server`, then this matches the instance running on your local machine.
 
+If you are testing the optional `mssql-python` backend, also enable its backend setting in `test.env` so the adapter selects that implementation instead of the legacy driver-based one.
+
 ```shell
 cp test.env.sample test.env
 ```
 
+When using the optional `mssql-python` backend, update `test.env` with:
+
+```shell
+SQLSERVER_TEST_BACKEND=mssql-python
+```
+
 You can tweak the contents of this file to test against a different database.
 
 Note that we need 3 users to be able to run tests related to the grants.
@@ -57,6 +94,8 @@ make unit
 make functional
 ```
 
+This remains the documented test procedure for both connection backends. When the `pyodbc` path is enabled, run the same commands after installing `dbt-sqlserver[pyodbc]` or `pyodbc`. When the `mssql-python` backend is enabled, run the same commands after installing `dbt-sqlserver[mssql]` or `mssql-python` and setting `SQLSERVER_TEST_BACKEND=mssql-python` in `test.env`.
+
 ## CI/CD
 
 We use Docker images that have all the things we need to test the adapter in the CI/CD workflows.

README.md

@@ -9,40 +9,87 @@ E.g. version 1.1.x of the adapter will be compatible with dbt-core 1.1.x.
 
 We've bundled all documentation on the dbt docs site:
 
-* [Profile setup & authentication](https://docs.getdbt.com/reference/warehouse-profiles/mssql-profile)
-* [Adapter documentation, usage and important notes](https://docs.getdbt.com/reference/resource-configs/mssql-configs)
+- [Profile setup & authentication](https://docs.getdbt.com/reference/warehouse-profiles/mssql-profile)
+- [Adapter documentation, usage and important notes](https://docs.getdbt.com/reference/resource-configs/mssql-configs)
 
 Join us on the [dbt Slack](https://getdbt.slack.com/archives/CMRMDDQ9W) to ask questions, get help, or to discuss the project.
 
 ## Installation
 
-This adapter requires the Microsoft ODBC driver to be installed:
+The default install uses the `pyodbc` backend and includes the `pyodbc` dependency. If you want the optional `mssql-python` backend instead, install the `mssql` extra.
+
+Latest version: ![PyPI](https://img.shields.io/pypi/v/dbt-sqlserver?label=latest%20stable&logo=pypi)  
+Latest pre-release: ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/dbt-msft/dbt-sqlserver?include_prereleases&label=latest%20pre-release&logo=pypi)
+
+
+### Backend requirements at a glance
+
+| Backend | Python package | Debian/Ubuntu system packages |
+|---|---|---|
+| `pyodbc` | `dbt-sqlserver[pyodbc]` or `pyodbc` | `unixodbc-dev` plus the Microsoft ODBC Driver for SQL Server |
+| `mssql-python` | `dbt-sqlserver[mssql]` or `mssql-python` | `libltdl7`, `libkrb5-3`, `libgssapi-krb5-2` |
+
+
+### `pyodbc` backend
+
+The legacy and currently default ODBC path uses `pyodbc` and the Microsoft ODBC driver.
+
+```shell
+pip install -U dbt-sqlserver
+```
+
+You should migrate to using an explicit extra in preparation for deprecation; the following is equivalent:
+
+```shell
+pip install -U "dbt-sqlserver[pyodbc]"
+```
+
+You also need the Microsoft ODBC driver for SQL Server installed on your system:
 [Windows](https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16#download-for-windows) |
 [macOS](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver16) |
-[Linux](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver16)
+[Linux](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-sql-server?view=sql-server-ver16)
 
 <details><summary>Debian/Ubuntu</summary>
-<p>
 
-Make sure to install the ODBC headers as well as the driver linked above:
+Install the ODBC headers as well as the driver linked above:
 
 ```shell
 sudo apt-get install -y unixodbc-dev
 ```
 
-</p>
 </details>
 
-Latest version: ![PyPI](https://img.shields.io/pypi/v/dbt-sqlserver?label=latest%20stable&logo=pypi)
+### `mssql-python` backend
+
+An alternative backend that does not require the ODBC driver.
 
 ```shell
-pip install -U dbt-sqlserver
+pip install -U "dbt-sqlserver[mssql]"
 ```
 
-Latest pre-release: ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/dbt-msft/dbt-sqlserver?include_prereleases&label=latest%20pre-release&logo=pypi)
+On Debian/Ubuntu-based systems, `mssql-python` requires these system libraries:
 
 ```shell
-pip install -U --pre dbt-sqlserver
+sudo apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2
+```
+
+Enable it per target in your `profiles.yml`:
+
+```yaml
+your_profile:
+  target: dev
+  outputs:
+    dev:
+      type: sqlserver
+      host: your-server
+      port: 1433
+      database: your-database
+      schema: dbo
+      user: your-user
+      password: your-password
+      encrypt: true
+      trust_cert: false
+      backend: mssql-python  # <-- enables this backend
 ```
 
 ## Changelog
@@ -51,8 +98,6 @@ See [the changelog](CHANGELOG.md)
 
 ## Configuration
 
-### Flags
-
 - `dbt_sqlserver_use_default_schema_concat`: *(default: `false`)* Controls schema name generation when a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas) is set on a model.
 
   | Flag value | `custom_schema_name` | Result |
@@ -77,6 +122,31 @@ See [the changelog](CHANGELOG.md)
   > **Note:** If you want to permanently customise schema generation and avoid any future changes, override the `sqlserver__generate_schema_name` macro directly in your project instead.
 
 
+### `dbt_sqlserver_use_default_schema_concat`
+
+*(default: `false`)* Controls schema name generation when a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas) is set on a model.
+
+| Value | `custom_schema_name` | Result |
+|---|---|---|
+| `false` (default) | *(none)* | `target.schema` |
+| `false` (default) | `"reporting"` | `reporting` |
+| `true` | *(none)* | `target.schema` |
+| `true` | `"reporting"` | `target.schema_reporting` |
+
+When `false`, `custom_schema_name` is used as-is without being prefixed by `target.schema`.  
+When `true`, the adapter delegates to dbt-core's `default__generate_schema_name`.
+
+```yaml
+# dbt_project.yml
+vars:
+  dbt_sqlserver_use_default_schema_concat: true
+```
+
+> **Note:** To permanently customise schema generation without a flag dependency, override the `sqlserver__generate_schema_name` macro directly in your project.
+
+### `backend`
+
+*(default: `pyodbc`)* Set to `mssql-python` in a profile target to use the `mssql-python` backend instead of `pyodbc`. The adapter fails if the required backend package (Python dependency), such as `pyodbc` or `mssql-python`, is not installed.
 
 ## Contributing
 
@@ -85,7 +155,7 @@ See [the changelog](CHANGELOG.md)
 [![Integration tests on Azure](https://github.com/dbt-msft/dbt-sqlserver/actions/workflows/integration-tests-azure.yml/badge.svg)](https://github.com/dbt-msft/dbt-sqlserver/actions/workflows/integration-tests-azure.yml)
 
 This adapter is community-maintained.
-You are welcome to contribute by creating issues, opening or reviewing pull requests or helping other users in Slack channel.
+You are welcome to contribute by creating issues, opening or reviewing pull requests, or helping other users in the Slack channel.
 If you're unsure how to get started, check out our [contributing guide](CONTRIBUTING.md).
 
 ## License