Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
133 lines (91 loc) · 5.44 KB

CONTRIBUTING.md

File metadata and controls

133 lines (91 loc) · 5.44 KB
Raw

Development of the adapter

Python 3.10 is used for developing the adapter. To get started, bootstrap your environment as follows:

Create a virtual environment, pyenv is used in the example:

pyenv install 3.10.7
pyenv virtualenv 3.10.7 dbt-sqlserver
pyenv activate dbt-sqlserver

Install the development dependencies and pre-commit and get information about possible make commands:

make dev
make help

Pre-commit helps us to maintain a consistent style and code quality across the entire project. After running make dev, pre-commit will automatically validate your commits and fix any formatting issues whenever possible.

Devcontainer

A devcontainer file has been added since 1.7.2 to simpify creating the development environment.

Testing

The functional tests require a running SQL Server instance. You can easily spin up a local instance with the following command:

make server

Backend requirements at a glance

Backend Python package Debian/Ubuntu system packages
pyodbc dbt-sqlserver[pyodbc] or pyodbc unixodbc-dev plus the Microsoft ODBC Driver for SQL Server
mssql-python dbt-sqlserver[mssql] or mssql-python libltdl7, libkrb5-3, libgssapi-krb5-2

The default development flow uses the ODBC-based path, but the ODBC driver itself is now an optional dependency. If you want to develop or test that backend, install either the adapter extra or the driver itself before running tests.

pip install -U "dbt-sqlserver[pyodbc]"
# or
pip install -U pyodbc

If you want to develop or test the optional mssql-python backend instead, install either the adapter extra or the driver itself before running tests.

pip install -U "dbt-sqlserver[mssql]"
# or
pip install -U mssql-python

On Debian/Ubuntu-based environments, mssql-python may also require these system libraries:

sudo apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2

This will use Docker Compose to spin up a local instance of SQL Server. Docker Compose is now bundled with Docker, so make sure to install the latest version of Docker.

Next, tell our tests how they should connect to the local instance by creating a file called test.env in the root of the project. You can use the provided test.env.sample as a base and if you started the server with make server, then this matches the instance running on your local machine.

If you are testing the optional mssql-python backend, also enable its backend setting in test.env so the adapter selects that implementation instead of the legacy driver-based one.

cp test.env.sample test.env

When using the optional mssql-python backend, update test.env with:

SQLSERVER_TEST_BACKEND=mssql-python

You can tweak the contents of this file to test against a different database.

Note that we need 3 users to be able to run tests related to the grants. The 3 users are defined by the following environment variables containing their usernames.

  • DBT_TEST_USER_1
  • DBT_TEST_USER_2
  • DBT_TEST_USER_3

You can use the following commands to run the unit and the functional tests respectively:

make unit
make functional

This remains the documented test procedure for both connection backends. When the pyodbc path is enabled, run the same commands after installing dbt-sqlserver[pyodbc] or pyodbc. When the mssql-python backend is enabled, run the same commands after installing dbt-sqlserver[mssql] or mssql-python and setting SQLSERVER_TEST_BACKEND=mssql-python in test.env.

CI/CD

We use Docker images that have all the things we need to test the adapter in the CI/CD workflows. The Dockerfile is located in the devops directory and pushed to GitHub Packages to this repo. There is one tag per supported Python version.

All CI/CD pipelines are using GitHub Actions. The following pipelines are available:

  • publish-docker: publishes the image we use in all other pipelines.
  • unit-tests: runs the unit tests for each supported Python version.
  • integration-tests-sqlserver: runs the integration tests for SQL Server.
  • release-version: publishes the adapter to PyPI.

There is an additional Pre-commit pipeline that validates the code style.

Azure integration tests

The following environment variables are available:

  • DBT_AZURESQL_SERVER: full hostname of the server hosting the Azure SQL database
  • DBT_AZURESQL_DB: name of the Azure SQL database
  • DBT_AZURESQL_UID: username of the SQL admin on the server hosting the Azure SQL database
  • DBT_AZURESQL_PWD: password of the SQL admin on the server hosting the Azure SQL database
  • DBT_AZURE_TENANT: Azure tenant ID
  • DBT_AZURE_SUBSCRIPTION_ID: Azure subscription ID
  • DBT_AZURE_RESOURCE_GROUP_NAME: Azure resource group name
  • DBT_AZURE_SP_NAME: Client/application ID of the service principal used to connect to Azure AD
  • DBT_AZURE_SP_SECRET: Password of the service principal used to connect to Azure AD

Releasing a new version

Make sure the version number is bumped in dbt/adapters/sqlserver/__version__.py. Then publish a GitHub Release with a tag named v<version>. A GitHub Actions workflow will be triggered to build the package and push it to PyPI.

If you're releasing support for a new version of dbt-core, also bump the dbt-core constraint in dependencies in pyproject.toml.