Skip to content

CHANGELOG.md

Latest commit

 

History

History
579 lines (380 loc) · 33.3 KB

CHANGELOG.md

File metadata and controls

579 lines (380 loc) · 33.3 KB
Raw

Changelog

v1.10.1

Features

  • Add Postgres-style indexes model config for tables, incrementals, seeds and snapshots, covering most CREATE INDEX options. #535
  • Index names are deterministic definition hashes (dbt_idx_ prefix); creation is idempotent and unchanged definitions are never rebuilt.
  • Reconcile indexes against the config on incremental, DML-refresh and snapshot runs, applied as one atomic batch. Constraint-backing, legacy post-hook and as_columnstore indexes are never dropped.
  • Index introspection reads the catalog lock-free throughout (NOLOCK on every sys view, not just sys.indexes), so it no longer queues behind concurrent index DDL.
  • Add drop_unmanaged_indexes config (false (default) / warn / true) for indexes dbt didn't create.
  • Validate cross-index config conflicts (multiple clustered indexes, clustered vs as_columnstore).
  • Document the minimum supported SQL Server version (2017). Partitioning, XML_COMPRESSION and ordered columnstore are not yet expressible in the indexes config.
  • Add SQL Server 2025 to the integration-test matrix (pyodbc and mssql-python backends, ODBC Driver 18) and document it as a supported version.
  • Add dbt_sqlserver_enable_safe_type_expansion behaviour flag to allow safe column type widening during schema expansion: varchar → nvarchar, integer family promotions (bit → tinyint → smallint → int → bigint), and numeric/decimal precision/scale upgrades. Gated by the per-model column_type_expansion_max_rows config (default 1,000,000 rows). #699.
  • Add prefer_single_alter_column model config to use a single ALTER COLUMN statement instead of the add+update+drop+rename pattern when altering column types on tables.
  • Add string_type_instance() to preserve the NVARCHAR/NCHAR type family during column expansion, fixing incorrect promotion of NVARCHAR/NCHAR to VARCHAR.
  • Add tinyint and bit to the is_integer() type list for correct type detection.
  • Add dbt_sqlserver_use_dbt_transactions flag for proper T-SQL BEGIN TRAN/COMMIT/ROLLBACK transaction handling. When enabled, operations use real server-side transactions instead of the default autocommit mode. #708
  • Implement relation and column comment persistence via persist_docs. #289
  • Add optional mssql-python connection backend alongside the default pyodbc backend. #681

Bugfixes

  • Fix unit tests with empty fixtures (rows: []) generating invalid limit 0 syntax; emit top 0 instead. Also fix get_columns_in_query() for queries starting with a CTE, which broke unit tests with an empty expect block; such queries are now described via sp_describe_first_result_set instead of being executed. #698
  • Fix catalog generation for NVARCHAR/NCHAR columns: use user_type_id instead of system_type_id in catalog.sql, preventing them from appearing as SYSNAME in dbt docs. #637
  • Fix varchar(max) / nvarchar(max) columns being incorrectly treated as size -1 during type expansion, preventing varchar(max) → varchar(100) narrowing and properly allowing varchar(100) → varchar(max) expansion.
  • Fix seed table ingestion of empty numeric cells by inlining null literals instead of binding parameters. #425
  • Guard run_hooks commit with @@trancount check for autocommit safety when running post-hooks. #444
  • Fix columnstore IF EXISTS guard to check object_id('schema.table') correctly.
  • Gate the optimize_for_sequential_key and resumable index options on the detected engine version: they require SQL Server 2019+, so on 2017/2016 the index is now built without them (with a warning) instead of failing with "is not a recognized CREATE INDEX option".
  • Escape single quotes in query_tag before building OPTION (LABEL) clause.
  • Map Python float to SQL Server float, not bigint.
  • Set default port to 1433 (instead of Postgres 5432) in dbt init profile template.
  • Make drop_schema_name usable again. #563
  • Raise DbtRuntimeError instead of ValueError for missing access token.

Under the hood

  • Consolidate linting tooling (isort, flake8, pycln, absolufy-imports) into Ruff. #707
  • Drop Python 3.9 from publish-docker CI matrix.
  • Remove dead clone macro and unused view scaffolding variable.
  • Standardise devcontainer ODBC environment setup. #722
  • Disable dbt telemetry and version check in test environment.
  • Add pytest-cov and fix dead from_dict override found by coverage.
  • Bump pip and GitHub Actions dependencies.

v1.10.0

Features

  • Official support for dbt-core 1.10.
  • Add query_options / query_options_raw model configs for emitting SQL Server OPTION clauses on table, incremental (delete+insert / microbatch), snapshot, and unit_test materializations. See dbt-msft#613.
  • get_query_options() is the new extension point for customising the emitted OPTION clause.
  • Add DML table refresh support for table materializations.
  • Add opt-in native string type mappings via a behaviour flag.
  • Add default schema concatenation flag support and update the documentation.
  • Enable SQL Server limited-relation no-alias behavior by default.
  • Support catalog generation across multiple databases. #603
  • Fix view rematerialization when a view already exists, including preserving grants and avoiding unnecessary rebuilds. #610

Bugfixes

  • Fix reserved-keyword quoting in table-create DROP VIEW handling.
  • Fix CTE detection in empty-subquery wrapping when leading comments are present.
  • Fix snapshot meta column name overrides on a second run.
  • Fix connection-string port handling regressions.
  • Fix TABLOCK interaction with contract-enforced inserts and query_options.
  • Fix get_view_definition() escaping for ] characters.
  • Make the highest-frequency catalog queries sargable so they seek instead of scan. #686

Under the hood

  • Drop Python 3.9 support; this release targets Python 3.10 and newer.
  • Adopt standard pyproject.toml/PEP 621 packaging metadata for source installs and downloads.

Migration note

  • apply_label() is preserved as a callable alias (emits LABEL only) in case you use it in your own project but is no longer called by adapter macros. Projects that override apply_label() to customise the OPTION clause must override get_query_options() instead.

v1.9.2

  • Add default schema concatenation flag support and update the documentation. See PR #685.

v1.9.1

  • Removes the dependency on dbt-fabric.

v1.9.0

  • Update to support dbt-core 1.9.
  • Remove Python 3.8 from official support.

v1.8.7

  • Bump version for release 1.8.7.

v1.8.6

  • Bump version for release 1.8.6.

v1.8.5

  • Fix broken imports from Fabric.

v1.8.4

  • Minor fix to tests with CTEs which do not start on first line. dbt-msft#560

v1.8.3

v1.8.2

  • Restores no lock behaviour from 1.4.2

v1.8.1

  • Fixes problem where databases with dashes are rendered properly

v1.8.0

Updates dbt-sqlserver to support dbt 1.8.

Notable changes

  • Adopts dbt-common and dbt-adapters as the upstream, in line with dbt projects.
  • Implements the majority of the tests from the dbt-test-adapters project to provide better coverage.
  • Implements better testing for dbt-sqlserver specific functions, including indexes.
  • Realigns to closer to the global project, overriding some fabric specific implementations
  • Adds new 1.8 features (and tests), including Unit Tests

Update also fixes a number of regressions related to the fabric adapter and 1.7 releases. These include

There is a number of other changes as well, which can be found in the 1.8.0rc1, 1.8.0rc2 and 1.8.0 release notes.

While not directly included, credit also to @ms32035, @axellpadilla, @gbarrington, @tkirschke and others for their help with testing.

v1.7.2

Huge thanks to GitHub users @cody-scott and @prescode for help with this long-awaited update to enable dbt-core 1.7.2 compatibility!

Updated to use dbt-fabric as the upstream adapter (https://github.com/dbt-msft/dbt-sqlserver/issues/441#issuecomment-1815837171)[https://github.com/dbt-msft/dbt-sqlserver/issues/441#issuecomment-1815837171] and (https://github.com/microsoft/dbt-fabric/issues/105)[https://github.com/microsoft/dbt-fabric/issues/105]

As the fabric adapter implements the majority of auth and required t-sql, this adapter delegates primarily to SQL auth and SQL Server specific adaptations (using SELECT INTO vs CREATE TABLE AS).

Additional major changes pulled from fabric adapter:

  • TIMESTAMP changing from DATETIMEOFFSET to DATETIME2(6)
  • STRING changing from VARCHAR(MAX) to VARCHAR(8000)

Future work to be validated

  • Fabric specific items that need further over-rides (clone for example needed overriding)
  • Azure Auth elements to be deferred to Fabric, but should be validated
  • T-SQL Package to be updated and validated with these changes.
  • Integration of newer dbt-core features.

v1.4.3

Another minor release to follow up on the 1.4 releases.

Replacing the usage of the dm_sql_referencing_entities stored procedure with a query to sys.sql_expression_dependencies for better compatibility with child adapters.

v1.4.2

Minor release to follow up on 1.4.1 and 1.4.0.

Adding nolock to information_schema and sys tables/views can be overridden with the dispatched information_schema_hints macro. This is required for adapters inheriting from this one.

v1.4.1

This is a minor release following up on 1.4.0 with fixes for long outstanding issues. Contributors to this release are @cbini, @rlshuhart, @jacobm001, @baldwicc and @sdebruyn.

Features

  • Added support for a custom schema owner. You can now add schema_authorization (or schema_auth) to your profile. If you do so, dbt will create schemas with the authorization option suffixed by this value. If you are authorizing dbt users or service principals on Azure SQL based on an Azure AD group, it's recommended to set this value to the name of the group. #153 #382
  • Documentation: added more information about the permissions which you'll need to grant to run dbt.
  • Support for DATETIMEOFFSET as type to be used in dbt source freshness tests. #254 #346
  • Added 2 options related to timeouts to the profile: login_timeout and query_timeout. The default values are 0 (no timeout). #162 #395

Bugfixes

  • Fixed issues with databases with a case-sensitive collation and added automated testing for it so that we won't break it again. #212 #391
  • Index names are now MD5 hashed to avoid running into the maximum amount of characters in index names with index with lots of columns with long names. #317 #386
  • Fixed the batch size calculation for seeds. Seeds will run more efficiently now. #396 #179 #210 #211
  • Added nolock to queries for all information_schema/sys tables and views. dbt runs a lot of queries on these metadata schemas. This can often lead to deadlock issues if you are using a high number of threads or dbt processes. Adding nolock to these queries avoids the deadlocks. #379 #381
  • Fixed implementation of {{ hash(...) }} for null values. #392

Under the hood

  • Fixed more concurrency issues with automated Azure integration testing.
  • Removed extra __init__.py files. #171 #202
  • Added commits to be ignored in git blame for easier blaming. #385

v1.4.0

Features

Bugfixes

Under the hood

  • Fixed lots of testing concurrency issues
  • Added all available tests as of dbt 1.4.6

Full Changelog: https://github.com/dbt-msft/dbt-sqlserver/compare/v1.3.1...v1.4.0

PR changelog

1.3.1

Minor release to loosen dependency on dbt-core and pyodbc

v1.3.0

Features

  • Support for dbt-core 1.3
    • Python models are currently not supported in this adapter
    • The following cross-db macros are not supported in this adapter: bool_or, array_construct, array_concat, array_append

Fixes

  • The macro type_boolean now returns the correct data type (bit)

Chores

  • Update adapter testing framework
  • Update dependencies and pre-commit hooks

v1.2.0

Possibly breaking change: connection encryption

For compatibility with MS ODBC Driver 18, the settings Encrypt and TrustServerCertificate are now always added to the connection string. These are configured with the keys encrypt and trust_cert in your profile. In previous versions, these settings were only added if they were set to True.

The new version of the MS ODBC Driver sets Encrypt to True by default. The adapter is following this change and also defaults to True for Encrypt.

The default value for TrustServerConnection remains False as it would be a security risk otherwise.

This means that connections made with this version of the adapter will now have Encrypt=Yes and TrustServerCertificate=No set if you are using the default settings. You should change the settings encrypt or trust_cert to accommodate for your use case.

Features

  • Support for dbt-core 1.2
    • Full support for the new grants config
    • New configuration option: auto_provision_aad_principals - setting this to true will automatically create contained database users linked to Azure AD principals or groups if they don't exist yet when they're being used in grant configs
  • Support for MS ODBC Driver 18
  • Support automatic retries with new retries setting introduced in core
  • The correct owner of a table/view is now visible in generated documentation (and in catalog.json)
  • A lot of features of dbt-utils & T-SQL utils are now available out-of-the-box in dbt-core and this adapter. A new release of T-SQL utils will follow.
    • Support for all type_* macros
    • Support for all cross-database macros, except:
    • bool_or
    • listagg will only work in SQL Server 2017 or newer or the cloud versions. The limit_num option is unsupported. DISTINCT cannot be used in the measure.

Fixes

  • In some cases the TIMESTAMP would be used as data type instead of DATETIMEOFFSET, fixed that

Chores

  • Update adapter testing framework to 1.2.1
  • Update pre-commit, tox, pytest and pre-commit hooks
  • Type hinting in connection class
  • Automated testing with SQL Server 2017, 2019 and 2022
  • Automated testing with MS ODBC 17 and MS ODBC 18

v1.1.0

See changes included in v1.1.0rc1 below as well

Fixes

Chores

v1.1.0.rc1

Features

  • update to dbt 1.1

Fixes

  • #194 uppercased information_schema (@TrololoLi)
  • #215 Escape schema names so they can contain strange characters (@johnf)

Chores

  • Documentation on how to contribute to the adapter
  • Automatic release process by adding a new tag
  • Consistent code style with pre-commit
  • #201 use new dbt 1.0 logger (@semcha)
  • #216 use new dbt testing framework (@dataders & @sdebruyn)

v1.0.0

Please see dbt-core v1.0.0 release notes for upstream changes

Fixes

  • fix index naming when columns contain spaces #175

Under the Hood

  • re-organize macros to match new structure #184

v0.21.1

features

  • Added support for more authentication methods: automatic, environment variables, managed identity. All of them are documented in the readme. #178 contributed by @sdebruyn

fixes

  • fix for #186 and #177 where new columns weren't being added when snapshotting or incrementing #188

v0.21.0

Please see dbt-core v0.21.0 release notes for upstream changes

fixes

  • in dbt-sqlserver v0.20.0, users couldn't use some out of the box tests, such as accepted_values. users can now also use CTEs in their bespoke custom data tests
  • fixes issue with changing column types in incremental table column type #152 #169
  • workaround for Azure CLI token expires after one hour. Now we get new tokens for every transaction. #156 #158

v0.20.1

fixes:

  • workaround for Azure CLI token expires after one hour. Now we get new tokens for every transaction. #156 #158

v0.20.0

features:

  • dbt-sqlserver will now work with dbt v0.20.0. Please see dbt's upgrading to v0.20.0 docs for more info.
  • users can now declare a custom max_batch_size in the project configuration to set the batch size used by the seed file loader. #127 and #151 thanks @jacobm001

under the hood

  • sqlserver__load_csv_rows now has a safety provided by calc_batch_size() to ensure the insert statements won't exceed SQL Server's 2100 parameter limit. #127 and #151 thanks @jacobm001
  • switched to using a MANIFEST.in to declare which files should be included
  • updated pyodbc and azure-identity dependencies to their latest versions

v0.19.2

fixes

  • fixing and issue with empty seed table that dbt-redshift already addressed with fishtown-analytics/dbt#2255 #147
  • drop unneeded debugging code that only was run when "Active Directory integrated" was given as the auth method #149
  • hotfix for regression introduced by #126 that wouldn't surface syntax errors from the SQL engine #140 thanks @jeroen-mostert!

under the hood:

  • ensure that macros are not recreated for incremental models #116 thanks @infused-kim
  • authentication now is case-insensitive and accepts both CLI and cli as options. #100 thanks (@JCZuurmond)[https://github.com/JCZuurmond]
  • add unit tests for azure-identity related token fetching

v0.19.1

features:

  • users can now declare a model's database to be other than the one specified in the profile. This will only work for on-premise SQL Server and Azure SQL Managed Instance. #126 thanks @semcha!

under the hood

  • abandon four-part version names (v0.19.0.2) in favor of three-part version names because it isn't SemVer and it causes problems with the ~= pip operator used dbt-synapse, a package that depends on dbt-sqlserver
  • allow CI to work with the lower-cost serverless Azure SQL #132

v0.19.0.2

fixes

  • solved a bug in snapshots introduced in v0.19.0. Fixes: #108, #117.

v0.19.0.1

fixes

  • we now use the correct connection string parameter so MSFT can montior dbt adoption in their telemetry. #98

under the hood

  • dbt-sqlserver's incremental materialization is now 100% aligneed logically to dbt's global_project behavior! this makes maintaining dbt-sqlserver easier by decreasing code footprint. #102
  • clean up CI config and corresponding Docker image #122

v0.19.0

New Features:

  • dbt-sqlserver's snapshotting now 100% aligneed logically to dbt's snapshotting behavior! Users can now snapshot 'hard-deleted' record as mentioned in the dbt v0.19.0 release notes. An added benefit is that it makes maintaining dbt-sqlserver by decreasing code footprint. #81 fishtown-analytics/dbt#3003

Fixes:

  • small snapshot bug addressed via #81
  • support for clustered columnstore index creation pre SQL Server 2016. #88 thanks @alangsbo
  • support for scenarios where the target db's collation is different than the server's #87 @alangsbo

Under the hood:

  • This adapter has separate CI tests to ensure all the connection methods are working as they should #75
  • This adapter has a CI job for running unit tests #103
  • Update the tox setup #105

v0.18.1

New Features:

Adds support for:

  • SQL Server down to version 2012
  • authentication via:
  • using a named instance (#51 thanks @alangsbo)
  • Adds support down to SQL Server 2012
  • The adapter is now automatically tested with Fishtowns official adapter-tests to increase stability when making changes and upgrades to the adapter.

Fixes:

  • Fix for lack of precision in the snapshot check strategy. Previously when executing two check snapshots the same second, there was inconsistent data as a result. This was mostly noted when running the automatic adapter tests. NOTE: This fix will create a new snapshot version in the target table on first run after upgrade.

v0.18.0.1

New Features:

  • Adds support for Azure Active Directory as authentication provider

Fixes:

  • Fix for lack of precision in the snapshot check strategy. (#74 and #56 thanks @qed) Previously when executing two check snapshots the same second, there was inconsistent data as a result. This was mostly noted when running the automatic adapter tests. NOTE: This fix will create a new snapshot version in the target table on first run after upgrade.
  • #52 Fix deprecation warning (Thanks @jnoynaert)

Testing

  • The adapter is now automatically tested with Fishtowns official adapter-tests to increase stability when making changes and upgrades to the adapter. (#62 #64 #69 #74)
  • We are also now testing specific target configs to make the devs more confident that everything is in working order (#75)

v0.18.0

New Features:

  • Adds support for dbt v0.18.0

v0.15.3.1

Fixes:

  • Snapshots did not work on dbt v0.15.1 to v0.15.3

v0.15.3

Fixes:

  • Fix output of sql in the log files.
  • Limited the version of dbt to 0.15, since later versions are unsupported.

v0.15.2

Fixes:

  • Fixes an issue with clustered columnstore index not being created.

v0.15.1

New Features:

  • Ability to define an index in a poosthook

Fixes:

  • Previously when a model run was interrupted unfinished models prevented the next run and you had to manually delete them. This is now fixed so that unfinished models will be deleted on next run.

v0.15.0.1

Fix release for v0.15.0

Fixes:

  • Setting the port had no effect. Issue #9
  • Unable to generate docs. Issue #12

v0.15.0

Requires dbt v0.15.0 or greater

pre v0.15.0

Requires dbt v0.14.x