Merge pull request #702 from dbt-msft/fix/missing-type-handling

feat: safe type expansion, NVARCHAR/NCHAR catalog fix, seed empty cell fix

Author: axellpadilla

CHANGELOG.md

@@ -2,9 +2,19 @@
 
 ### v1.10.1
 
+#### Features
+
+- Add `dbt_sqlserver_enable_safe_type_expansion` behaviour flag to allow safe column type widening during schema expansion: `varchar` → `nvarchar`, integer family promotions (`bit` → `tinyint` → `smallint` → `int` → `bigint`), and `numeric`/`decimal` precision/scale upgrades. Gated by the per-model `column_type_expansion_max_rows` config (default 1,000,000 rows). See [#699](https://github.com/dbt-msft/dbt-sqlserver/issues/699).
+- Add `prefer_single_alter_column` model config to use a single `ALTER COLUMN` statement instead of the add+update+drop+rename pattern when altering column types on tables.
+- Add `string_type_instance()` to preserve the NVARCHAR/NCHAR type family during column expansion, fixing incorrect promotion of NVARCHAR/NCHAR to VARCHAR.
+- Add `tinyint` and `bit` to the `is_integer()` type list for correct type detection.
+
 #### Bugfixes
 
 - Fix unit tests with empty fixtures (`rows: []`) generating invalid `limit 0` syntax; emit `top 0` instead. Also fix `get_columns_in_query()` for queries starting with a CTE, which broke unit tests with an empty `expect` block; such queries are now described via `sp_describe_first_result_set` instead of being executed. [#698](https://github.com/dbt-msft/dbt-sqlserver/issues/698)
+- Fix catalog generation for NVARCHAR/NCHAR columns: use `user_type_id` instead of `system_type_id` in catalog.sql, preventing them from appearing as `SYSNAME` in `dbt docs`. [#637](https://github.com/dbt-msft/dbt-sqlserver/issues/637)
+- Fix `varchar(max)` / `nvarchar(max)` columns being incorrectly treated as size `-1` during type expansion, preventing `varchar(max)` → `varchar(100)` narrowing and properly allowing `varchar(100)` → `varchar(max)` expansion.
+- Fix seed table ingestion of empty numeric cells by inlining `null` literals instead of binding parameters. [#425](https://github.com/dbt-msft/dbt-sqlserver/issues/425)
 
 #### Features
 

README.md

@@ -129,6 +129,18 @@ The same setting is also honoured via `vars:` for backwards compatibility; the b
 
 *(default: `pyodbc`)* Set to `mssql-python` in a profile target to use the `mssql-python` backend instead of `pyodbc`. The adapter fails if the required backend package (Python dependency), such as `pyodbc` or `mssql-python`, is not installed.
 
+### `dbt_sqlserver_enable_safe_type_expansion`
+
+*(default: `false`)* When enabled, allows the adapter to widen column types during incremental model schema expansion beyond same-family string resizes. Supported safe expansions include:
+
+- **Cross-family string**: `varchar`/`char` → `nvarchar`/`nchar` (same or larger size)
+- **Integer family**: `bit` → `tinyint` → `smallint` → `int` → `bigint`
+- **Integer → numeric**: `int` → `numeric` (with sufficient precision to hold the integer range)
+- **Numeric precision/scale**: `numeric(p,s)` → `numeric(p2,s2)` where precision and scale both increase
+- **Fixed-money**: `smallmoney` → `money`, `money` → `numeric` (with sufficient precision)
+
+Safe expansions are further gated by `column_type_expansion_max_rows` (default 1,000,000 rows) to avoid long-running operations on large tables.
+
 ### `dbt_sqlserver_use_dbt_transactions`
 
 _(default: `false`)_ When enabled, makes dbt's transaction hooks real at the SQL Server level by emitting `BEGIN TRANSACTION` / `COMMIT TRANSACTION` through the adapter's `add_begin_query` and `add_commit_query` methods. 
@@ -142,7 +154,28 @@ This mode is opt-in and should be tested carefully with project-specific materia
 ```yaml
 # dbt_project.yml
 flags:
-    dbt_sqlserver_use_dbt_transactions: true # <-- opt-in; default is false
+  dbt_sqlserver_enable_safe_type_expansion: true
+  dbt_sqlserver_use_dbt_transactions: true # <-- opt-in; default is false
+```
+
+### `column_type_expansion_max_rows`
+
+*(default: `1000000`)* Per-model config that limits when safe type expansion runs. When the target table exceeds this row count, safe type expansion is skipped (basic same-family string resizes still proceed). Set to `-1` to disable the check entirely.
+
+```sql
+-- In an incremental model
+{{ config(materialized='incremental', unique_key='id',
+           column_type_expansion_max_rows=500000) }}
+```
+
+### `prefer_single_alter_column`
+
+*(default: `false`)* Model-level config that controls how `alter_column_type` changes column types on tables. When `false` (default), the adapter uses the safer approach: add a temporary column, copy data, drop the original, and rename. When `true`, the adapter uses a single `ALTER COLUMN` statement, which is faster on small, medium tables and instant on safe type expansions but may fail for types that cannot be implicitly converted.
+
+```sql
+-- In an incremental model
+{{ config(materialized='incremental', unique_key='id',
+           prefer_single_alter_column=true) }}
 ```
 
 **Compatibility notes:** Enabling `dbt_sqlserver_use_dbt_transactions: true` may expose transaction-state assumptions hidden by autocommit-only mode. Explicit transaction macros may interact with dbt-managed transactions, and cleanup after failed DDL/DML may differ. Review pre/post hooks for in-transaction vs out-of-transaction semantics.

dbt/adapters/sqlserver/sqlserver_adapter.py

@@ -15,7 +15,8 @@
 from dbt.adapters.base.meta import available
 from dbt.adapters.base.relation import BaseRelation
 from dbt.adapters.capability import Capability, CapabilityDict, CapabilitySupport, Support
-from dbt.adapters.events.types import SchemaCreation
+from dbt.adapters.events.logging import AdapterLogger
+from dbt.adapters.events.types import ColTypeChange, SchemaCreation
 from dbt.adapters.reference_keys import _make_ref_key_dict
 from dbt.adapters.relation_configs import RelationConfigChangeAction
 from dbt.adapters.sql.impl import CREATE_SCHEMA_MACRO_NAME, SQLAdapter
@@ -30,6 +31,8 @@
 from dbt.adapters.sqlserver.sqlserver_connections import SQLServerConnectionManager
 from dbt.adapters.sqlserver.sqlserver_relation import SQLServerRelation
 
+logger = AdapterLogger("SQLServer")
+
 
 class SQLServerAdapter(SQLAdapter):
     """
@@ -111,6 +114,16 @@ def _behavior_flags(self) -> List[BehaviorFlag]:
                     "The new behaviour is intended to become the default in a future release."
                 ),
             },
+            {
+                "name": "dbt_sqlserver_enable_safe_type_expansion",
+                "default": False,
+                "description": (
+                    "Allow the SQL Server adapter to widen column types during schema expansion. "
+                    "This enables promotions like varchar -> nvarchar, "
+                    "bit -> tinyint -> smallint -> int -> bigint, "
+                    "and numeric(p,s) -> numeric(p2,s2) using alter column."
+                ),
+            },
             {
                 "name": "dbt_sqlserver_use_dbt_transactions",
                 "default": False,
@@ -312,6 +325,101 @@ def render_model_constraint(cls, constraint: ModelLevelConstraint) -> Optional[s
         else:
             return None
 
+    def _get_row_count(self, relation) -> int:
+        """Return the number of rows in the given relation."""
+        sql = f"SELECT COUNT_BIG(*) FROM {relation}"
+        _, cursor = self.connections.add_select_query(sql)
+        row = cursor.fetchone()
+        return int(row[0]) if row else 0
+
+    def expand_column_types(self, goal, current, max_rows: int = 1000000):
+        """Override to ensure we preserve nvarchar/nchar type family during
+        column expansion. Necessary same-family resizes (e.g. varchar size)
+        always proceed. Safe type expansions (cross-family promotions like
+        varchar -> nvarchar) are guarded by column_type_expansion_max_rows.
+        enable_safe_type_expansion is the future approach for widening."""
+
+        reference_columns = {c.name: c for c in self.get_columns_in_relation(goal)}
+        target_columns = {c.name: c for c in self.get_columns_in_relation(current)}
+
+        enable_safe = self.behavior.dbt_sqlserver_enable_safe_type_expansion
+
+        row_count_exceeds = False
+        if enable_safe and max_rows != -1:
+            if max_rows == 0:
+                row_count_exceeds = True
+                logger.info(
+                    "Safe type expansion skipped for %s: column_type_expansion_max_rows is 0.",
+                    current,
+                )
+            else:
+                row_count = self._get_row_count(current)
+                if row_count > max_rows:
+                    row_count_exceeds = True
+                    logger.warning(
+                        "Safe type expansion skipped for %s: "
+                        "%s rows exceeds column_type_expansion_max_rows (%s). "
+                        "Set column_type_expansion_max_rows=-1 to disable "
+                        "this check, or increase the limit.",
+                        current,
+                        row_count,
+                        max_rows,
+                    )
+
+        for column_name, reference_column in reference_columns.items():
+            target_column = target_columns.get(column_name)
+            if target_column is None:
+                continue
+
+            if target_column.can_expand_to(reference_column):
+                pass
+            elif (
+                enable_safe
+                and not row_count_exceeds
+                and target_column.can_expand_safe(reference_column)
+            ):
+                pass
+            else:
+                continue
+
+            if reference_column.is_string():
+                col_string_size = reference_column.string_size()
+                new_type = reference_column.string_type_instance(col_string_size)
+            else:
+                new_type = reference_column.data_type
+            fire_event(
+                ColTypeChange(
+                    orig_type=target_column.data_type,
+                    new_type=new_type,
+                    table=_make_ref_key_dict(current),
+                )
+            )
+            self.alter_column_type(current, column_name, new_type)
+
+    @available.parse_none
+    def expand_target_column_types(
+        self, from_relation: BaseRelation, to_relation: BaseRelation, max_rows: int = 1000000
+    ) -> None:
+        if not isinstance(from_relation, self.Relation):
+            from dbt.adapters.base.impl import MacroArgTypeError
+
+            raise MacroArgTypeError(
+                method_name="expand_target_column_types",
+                arg_name="from_relation",
+                got_value=from_relation,
+                expected_type=self.Relation,
+            )
+        if not isinstance(to_relation, self.Relation):
+            from dbt.adapters.base.impl import MacroArgTypeError
+
+            raise MacroArgTypeError(
+                method_name="expand_target_column_types",
+                arg_name="to_relation",
+                got_value=to_relation,
+                expected_type=self.Relation,
+            )
+        self.expand_column_types(from_relation, to_relation, max_rows)
+
     @available
     def parse_index(self, raw_index: Any) -> Optional[SQLServerIndexConfig]:
         return SQLServerIndexConfig.parse(raw_index)