feat: add overwrite_src_cols parameter to derived_columns by pkumar-data · Pull Request #474 · ScalefreeCOM/datavault4dbt

pkumar-data · 2026-06-16T13:39:57Z

Adds a new per-derived-column boolean parameter overwrite_src_cols (default: false). When true, the columns listed in src_cols_required are suppressed from the derived_columns CTE SELECT onward, so the source column is replaced by the alias rather than duplicated alongside it. Fixes the case where renaming a column with special characters (e.g. German umlauts on Databricks) resulted in both the original and the alias appearing in all downstream CTEs.

Implemented via a new extract_overwrite_columns() helper macro in stage_processing_macros.sql. The helper is called before derived_columns_datatypes() to avoid the in-place dict mutation that strips custom keys from the columns dict.

Fixes ScalefreeCOM/product_development_milestones#328

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Tested on BigQuery using a stage model with a source column renamed via overwrite_src_cols: true:

dbt compile — verified the derived_columns CTE no longer includes the original source column alongside the alias.
dbt run — view created successfully with the correct output schema (original column absent, alias present).

Test Configuration:

datavault4dbt-Version: release-v2
dbt-Version: core 1.11.7
dbt-adapter-Version: dbt-bigquery 1.11.1

Checklist:

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation or included information that needs updates (e.g. in the Wiki)

Copilot

Pull request overview

Adds support for a new per-derived-column boolean parameter overwrite_src_cols that suppresses specified source columns (from src_cols_required) after the derived_columns CTE, enabling true “rename/replace” behavior and preventing duplicated original+alias columns downstream.

Changes:

Introduces extract_overwrite_columns() helper macro to collect src_cols_required entries for derived columns marked with overwrite_src_cols: true.
Updates adapter-specific stage.sql macros to (a) extract overwrite columns before datatype detection and (b) remove those source columns from final_columns_to_select in the derived_columns CTE.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
macros/staging/stage_processing_macros.sql	Adds `extract_overwrite_columns()` helper macro used by staging to compute which source columns to drop.
macros/staging/bigquery/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/databricks/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/exasol/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/fabric/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/oracle/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/postgres/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/redshift/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/snowflake/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/sqlserver/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/synapse/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.
macros/staging/trino/stage.sql	Extracts overwrite columns and excludes them from `derived_columns` CTE output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Adds a new per-derived-column boolean parameter overwrite_src_cols (default: false). When true, the columns listed in src_cols_required are suppressed from the derived_columns CTE SELECT onward, so the source column is replaced by the alias rather than duplicated alongside it. Fixes the case where renaming a column with special characters (e.g. German umlauts on Databricks) resulted in both the original and the alias appearing in all downstream CTEs. Implemented via a new extract_overwrite_columns() helper macro in stage_processing_macros.sql. The helper is called before derived_columns_datatypes() to avoid the in-place dict mutation that strips custom keys from the columns dict.

…exclusion Databricks normalizes final_columns_to_select to lowercase in the first process_columns_to_select call, so overwrite_src_columns must also be lowercased to ensure case-insensitive matching works correctly.

Initialize overwrite_src_columns = [] before the if block rather than in the else clause. This eliminates the redundant else block, makes the intent clearer (variable is always needed), and follows DRY principle. Applies to all staging macros: bigquery, snowflake, postgres, redshift, oracle, exasol, trino, databricks, sqlserver, synapse, fabric.

tkiehn

Looks good in general, nice and easy implementation!

Lastly please also add documentation for the new parameter in the top-level macro macros/staging/stage.sql and the docs-markdowns.

Documents the new per-column boolean parameter in macros/staging/stage.sql (derived_columns_description) and in the derived-columns markdown guide, including a umlaut-rename example and an OVERWRITE SOURCE COLUMNS section.

tkiehn · 2026-07-13T14:38:17Z

+
+The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are **excluded from the staging CTE output**, so only the derived alias column appears — preventing duplicate columns.
+
+This is particularly useful when **renaming columns whose original name cannot be referenced downstream**, for example columns containing special characters (e.g. German umlauts like `ä`, `ö`, `ü`) on platforms such as Databricks. Without `overwrite_src_cols: true`, both the original and the renamed column would appear side by side in the output.


Suggested change

This is particularly useful when **renaming columns whose original name cannot be referenced downstream**, for example columns containing special characters (e.g. German umlauts like `ä`, `ö`, `ü`) on platforms such as Databricks. Without `overwrite_src_cols: true`, both the original and the renamed column would appear side by side in the output.

tkiehn · 2026-07-13T14:39:23Z

+
+## OVERWRITE SOURCE COLUMNS
+
+The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are **excluded from the staging CTE output**, so only the derived alias column appears — preventing duplicate columns.


Suggested change

The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are **excluded from the staging CTE output**, so only the derived alias column appears — preventing duplicate columns.

The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are **excluded from the staging CTE output**, so only the derived alias column appears. The parameter defaults to `false` when omitted.

tkiehn · 2026-07-13T14:42:01Z

+    account_name_clean:
+        value: Kontonäme
+        datatype: STRING
+        src_cols_required: Kontonäme
+        overwrite_src_cols: true


Suggested change

account_name_clean:

value: Kontonäme

datatype: STRING

src_cols_required: Kontonäme

overwrite_src_cols: true

domain:

value: Domäne

datatype: STRING

src_cols_required: Domäne

overwrite_src_cols: true

tkiehn · 2026-07-13T14:42:20Z

+        overwrite_src_cols: true
+```
+
+In this example, `Kontonäme` is excluded from the staging CTE and only `account_name_clean` appears in the output. The parameter defaults to `false` when omitted.


Suggested change

In this example, `Kontonäme` is excluded from the staging CTE and only `account_name_clean` appears in the output. The parameter defaults to `false` when omitted.

In this example, `Domäne` is excluded from the staging CTE and only `domain` appears in the output.

tkiehn · 2026-07-13T14:44:11Z

+                                            {'account_name_clean': {'value': 'Kontonäme',                                              Renames a source column whose name contains a special character. Setting
+                                                                    'datatype': 'STRING',                                              'overwrite_src_cols: true' ensures the original column is dropped from the
+                                                                    'src_cols_required': 'Kontonäme',                                  CTE output, so only 'account_name_clean' appears and the special-character


Suggested change

{'account_name_clean': {'value': 'Kontonäme', Renames a source column whose name contains a special character. Setting

'datatype': 'STRING', 'overwrite_src_cols: true' ensures the original column is dropped from the

'src_cols_required': 'Kontonäme', CTE output, so only 'account_name_clean' appears and the special-character

{'domain': {'value': 'Domäne', Renames a source column whose name contains a special character. Setting

'datatype': 'STRING', 'overwrite_src_cols: true' ensures the original column is dropped from the

'src_cols_required': 'Domäne', CTE output, so only 'domain' appears and the special-character

pkumar-data linked an issue Jun 16, 2026 that may be closed by this pull request

[FEATURE] Derived column does not replace original #461

Open

tkirschke changed the base branch from release-v2 to main June 17, 2026 07:32

pkumar-data force-pushed the fix/328-derived-columns-overwrite-src-cols branch from 66403e7 to ca5a1e8 Compare June 17, 2026 09:21

tkiehn requested review from Copilot and tkiehn June 17, 2026 10:36

Copilot started reviewing on behalf of tkiehn June 17, 2026 10:36 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread macros/staging/databricks/stage.sql Outdated

tkiehn requested changes Jun 22, 2026

View reviewed changes

Comment thread macros/staging/bigquery/stage.sql Outdated

pkumar-data added 3 commits June 24, 2026 09:27

pkumar-data force-pushed the fix/328-derived-columns-overwrite-src-cols branch from 78ede82 to c429a44 Compare June 24, 2026 07:28

pkumar-data requested a review from tkiehn June 24, 2026 07:41

tkiehn requested changes Jul 6, 2026

View reviewed changes

docs: add overwrite_src_cols documentation for derived_columns

212f037

Documents the new per-column boolean parameter in macros/staging/stage.sql (derived_columns_description) and in the derived-columns markdown guide, including a umlaut-rename example and an OVERWRITE SOURCE COLUMNS section.

pkumar-data force-pushed the fix/328-derived-columns-overwrite-src-cols branch from ac2bfa3 to 212f037 Compare July 7, 2026 13:24

pkumar-data requested a review from tkiehn July 7, 2026 13:30

tkiehn requested changes Jul 13, 2026

View reviewed changes

tkiehn added the waiting-on-response Waiting on a response from the author or others, except maintainers. label Jul 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add overwrite_src_cols parameter to derived_columns#474

feat: add overwrite_src_cols parameter to derived_columns#474
pkumar-data wants to merge 4 commits into
mainfrom
fix/328-derived-columns-overwrite-src-cols

pkumar-data commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

tkiehn left a comment

Uh oh!

tkiehn Jul 13, 2026

Uh oh!

tkiehn Jul 13, 2026

Uh oh!

tkiehn Jul 13, 2026

Uh oh!

tkiehn Jul 13, 2026

Uh oh!

tkiehn Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are excluded from the staging CTE output, so only the derived alias column appears — preventing duplicate columns.

		This is particularly useful when renaming columns whose original name cannot be referenced downstream, for example columns containing special characters (e.g. German umlauts like `ä`, `ö`, `ü`) on platforms such as Databricks. Without `overwrite_src_cols: true`, both the original and the renamed column would appear side by side in the output.


		## OVERWRITE SOURCE COLUMNS

		The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are excluded from the staging CTE output, so only the derived alias column appears — preventing duplicate columns.

	In this example, `Kontonäme` is excluded from the staging CTE and only `account_name_clean` appears in the output. The parameter defaults to `false` when omitted.
	In this example, `Domäne` is excluded from the staging CTE and only `domain` appears in the output.

Uh oh!

Conversation

pkumar-data commented Jun 16, 2026

Type of change

How Has This Been Tested?

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

tkiehn left a comment

Choose a reason for hiding this comment

Uh oh!

tkiehn Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

tkiehn Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

tkiehn Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

tkiehn Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

tkiehn Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants