Skip to content

add ghost records to eff sat#485

Open
umattern wants to merge 2 commits into
mainfrom
fix/add-ghost-records-to-effectivity-sat
Open

add ghost records to eff sat#485
umattern wants to merge 2 commits into
mainfrom
fix/add-ghost-records-to-effectivity-sat

Conversation

@umattern

@umattern umattern commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Description

Should only insert two additional Ghost Records to an effectivity Sat.

Would be nice it someone could verify it from a technical and logical perspective. Especially all other adapters than Snowflake.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the effectivity satellite (eff_sat_v0) macros across multiple adapters to ensure ghost records (beginning/end-of-all-times ldts) are inserted into the target eff sat while also preventing those ghost rows from interfering with incremental “current status” logic.

Changes:

  • Exclude beginning/end-of-all-times src_ldts rows from current_status selection so they can’t become the “latest” status in incremental runs.
  • Add a final UNION/UNION ALL branch to records_to_insert that re-includes the ghost rows (selected directly from the source_relation by boundary src_ldts).

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
macros/tables/trino/eff_sat_v0.sql Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert.
macros/tables/synapse/eff_sat_v0.sql Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert.
macros/tables/sqlserver/eff_sat_v0.sql Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert.
macros/tables/snowflake/eff_sat_v0.sql Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert.
macros/tables/redshift/eff_sat_v0.sql Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert.
macros/tables/postgres/eff_sat_v0.sql Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert.
macros/tables/oracle/eff_sat_v0.sql Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert.
macros/tables/fabric/eff_sat_v0.sql Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert.
macros/tables/exasol/eff_sat_v0.sql Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert.
macros/tables/databricks/eff_sat_v0.sql Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert.
macros/tables/bigquery/eff_sat_v0.sql Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread macros/tables/snowflake/eff_sat_v0.sql
Comment thread macros/tables/redshift/eff_sat_v0.sql
Comment thread macros/tables/sqlserver/eff_sat_v0.sql
Comment thread macros/tables/synapse/eff_sat_v0.sql
Comment thread macros/tables/postgres/eff_sat_v0.sql
Comment thread macros/tables/databricks/eff_sat_v0.sql
Comment thread macros/tables/exasol/eff_sat_v0.sql
Comment thread macros/tables/trino/eff_sat_v0.sql
Comment thread macros/tables/bigquery/eff_sat_v0.sql
Comment thread macros/tables/oracle/eff_sat_v0.sql
tkiehn
tkiehn previously approved these changes Jul 13, 2026

@tkiehn tkiehn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as discussed with @tkirschke and @umattern

@tkiehn tkiehn added testing To trigger the automated test workflow as internal User. waiting-on-maintainers Waiting for a response from the maintainers. labels Jul 13, 2026
@remoteworkflow

This comment was marked as outdated.

@remoteworkflow remoteworkflow Bot removed the testing To trigger the automated test workflow as internal User. label Jul 13, 2026
@remoteworkflow

Copy link
Copy Markdown

dbt test combined result: ❌


Details

RESULTS for Synapse:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Postgres:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for BigQuery:
❌ dbt-core-tests
❌ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Redshift:
⚠️ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Snowflake:
❌ dbt-core-tests
❌ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Exasol:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Fabric:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Oracle:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Databricks:
❌ dbt-core-tests
❌ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for SQL Server:
❌ dbt-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


RESULTS for Trino:
❌ dbt-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests


Link to workflow summary: https://github.com/ScalefreeCOM/datavault4dbt-ci-cd/actions/runs/29250957277

@tkiehn tkiehn self-requested a review July 13, 2026 14:51
@tkiehn tkiehn dismissed their stale review July 13, 2026 14:52

Waiting for adjusted test-cases

@tkiehn tkiehn added waiting-on-response Waiting on a response from the author or others, except maintainers. and removed waiting-on-maintainers Waiting for a response from the maintainers. labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-response Waiting on a response from the author or others, except maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants