add ghost records to eff sat#485
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adjusts the effectivity satellite (eff_sat_v0) macros across multiple adapters to ensure ghost records (beginning/end-of-all-times ldts) are inserted into the target eff sat while also preventing those ghost rows from interfering with incremental “current status” logic.
Changes:
- Exclude beginning/end-of-all-times
src_ldtsrows fromcurrent_statusselection so they can’t become the “latest” status in incremental runs. - Add a final
UNION/UNION ALLbranch torecords_to_insertthat re-includes the ghost rows (selected directly from thesource_relationby boundarysrc_ldts).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| macros/tables/trino/eff_sat_v0.sql | Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert. |
| macros/tables/synapse/eff_sat_v0.sql | Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert. |
| macros/tables/sqlserver/eff_sat_v0.sql | Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert. |
| macros/tables/snowflake/eff_sat_v0.sql | Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert. |
| macros/tables/redshift/eff_sat_v0.sql | Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert. |
| macros/tables/postgres/eff_sat_v0.sql | Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert. |
| macros/tables/oracle/eff_sat_v0.sql | Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert. |
| macros/tables/fabric/eff_sat_v0.sql | Filters ghost ldts out of current_status_prep; unions ghost rows back into records_to_insert. |
| macros/tables/exasol/eff_sat_v0.sql | Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert. |
| macros/tables/databricks/eff_sat_v0.sql | Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert. |
| macros/tables/bigquery/eff_sat_v0.sql | Filters ghost ldts out of current_status; unions ghost rows back into records_to_insert. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tkiehn
left a comment
There was a problem hiding this comment.
Looks good as discussed with @tkirschke and @umattern
This comment was marked as outdated.
This comment was marked as outdated.
|
dbt test combined result: ❌ DetailsRESULTS for Synapse: RESULTS for Postgres: RESULTS for BigQuery: RESULTS for Redshift: RESULTS for Snowflake: RESULTS for Exasol: RESULTS for Fabric: RESULTS for Oracle: RESULTS for Databricks: RESULTS for SQL Server: RESULTS for Trino: Link to workflow summary: https://github.com/ScalefreeCOM/datavault4dbt-ci-cd/actions/runs/29250957277 |
Description
Should only insert two additional Ghost Records to an effectivity Sat.
Would be nice it someone could verify it from a technical and logical perspective. Especially all other adapters than Snowflake.
Type of change