Skip to content

duplicates handling in case of multiple records for src_pk#266

Open
Kalyani5665 wants to merge 1 commit into
Datavault-UK:masterfrom
Kalyani5665:feat/ref_table_macro_fix
Open

duplicates handling in case of multiple records for src_pk#266
Kalyani5665 wants to merge 1 commit into
Datavault-UK:masterfrom
Kalyani5665:feat/ref_table_macro_fix

Conversation

@Kalyani5665
Copy link
Copy Markdown

This PR resolves an issue where multiple records for the same src_pk were being loaded into the reference table. When the source contained more than one entry for a given src_pk, all records were previously passed through, causing duplicates in the final table.

What This Fix Does:

  • Deduplicates records at source ingestion level.
  • Selects a single record per src_pk based on src_ldts ordering.
  • Prevents duplicate loads into the reference table and ensures downstream consistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant