Skip to content

Commit 29c3532

Browse files
authored
fix: correctly document invalidate_hard_delete behavior (#3655)
1 parent 7fd6b7d commit 29c3532

1 file changed

Lines changed: 16 additions & 16 deletions

File tree

docs/concepts/models/model_kinds.md

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -626,12 +626,7 @@ TABLE db.menu_items (
626626

627627
A hard delete is when a record no longer exists in the source table. When this happens,
628628

629-
If `invalidate_hard_deletes` is set to `true` (default):
630-
631-
* `valid_to` column will be set to the time when the SQLMesh run started that detected the missing record (called `execution_time`).
632-
* If the record is added back, then the `valid_to` column will remain unchanged.
633-
634-
If `invalidate_hard_deletes` is set to `false`:
629+
If `invalidate_hard_deletes` is set to `false` (default):
635630

636631
* `valid_to` column will continue to be set to `NULL` (therefore still considered "valid")
637632
* If the record is added back, then the `valid_to` column will be set to the `valid_from` of the new record.
@@ -641,13 +636,18 @@ When a record is added back, the new record will be inserted into the table with
641636
* SCD_TYPE_2_BY_TIME: the largest of either the `updated_at` timestamp of the new record or the `valid_from` timestamp of the deleted record in the SCD Type 2 table
642637
* SCD_TYPE_2_BY_COLUMN: the `execution_time` when the record was detected again
643638

644-
One way to think about `invalidate_hard_deletes` is that, if enabled, deletes are most accurately tracked in the SCD Type 2 table since it records when the delete occurred.
639+
If `invalidate_hard_deletes` is set to `true`:
640+
641+
* `valid_to` column will be set to the time when the SQLMesh run started that detected the missing record (called `execution_time`).
642+
* If the record is added back, then the `valid_to` column will remain unchanged.
643+
644+
One way to think about `invalidate_hard_deletes` is that, if `invalidate_hard_deletes` is set to `true`, deletes are most accurately tracked in the SCD Type 2 table since it records when the delete occurred.
645645
As a result though, you can have gaps between records if the there is a gap of time between when it was deleted and added back.
646-
If you would prefer to not have gaps, and a result consider missing records in source as still "valid", then you can set `invalidate_hard_deletes` to `false`.
646+
If you would prefer to not have gaps, and a result consider missing records in source as still "valid", then you can leave the default value or set `invalidate_hard_deletes` to `false`.
647647

648648
### Example of SCD Type 2 By Time in Action
649649

650-
Lets say that you started with the following data in your source table:
650+
Lets say that you started with the following data in your source table and `invalidate_hard_deletes` is set to `true`:
651651

652652
| ID | Name | Price | Updated At |
653653
|----|------------------|:-----:|:-------------------:|
@@ -723,7 +723,7 @@ Since in this case the updated at timestamp did not change it is likely the item
723723

724724
### Example of SCD Type 2 By Column in Action
725725

726-
Lets say that you started with the following data in your source table:
726+
Lets say that you started with the following data in your source table and `invalidate_hard_deletes` is set to `true`:
727727

728728
| ID | Name | Price |
729729
|----|------------------|:-----:|
@@ -798,12 +798,12 @@ This is the most accurate representation of the menu based on the source data pr
798798

799799
### Shared Configuration Options
800800

801-
| Name | Description | Type |
802-
|-------------------------|----------------------------------------------------------------------------------------------------------------|---------------------------|
803-
| unique_key | Unique key used for identifying rows between source and target | List of strings or string |
804-
| valid_from_name | The name of the `valid_from` column to create in the target table. Default: `valid_from` | string |
805-
| valid_to_name | The name of the `valid_to` column to create in the target table. Default: `valid_to` | string |
806-
| invalidate_hard_deletes | If set to `true`, when a record is missing from the source table it will be marked as invalid. Default: `true` | bool |
801+
| Name | Description | Type |
802+
|-------------------------|-----------------------------------------------------------------------------------------------------------------|---------------------------|
803+
| unique_key | Unique key used for identifying rows between source and target | List of strings or string |
804+
| valid_from_name | The name of the `valid_from` column to create in the target table. Default: `valid_from` | string |
805+
| valid_to_name | The name of the `valid_to` column to create in the target table. Default: `valid_to` | string |
806+
| invalidate_hard_deletes | If set to `true`, when a record is missing from the source table it will be marked as invalid. Default: `false` | bool |
807807

808808
!!! tip "Important"
809809

0 commit comments

Comments
 (0)