Skip to content

Commit 94a07ab

Browse files
Tobias-Pahtkiehntkirschke
authored
26 create additional metadata columns for control snap tables v0 and v1 (#419)
* REDSHIFT.control_snap_v0: added is_eom, is_quarterly, is_eoq, is_eoy * BIGQUERY.control_snap_v0: added is_eoq * SNOWFLAKE.control_snap_v0: added is_eom, is_quarterly, is_eoq, is_eoy * DATABRICKS.control_snap_v0: added is_eoq * POSTGRES.control_snap_v0: added is_eom, is_quarterly, is_eoq, is_eoy * ORACLE.control_snap_v0: added is_eom, is_quarterly, is_eoq, is_eoy * feat: reworked control_snap_v0 functionality * feat: updated the v1 dependencies * feat: added documentation for weekly calculations * feat: removed explicit date columns in v0 and added all boolean flags to v1 snapshot tables * fix: syntax corrections * fix: added macro file for correct call and updated adapter calls * fix: corrected snowflake syntax * feat: added adapter specific macros to first_day_of_week macro * fix: fixed macro call * fix: added else statement to macro block for missing var * feat: updated snapshot logic for user handling * fix: fixed bigquery control snap adapter * fix: fixed some snapshot logic that was wrong in testing * fix: reworked snapshot logic once more and fixed the adapter tables * feat: updated docs --------- Co-authored-by: umattern Co-authored-by: Theo Kiehn <162969167+tkiehn@users.noreply.github.com> Co-authored-by: Tim Kirschke <81677440+tkirschke@users.noreply.github.com>
1 parent 5cfd8b0 commit 94a07ab

23 files changed

Lines changed: 854 additions & 533 deletions

‎dbt_project.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ clean-targets: # directories to be removed by `dbt clean`
1414
- "dbt_packages"
1515

1616

17-
17+
1818
vars:
1919
#Column Aliases
2020
datavault4dbt.ldts_alias: 'ldts'
@@ -39,7 +39,10 @@ vars:
3939
datavault4dbt.include_business_objects_before_appearance: false
4040
datavault4dbt.enable_static_analysis_overwrite: true
4141
datavault4dbt.multi_source_models__execution_aware_loading: true
42-
42+
43+
# Default start of the week is Monday
44+
datavault4dbt.first_day_of_week: {"snowflake": "1", "exasol": "1", "postgres": "1", "databricks": "1", "bigquery": "2", "synapse": "2", "fabric": "2", "oracle": "2", "redshift": "1", "sqlserver": "2"}
45+
4346
#Ghost Record Configuration
4447
datavault4dbt.beginning_of_all_times: {"bigquery":"0001-01-01T00-00-01","snowflake":"0001-01-01T00:00:01", "exasol": "0001-01-01 00:00:01", "postgres": "0001-01-01 00:00:01", "redshift": "0001-01-01 00:00:01", "synapse": "1901-01-01T00:00:01", "fabric": "0001-01-01T00:00:01", "oracle":"0001-01-01 00:00:01", databricks: "0001-01-01 00:00:01"}
4548
datavault4dbt.end_of_all_times: {"bigquery":"8888-12-31T23-59-59","snowflake":"8888-12-31T23:59:59", "exasol": "8888-12-31 23:59:59", "postgres": "8888-12-31 23:59:59", "redshift": "8888-12-31 23:59:59", "synapse": "8888-12-31T23:59:59", "fabric": "8888-12-31T23:59:59", "oracle":"8888-12-31 23:59:59", databricks: "8888-12-31 23:59:59"}

‎docs/01_Macro_Instructions/22_Business_Vault/24_Snapshot_Control/24_snapshot_control_v0.md‎

Lines changed: 50 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -8,29 +8,58 @@ title: Snapshot Control v0
88

99
---
1010

11-
This macro creates a snapshot table to control snapshot-based tables like PITs and Bridges. The snapshot table will hold daily snapshots starting at a specific start_date and has a configurable daytime. Usual application would involve creating one snapshot table per Data Vault environment, and therefore creating one dbt model using this macro. The model needs to be scheduled daily, with a execution time that matches your desired snapshot time, since the macro automatically inserts all snapshots until the current day, no matter which time it is, and a snapshot for 08:00:00 should be calculated at that time. So if the snapshot table is configured to have a ‘daily_snapshot_time’ of ’07:00:00′, all snapshots in the table will have the timestamp ’07:00:00′. Therefore you need to schedule the building of the snapshot table also to ’07:00:00′.
11+
This macro creates a snapshot table to control snapshot-based tables like PITs and Bridges. The snapshot table will hold daily snapshots starting at a specific start_date and has a configurable daytime. Usual application would involve creating one snapshot table per Data Vault environment, and therefore creating one dbt model using this macro. The model needs to be scheduled daily, with an execution time that matches your desired snapshot time, since the macro automatically inserts all snapshots until the current day, no matter which time it is, and a snapshot for 08:00:00 should be calculated at that time. So if the snapshot table is configured to have a `daily_snapshot_time` of `07:00:00`, all snapshots in the table will have the timestamp `07:00:00`. Therefore you need to schedule the building of the snapshot table also to `07:00:00`.
1212

1313
In addition to the actual snapshot-datetimestamp (sdts), the macro generates the following columns:
1414

15-
| Column | Data Type | Explanation |
16-
|------------------|-----------|-------------|
17-
| replacement_sdts | timestamp | Allows users to replace a sdts with another one, without having to update the actual sdts column. By default this column is filled with the regular sdts. |
18-
| caption | string | Allows users to title their snapshots. Examples would be something like: 'Christmas 2022', or 'End-of-year report 2021'. By default this is filled with 'Snapshot \{sdts\}', holding the respective sdts. |
19-
| is_hourly | boolean | Captures if the time of a sdts is on an exact hour, meaning minutes=0 and seconds=0. All sdts created by this macro are daily and therefore always hourly, but this column enables future inserts of custom, user-defined sdts. |
20-
| is_daily | boolean | Captures if the time of a sdts is on exactly midnight, meaning hours=0, minutes=0 and seconds=0. This depends on your desired daily_snapshot_time, but is not used by the downstream macros, and just generates additional metadata for potential future use. |
21-
| is_weekly | boolean | Captures if the day of the week of a sdts is Monday. |
22-
| is_monthly | boolean | Captures if a sdts is the first day of a month. |
23-
| is_end_of_month | boolean | Captures if a sdts is the last day of a month. |
24-
| is_quarterly | boolean | Captures if a sdts is the first day of a quarter. |
25-
| is_yearly | boolean | Captures if a sdts is the first day of a year. |
26-
| is_end_of_year | boolean | Captures if a sdts is the last day of a year. |
27-
| comment | string | Allows users to write custom comments for each sdts. By default this column is set to NULL. |
28-
29-
| Parameters | Data Type | Required | Default Value | Explanation |
30-
|---------------------|-----------|-----------|----------------------------|-------------|
31-
| start_date | date | mandatory | – | Defines the earliest date that should be available inside the snapshot_table. The format of this date must be 'YYYY-MM-DD'. |
32-
| daily_snapshot_time | time | mandatory | – | Defines the time that your daily snapshots should have. Usually this is either something right before daily business starts, or after daily business is over. Needs to be in the format 'hh:mm:ss'. |
33-
| sdts_alias | string | optional | datavault4dbt.sdts_alias | Defines the name of the snapshot date timestamp column inside the snapshot_table. |
15+
| Column | Data Type | Explanation |
16+
|---|---|---|
17+
| replacement_sdts | timestamp | Allows users to replace an sdts with another one, without having to update the actual sdts column. By default this column is filled with the regular sdts. |
18+
| caption | string | Allows users to title their snapshots. Examples would be something like: 'Christmas 2022', or 'End-of-year report 2021'. By default this is filled with 'Snapshot \{sdts\}', holding the respective sdts. |
19+
| is_hourly | boolean | Captures if the time of an sdts is on an exact hour, meaning minutes=0 and seconds=0. All sdts created by this macro are daily and therefore always hourly, but this column enables future inserts of custom, user-defined sdts. |
20+
| is_daily | boolean | Captures if the time of an sdts is on exactly midnight, meaning hours=0, minutes=0 and seconds=0. This depends on your desired daily_snapshot_time, but is not used by the downstream macros, and just generates additional metadata for potential future use. |
21+
| is_beginning_of_week | boolean | Captures if a sdts is the first day of the week. This is fully configurable via the `datavault4dbt.first_day_of_week` variable. |
22+
| is_end_of_week | boolean | Captures if a sdts is the last day of the week, automatically calculated based on your configured first day of the week. |
23+
| is_beginning_of_month | boolean | Captures if a sdts is the first day of a month. |
24+
| is_end_of_month | boolean | Captures if a sdts is the last day of a month. |
25+
| is_beginning_of_quarter | boolean | Captures if a sdts is the first day of a quarter. |
26+
| is_end_of_quarter | boolean | Captures if a sdts is the last day of a quarter. |
27+
| is_beginning_of_year | boolean | Captures if a sdts is the first day of a year. |
28+
| is_end_of_year | boolean| Captures if a sdts is the last day of a year. |
29+
| comment | string | Allows users to write custom comments for each sdts. By default this column is set to NULL. |
30+
31+
32+
### Macro Parameters
33+
34+
| Parameters | Data Type | Required | Default Value | Explanation |
35+
|---|---|---|---|---|
36+
| start_date | date | mandatory | – | Defines the earliest date that should be available inside the snapshot_table. The format of this date must be 'YYYY-MM-DD'. |
37+
| daily_snapshot_time | time | mandatory | – | Defines the time that your daily snapshots should have. Usually this is either something right before daily business starts, or after daily business is over. Needs to be in the format 'hh:mm:ss'. |
38+
| sdts_alias | string | optional | datavault4dbt.sdts_alias | Defines the name of the snapshot date timestamp column inside the snapshot_table. |
39+
40+
41+
---
42+
43+
## DAY OF WEEK ADAPTER REFERENCE MATRIX
44+
45+
Because different database dialects use different standards and functions for extracting the day of the week, configuring your `datavault4dbt.first_day_of_week` variable requires knowing how your specific target database behaves natively.
46+
47+
Use the table below to determine the correct integer to input for your target database in your `dbt_project.yml` based on whether you want your weeks to start on Sunday or Monday.
48+
49+
| Dialect | Native Function | Sunday | Monday | Behavior |
50+
|---|---|:---:|:---:|---|
51+
| **BigQuery** | `EXTRACT(DAYOFWEEK)` | 1 | 2 | US Standard |
52+
| **Synapse** | `DATEPART(WEEKDAY)` | 1 | 2 | US Standard |
53+
| **Fabric** | `DATEPART(WEEKDAY)` | 1 | 2 | US Standard |
54+
| **Oracle** | `TRUNC(..., 'IW') Math` | 1 | 2 | US Standard |
55+
| **SQLServer** | `DATEPART(WEEKDAY)` | 1 | 2 | US Standard |
56+
| **Postgres** | `EXTRACT(ISODOW)` | 7 | 1 | ISO Standard |
57+
| **Snowflake** | `EXTRACT(DAYOFWEEK_ISO)` | 7 | 1 | ISO Standard |
58+
| **Databricks** | `dayofweek_iso()` | 7 | 1 | ISO Standard |
59+
| **Exasol** | `TO_CHAR('ID')` | 7 | 1 | ISO Standard |
60+
| **Redshift** | `EXTRACT(DOW)` | 0 | 1 | Wild Card (0-6) |
61+
62+
---
3463

3564
## EXAMPLE 1
3665

@@ -42,9 +71,7 @@ start_date: '2015-01-01'
4271
daily_snapshot_time: '07:30:00'
4372
{%- endset -%}
4473
45-
{{ datavault4dbt.control_snap_v0(yaml_metadata=yaml_metadata) }}
46-
```
47-
74+
{{ datavault4dbt.control_snap_v0(yaml_metadata=yaml_metadata) }}
4875
### DESCRIPTION
4976
5077
- **start_date**: The parameter start_date has to be in the format ‘YYYY-MM-DD’. With the start_date set to ‘2015-01-01’, the snapshot table would hold daily snapshots beginning at 2015.

‎docs/01_Macro_Instructions/22_Business_Vault/24_Snapshot_Control/25_snapshot_control_v1.md‎

Lines changed: 38 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -8,30 +8,40 @@ title: Snapshot Control v1
88

99
---
1010

11-
This macro creates a view that extends an existing control_snap_v0 table by dynamically changing information. These information include a logic to implement logarithmic snapshots. That means that the further I look into the past, the more coarsely the snapshots should be granulated.
11+
This macro creates a view that extends an existing `control_snap_v0` table by dynamically adding information. This information includes a logic to implement logarithmic snapshots. That means that the further I look into the past, the more coarsely the snapshots should be granulated.
1212

1313
For example, I want to keep daily snapshots for the past 30 days, but I am not interested in daily snapshots for the past 10 years, and therefore I only keep weekly snapshots for the past 6 months, monthly snapshots for the past 3 years, and yearly snapshots forever. This procedure strongly reduces the number of active snapshots, and therefore also the number of rows, and the required computation inside all PITs and Bridges. This logic is optional and would be captured in a boolean column called `is_active`.
1414

15-
Whenever a logarithmic snapshot logic is used and picked up by PIT tables, a logic is required that deletes records out of PIT tables, that are no longer active. For this, a post_hook called “clean_up_pit” is provided in this package, that should be applied for each PIT table.
15+
Whenever a logarithmic snapshot logic is used and picked up by PIT tables, a logic is required that deletes records out of PIT tables that are no longer active. For this, a post_hook called `clean_up_pit` is provided in this package, that should be applied for each PIT table.
1616

17-
In addition to that, a few other dynamic columns are generated:
17+
In addition to inheriting the core columns (`sdts`, `replacement_sdts`, `caption`, `is_hourly`, `is_daily`, `comment`) from the `control_snap_v0` view, the following dynamic and pass-through columns are generated:
1818

19-
| Column | Data Type | Explanation |
20-
|----------------------|-----------|-------------|
21-
| is_latest | boolean | Captures if a sdts is the latest one inside the snapshot table. There is always only one snapshot inside the view that has TRUE here. |
22-
| is_current_year | boolean | Captures if a sdts is part of the current calendar year. |
23-
| is_last_year | boolean | Captures if a sdts is part of the last calendar year. |
24-
| is_rolling_year | boolean | Captures if a sdts is inside the past year, starting from the current date. |
25-
| is_last_rolling_year | boolean | Captures if a sdts is inside the range that starts two years ago (from the current date) and ranges until one year ago (from the current date). |
19+
| Column | Data Type | Explanation |
20+
|---|---|---|
21+
| snapshot_trigger_column | boolean | Defaults to `is_active`. Captures if a snapshot is considered "active" based on the logarithmic logic provided. If no logic is provided, this defaults to TRUE for all snapshots. |
22+
| is_latest | boolean | Captures if a sdts is the latest one inside the snapshot table. There is always exactly one snapshot inside the view that evaluates to TRUE here. |
23+
| is_beginning_of_week | boolean | Inherited from v0. Captures if a sdts is the first day of the week based on your global config. |
24+
| is_end_of_week | boolean | Inherited from v0. Captures if a sdts is the last day of the week. |
25+
| is_beginning_of_month | boolean | Inherited from v0. Captures if a sdts is the first day of a month. |
26+
| is_end_of_month | boolean | Inherited from v0. Captures if a sdts is the last day of a month. |
27+
| is_beginning_of_quarter | boolean | Inherited from v0. Captures if a sdts is the first day of a quarter. |
28+
| is_end_of_quarter | boolean | Inherited from v0. Captures if a sdts is the last day of a quarter. |
29+
| is_beginning_of_year | boolean | Inherited from v0. Captures if a sdts is the first day of a year. |
30+
| is_end_of_year | boolean | Inherited from v0. Captures if a sdts is the last day of a year. |
31+
| is_current_year | boolean | Captures if a sdts is part of the current calendar year. |
32+
| is_last_year | boolean | Captures if a sdts is part of the last calendar year. |
33+
| is_rolling_year | boolean | Captures if a sdts is inside the past year, starting from the current date. |
34+
| is_last_rolling_year | boolean | Captures if a sdts is inside the range that starts two years ago (from the current date) and ranges until one year ago (from the current date). |
2635

27-
| Parameters | Data Type | Required | Default Value | Explanation |
28-
|----------------|-----------|-----------|---------------|-------------|
29-
| control_snap_v0 | string | mandatory | – | The name of the underlying version 0 control snapshot table. Needs to be available as a dbt model. |
36+
### Macro Parameters
3037

31-
| Parameters | Data Type | Required | Default Value | Explanation |
32-
|------------|-----------|----------|--------------------------|-------------|
33-
| log_logic | dictionary | optional | None | Defining the desired durations of each granularity. Available granularities are 'daily', 'weekly', 'monthly', 'end_of_month', 'quarterly', 'yearly' and 'end_of_year'. For each granularity the duration can be defined as an integer and the time unit for that duration. The units include (in BigQuery): DAY, WEEK, MONTH, QUARTER, YEAR. Besides defining a duration and a unit for each granularity, there is also the option to set a granularity to 'forever'. E.g. reporting requires daily snapshots for 3 months, and after that the monthly snapshots should be kept forever. If log_logic is not set, no logic will be applied, and all snapshots will stay active. The other dynamic columns are calculated anyway. The duration is always counted from the current date. |
34-
| sdts_alias | string | optional | datavault4dbt.sdts_alias | Defines the name of the snapshot date timestamp column inside the snapshot_table. |
38+
| Parameters | Data Type | Required | Default Value | Explanation |
39+
|---|---|---|---|---|
40+
| control_snap_v0 | string | mandatory | – | The name of the underlying version 0 control snapshot table. Needs to be available as a dbt model. |
41+
| log_logic | dictionary | optional | None | Defines the desired durations of each granularity. Available granularities check against the V0 flags and include: `daily`, `weekly`, `monthly`, `yearly`. For each granularity, the duration can be defined as an integer and the time unit for that duration. The units include: DAY, MONTH, YEAR, etc. Besides defining a duration and a unit, there is also the option to set a granularity to `forever: TRUE`. E.g. reporting requires daily snapshots for 3 months, and after that the monthly snapshots should be kept forever. If `log_logic` is not set, no logic will be applied, and all snapshots will stay active. The duration is always counted from the current date. |
42+
| sdts_alias | string | optional | datavault4dbt.sdts_alias | Defines the name of the snapshot date timestamp column inside the snapshot_table. |
43+
44+
---
3545

3646
## EXAMPLE 1
3747

@@ -41,17 +51,17 @@ In addition to that, a few other dynamic columns are generated:
4151
{%- set yaml_metadata -%}
4252
control_snap_v0: 'control_snap_v0'
4353
log_logic:
44-
daily:
45-
duration: 3
46-
unit: 'MONTH'
47-
weekly:
48-
duration: 1
49-
unit: 'YEAR'
50-
monthly:
51-
duration: 5
52-
unit: 'YEAR'
53-
yearly:
54-
forever: true
54+
daily:
55+
duration: 3
56+
unit: 'MONTH'
57+
weekly:
58+
duration: 1
59+
unit: 'YEAR'
60+
monthly:
61+
duration: 5
62+
unit: 'YEAR'
63+
yearly:
64+
forever: true
5565
{%- endset -%}
5666
5767
{{ datavault4dbt.control_snap_v1(yaml_metadata=yaml_metadata) }}

‎docs/26_General_Usage_Notes/29_Global_Variables/29_Global_Variables.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ All the following variables are **prefixed with `datavault4dbt`**.
2929
| include_business_objects_before_appearance | Ref Table | If a Ref_Hub entry should appear in the ref_table (snapshot based), even if the snapshot date is before the first appearance of that business object. |
3030
| enable_static_analysis_overwrite | All macros | Relevant for Fusion compatibility. For more info, see here. |
3131
| multi_source_models_execution_aware_loading | Multi source entities | Whether multi source entities should respect the dbt command to reduce runtimes. |
32+
| first_day_of_week | Snapshot Table | A mapping dictionary that defines the integer representation of the first day of the week (Sunday vs. Monday) for each specific database adapter. For the full configuration matrix, see [Snapshot Control v0](../24_Snapshot_Control/24_snapshot_control_v0.md). |
3233

3334
### COLUMN ALIASES
3435

‎macros/supporting/first_day_of_week.sql‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{%- macro first_day_of_week() %}
2+
{{ return(adapter.dispatch('first_day_of_week', 'datavault4dbt')()) }}
3+
{%- endmacro -%}
4+
5+
{%- macro default__first_day_of_week() %}
6+
7+
{%- set global_var = var('datavault4dbt.first_day_of_week', none) -%}
8+
9+
10+
{%- if global_var is mapping and target.type in global_var.keys()|map('lower') -%}
11+
12+
{%- set first_day = global_var[target.type] -%}
13+
14+
{%- else -%}
15+
{%- if execute -%}
16+
{%- do exceptions.warn("Warning: Adapter '"~ target.type ~"' not found in 'first_day_of_week' variable. Defaulting to 1.") -%}
17+
{%- endif -%}
18+
{%- set first_day = 1 -%}
19+
{%- endif -%}
20+
21+
{{ return(first_day | int) }}
22+
23+
{%- endmacro -%}

0 commit comments

Comments
 (0)