Skip to content

Commit 57d4502

Browse files
TheEdgeOfRageTheEdgeOfRage
authored
Update dbt-to-datashare docs for S3 datashare (#73)
1 parent b5ccadc commit 57d4502

2 files changed

Lines changed: 75 additions & 20 deletions

File tree

‎docs/dune-datashares.md‎

Lines changed: 33 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Dune Datashares
22

3-
Datashares sync Dune tables to external data warehouses such as Snowflake and BigQuery so downstream consumers can query the data outside Dune.
3+
Datashares sync your Dune tables to external destinations such as Snowflake, BigQuery, and S3, so you can query the data outside Dune.
44

55
## Prerequisites
66

@@ -77,14 +77,14 @@ The post-hook macro evaluates `is_incremental()` at execution time and picks the
7777

7878
All datashare config lives under `meta.datashare` in the model `config()` block.
7979

80-
| Property | Required | Type | Description |
81-
| --- | --- | --- | --- |
82-
| `enabled` | Yes | `boolean` | Must be `true` to trigger sync. |
83-
| `time_column` | Yes | `string` | Column used to define the sync window. |
84-
| `time_start` | Yes | `string` | SQL expression for the start of the full-refresh sync window. |
85-
| `time_start_incremental` | No | `string` | SQL expression for incremental runs. Falls back to `time_start` if omitted. |
86-
| `time_end` | No | `string` | SQL expression for the end of the sync window. Defaults to `now()`. |
87-
| `unique_key_columns` | No | `list[string]` | Row identity columns. Falls back to the model `unique_key` if omitted. |
80+
| Property | Required | Type | Description |
81+
| ------------------------ | -------- | -------------- | --------------------------------------------------------------------------- |
82+
| `enabled` | Yes | `boolean` | Must be `true` to trigger sync. |
83+
| `time_column` | Yes | `string` | Column used to define the sync window. |
84+
| `time_start` | Yes | `string` | SQL expression for the start of the full-refresh sync window. |
85+
| `time_start_incremental` | No | `string` | SQL expression for incremental runs. Falls back to `time_start` if omitted. |
86+
| `time_end` | No | `string` | SQL expression for the end of the sync window. Defaults to `now()`. |
87+
| `unique_key_columns` | No | `list[string]` | Row identity columns. Falls back to the model `unique_key` if omitted. |
8888

8989
All time expressions are SQL, not literal timestamps. The macro wraps them in `CAST(... AS VARCHAR)` before calling the table procedure.
9090

@@ -102,11 +102,11 @@ remote_read_multiplier = MERGE read window / run cadence
102102

103103
Examples:
104104

105-
| Cadence | `time_column` | Incremental window | Multiplier | Notes |
106-
| ------- | ------------- | -------------------------- | ---------- | ----- |
107-
| Daily | `date` | `interval '1' day` | 1x | Safe default. The included example model uses this shape. |
108-
| Hourly | `timestamp` | `interval '2' hour` | 2x | Use only when `time_column` is timestamp-granular and the destination is partitioned/prunable on it. |
109-
| Hourly | `date` | `interval '1' day` | 24x | **Cost trap.** Every hourly run re-reads the full day's partition from the destination. |
105+
| Cadence | `time_column` | Incremental window | Multiplier | Notes |
106+
| ------- | ------------- | ------------------- | ---------- | ---------------------------------------------------------------------------------------------------- |
107+
| Daily | `date` | `interval '1' day` | 1x | Safe default. The included example model uses this shape. |
108+
| Hourly | `timestamp` | `interval '2' hour` | 2x | Use only when `time_column` is timestamp-granular and the destination is partitioned/prunable on it. |
109+
| Hourly | `date` | `interval '1' day` | 24x | **Cost trap.** Every hourly run re-reads the full day's partition from the destination. |
110110

111111
Rules of thumb:
112112

@@ -143,12 +143,12 @@ Rules of thumb:
143143

144144
The macro determines `full_refresh` automatically:
145145

146-
| Context | `full_refresh` |
147-
| --- | --- |
148-
| Incremental post-hook on a normal incremental run | `false` |
149-
| Incremental post-hook on first run or `--full-refresh` | `true` |
150-
| Table materialization post-hook | `true` |
151-
| `run-operation` | `false` unless overridden |
146+
| Context | `full_refresh` |
147+
| ------------------------------------------------------ | ------------------------- |
148+
| Incremental post-hook on a normal incremental run | `false` |
149+
| Incremental post-hook on first run or `--full-refresh` | `true` |
150+
| Table materialization post-hook | `true` |
151+
| `run-operation` | `false` unless overridden |
152152

153153
## Generated SQL
154154

@@ -225,6 +225,19 @@ Remove a table from datashare with:
225225
ALTER TABLE dune.<schema>.<table> EXECUTE delete_datashare
226226
```
227227

228+
This stops the sync and revokes access to the destination.
229+
230+
## S3 Export
231+
232+
S3 Export delivers your data as an Iceberg table in a Dune-managed S3 bucket. Dune adds a bucket policy to the export bucket which grants an AWS principal you control read access. You can either:
233+
234+
- create an IAM role in your own AWS account with read access to S3 and give Dune that role's ARN, so only that role can read the bucket; or
235+
- give Dune just your AWS account ID, and Dune grants the whole account access. You can then control which IAM users/roles have access to the S3 bucket by setting the appropriate IAM policy permissions.
236+
237+
You query the data directly from S3 with your own engine (e.g. Athena, Spark, DuckDB, etc.) using that principal, without going through Dune. S3 Export currently supports the **Iceberg** table format. To set up an S3 target, contact Dune with the bucket region you want, and either the IAM role ARN or the AWS account ID to grant read access.
238+
239+
Note that the bucket will be configured with [requester pays](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html), so to read the data you have to set a header on the S3 requests to accept that you will be charged for read requests. Most AWS SDKs have a way to just configure this directly without having to manually set request headers.
240+
228241
## Example Workflow
229242

230243
1. Configure a model with `meta.datashare`.

‎macros/dune_dbt_overrides/datashare_table_sync_post_hook.sql‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,39 @@
1919
{{ return('NULL' if value is none else 'CAST(' ~ value ~ ' AS VARCHAR)') }}
2020
{%- endmacro -%}
2121

22+
{#
23+
Returns true if an active (non-deleted) datashare sync exists for this table.
24+
Scoped to a target only when both target_type and target_region are given,
25+
since a partial key would match a different target of the same table.
26+
Returns true on probe failure so the caller keeps its is_incremental() behavior.
27+
#}
28+
{% macro _datashare_active_sync_exists(schema_name, table_name, target_type=None, target_region=None) %}
29+
{%- if not execute -%}
30+
{{ return(true) }}
31+
{%- endif -%}
32+
{%- set has_type = target_type is not none and target_type | string | trim != '' -%}
33+
{%- set has_region = target_region is not none and target_region | string | trim != '' -%}
34+
{%- set where = [] -%}
35+
{%- do where.append('source_schema = ' ~ _datashare_sql_string(schema_name)) -%}
36+
{%- do where.append('source_table = ' ~ _datashare_sql_string(table_name)) -%}
37+
{%- do where.append('deleted_at IS NULL') -%}
38+
{%- if has_type and has_region -%}
39+
{%- do where.append('target_type = ' ~ _datashare_sql_string(target_type)) -%}
40+
{%- do where.append('target_region = ' ~ _datashare_sql_string(target_region)) -%}
41+
{%- endif -%}
42+
{%- set probe_sql = 'SELECT count(*) AS c FROM dune.datashare.table_syncs WHERE ' ~ (where | join(' AND ')) -%}
43+
{%- set result = none -%}
44+
{%- set probe = run_query(probe_sql) -%}
45+
{%- if probe is not none and probe.columns | length > 0 and probe.columns[0].values() | length > 0 -%}
46+
{%- set result = probe.columns[0].values()[0] -%}
47+
{%- endif -%}
48+
{%- if result is none -%}
49+
{{ log('datashare sync probe for ' ~ schema_name ~ '.' ~ table_name ~ ' returned no rows; assuming sync exists.', info=True) }}
50+
{{ return(true) }}
51+
{%- endif -%}
52+
{{ return(result | int > 0) }}
53+
{%- endmacro -%}
54+
2255
{#
2356
Datashare sync macro - generates ALTER TABLE ... EXECUTE datashare() SQL.
2457
Config reference and usage: docs/dune-datashares.md
@@ -56,6 +89,15 @@
5689
{%- set include_target_type = target_type is not none and target_type | string | trim != '' -%}
5790
{%- set include_target_region = target_region is not none and target_region | string | trim != '' -%}
5891

92+
{#- An incremental sync targets an existing destination via MERGE. If the
93+
destination sync was revoked while the source table still exists, dbt
94+
builds incrementally but there is nothing to merge into. Force a full
95+
refresh when no active sync is registered for this table/target. -#}
96+
{%- if not full_refresh and not _datashare_active_sync_exists(schema_name, table_name, target_type, target_region) -%}
97+
{{ log('No active datashare sync for ' ~ model_ref ~ '; forcing full_refresh.', info=True) }}
98+
{%- set full_refresh = true -%}
99+
{%- endif -%}
100+
59101
{%- set sql -%}
60102
ALTER TABLE {{ catalog_name }}.{{ schema_name }}.{{ table_name }} EXECUTE datashare(
61103
time_column => {{ _datashare_sql_string(time_column | default('', true)) }},

0 commit comments

Comments
 (0)