Skip to content

Commit 3dcd8da

Browse files
committed
docs: parallel dbt jobs and metadata artifact upload guidance
Document contention when parallel jobs write Elementary metadata tables: disable_dbt_artifacts_autoupload on concurrent runs and a scheduled job using dbt run --select elementary with enable_elementary_models. Align on-run-end hooks and reduce-on-run-end-time examples with the same command; link to FAQ for the dbt_project.yml pattern. Made-with: Cursor
1 parent dca0c8c commit 3dcd8da

4 files changed

Lines changed: 42 additions & 11 deletions

File tree

docs/cloud/guides/troubleshoot.mdx

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,29 @@ Try following these steps to troubleshoot:
2121

2222
</Accordion>
2323

24+
<Accordion title="Parallel dbt jobs error when writing to Elementary metadata artifact tables">
25+
26+
When multiple dbt jobs run in parallel, they can contend on the same Elementary metadata artifact tables.
27+
28+
**Disable automatic metadata artifact uploads** in `dbt_project.yml`:
29+
30+
```yaml dbt_project.yml
31+
vars:
32+
disable_dbt_artifacts_autoupload: true
33+
```
34+
35+
**Add a separate scheduled job** (daily or hourly) to upload the artifacts:
36+
37+
```bash
38+
dbt run --select elementary --vars '{"enable_elementary_models": true}'
39+
```
40+
41+
For the `enable_elementary_models` pattern in `dbt_project.yml`, see [Can I disable / exclude the Elementary models?](/oss/general/faq#can-i-disable-exclude-the-elementary-models). For more context, see [on-run-end hooks](/data-tests/dbt/on-run-end_hooks), [Reduce on-run-end time](/data-tests/dbt/reduce-on-run-end-time), and [Dev / prod configuration](/cloud/guides/dev-prod-configuration).
42+
43+
Let us know if you run into anything — start a chat from the Elementary UI or reach us on [Slack](https://elementary-data.com/community).
44+
45+
</Accordion>
46+
2447
<Accordion title="Column information cannot be retrieved">
2548

2649
This error can happen because of a few reasons:

docs/data-tests/dbt/on-run-end_hooks.mdx

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ vars:
7070
disable_dbt_artifacts_autoupload: true
7171
```
7272
73-
2. Make sure to run `dbt run --select edr.dbt_artifacts` upon merging PRs.
73+
2. Make sure to run `dbt run --select elementary --vars '{"enable_elementary_models": true}'` upon merging PRs.
7474

7575
</Accordion>
7676

@@ -100,9 +100,7 @@ vars:
100100
```
101101

102102
<Warning>
103-
If you disable the artifacts autoupload, we recommend your run `dbt run
104-
--select elementary.edr.dbt_artifacts` every time you deploy changes to your
105-
project.
103+
If you disable the artifacts autoupload, we recommend you run `dbt run --select elementary --vars '{"enable_elementary_models": true}'` every time you deploy changes to your project.
106104
</Warning>
107105

108106
#### Disable result models updates

docs/oss/general/troubleshooting.mdx

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -210,19 +210,27 @@ If you want the Elementary UI to show data for a longer period of time, use the
210210
211211
212212
213-
<Accordion title="Error when trying to run parallel dbt jobs">
213+
<Accordion title="Parallel dbt jobs error when writing to Elementary metadata artifact tables">
214214
215-
When writing to the `dbt_artifacts` tables in the Elementary schema, data is deleted and reinserted. Running parallel jobs through an orchestrator can lead to errors, as multiple jobs may attempt to modify the same tables simultaneously.
216-
To prevent this, you should:
217-
1. [Disable the on-run-end hooks](/oss/general/faq#can-i-disable-the-on-run-end-hooks-or-results-uploading)
218-
2. [Exclude the Elementary models](/oss/general/faq#can-i-disable-exclude-the-elementary-models)
215+
When multiple dbt jobs run in parallel, they can contend on the same Elementary metadata artifact tables.
219216
220-
For scheduled updates to `dbt_artifacts` (e.g., a daily job), run:
217+
**Disable automatic metadata artifact uploads** in `dbt_project.yml`:
221218
219+
```yaml dbt_project.yml
220+
vars:
221+
disable_dbt_artifacts_autoupload: true
222222
```
223+
224+
**Add a separate scheduled job** (daily or hourly) to upload the artifacts:
225+
226+
```bash
223227
dbt run --select elementary --vars '{"enable_elementary_models": true}'
224228
```
225229

230+
For the `enable_elementary_models` pattern in `dbt_project.yml`, see [Can I disable / exclude the Elementary models?](/oss/general/faq#can-i-disable-exclude-the-elementary-models). For more context, see [on-run-end hooks](/data-tests/dbt/on-run-end_hooks) and [Reduce on-run-end time](/oss/guides/reduce-on-run-end-time).
231+
232+
Let us know if you run into anything — we’re responsive on [#community-support](https://elementary-community.slack.com/archives/C02CTC89LAX).
233+
226234
</Accordion>
227235

228236

docs/snippets/guides/reduce-on-run-end-time.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,11 @@ vars:
2828
When you need to update metadata (e.g., after schema changes or adding new models), upload artifacts either manually or via a recurring job (e.g., daily):
2929
3030
```bash
31-
dbt run --select elementary.edr.dbt_artifacts
31+
dbt run --select elementary --vars '{"enable_elementary_models": true}'
3232
```
3333
34+
For the `enable_elementary_models` pattern in `dbt_project.yml`, see [Can I disable / exclude the Elementary models?](/oss/general/faq#can-i-disable-exclude-the-elementary-models).
35+
3436
<Info>
3537
Metadata artifacts only need to be uploaded when your project structure changes (new models, tests, sources, etc.). For most runs, you only need run results, which are much faster to upload.
3638
</Info>

0 commit comments

Comments
 (0)