Skip to content

Commit 1a19ae0

Browse files
mintlify[bot]mintlify[bot]
authored
docs: add overview of the three ways to sync dbt with Lightdash (#701)
* docs: add overview of the three ways to sync dbt with Lightdash * docs: expand dbt sync section to five mechanisms with decision matrix * docs: make dbt sync decision matrix headers explicit with commands * docs: generalize GitHub Action references to CI in dbt sync section --------- Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
1 parent c99d7ac commit 1a19ae0

1 file changed

Lines changed: 172 additions & 9 deletions

File tree

‎references/integrations/dbt-projects.mdx‎

Lines changed: 172 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,36 +8,199 @@ description: "You can easily make changes in dbt and see them updated in your Li
88
Lightdash supports dbt v1.4.0 and above. If you are using an older version of dbt, you will need to upgrade to sync your project to Lightdash.
99
</Info>
1010

11-
## Syncing your dbt project to Lightdash
11+
## Syncing your dbt project to Lightdash
12+
13+
There are five ways to sync changes from dbt into Lightdash. They differ in **where the code comes from**, **where compilation happens**, **which `profiles.yml` / target governs the result**, and **whether they respect the project's UI Connection Settings**.
14+
15+
For ready-to-use workflow templates, see the [`lightdash/cli-actions`](https://github.com/lightdash/cli-actions) repo.
16+
17+
### Sync mechanisms
18+
19+
<AccordionGroup>
20+
<Accordion title="1. lightdash deploy from a developer's laptop">
21+
Compiles the dbt project locally and pushes the resulting `manifest.json` to Lightdash.
22+
23+
| Property | Value |
24+
| ------------------------------------- | --------------------------------------------------------------------------- |
25+
| dbt code source | Local working copy (uncommitted changes included) |
26+
| Compile location | Laptop |
27+
| `profiles.yml` source | Local (`~/.dbt/profiles.yml` by default, or `--profiles-dir`) |
28+
| Target | Default in that `profiles.yml`, overridden by `--target` |
29+
| UI Connection Settings respected? | No |
30+
| Auth | `lightdash login` session or `LIGHTDASH_API_KEY` env var |
31+
| Prerequisites | Lightdash CLI installed; dbt installed; warehouse creds reachable locally |
32+
| Remote git connection required? | No |
33+
34+
**Use cases:** initial project setup, debugging, one-off manual deploys.
35+
36+
**Trade-offs:** bypasses PR review; relies on per-developer `profiles.yml`; high footgun risk if a developer's prod target accidentally points at a non-prod warehouse.
37+
</Accordion>
38+
39+
<Accordion title="2. lightdash deploy from CI">
40+
Same as #1, but runs inside a CI workflow (GitHub Actions, GitLab CI, Azure Pipelines, Bitbucket Pipelines, etc.) with credentials stored as CI secrets.
41+
42+
| Property | Value |
43+
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
44+
| dbt code source | CI runner's checkout of the repo |
45+
| Compile location | CI runner |
46+
| `profiles.yml` source | Provided by the workflow — either committed as an `env_var()` template, or written to disk from a CI secret at runtime |
47+
| Target | Default in the CI `profiles.yml`, overridden by `--target` |
48+
| UI Connection Settings respected? | No |
49+
| Auth | `LIGHTDASH_API_KEY` from CI secrets |
50+
| Prerequisites | Lightdash CLI + dbt installed in the workflow; warehouse creds available to CI; valid `profiles.yml` produced at runtime |
51+
| Remote git connection required? | No |
52+
| Workflow template | [`deploy.yml`](https://github.com/lightdash/cli-actions) in `lightdash/cli-actions` (GitHub Actions example) |
53+
54+
**Use cases:** automated production deploys gated by PR merge; PR preview workflows (`lightdash start-preview` + `lightdash validate`).
55+
56+
**Trade-offs:** requires duplicating warehouse credentials into CI on top of what's already in Lightdash; needs dbt available in the runner.
57+
58+
<Warning>
59+
Never commit a `profiles.yml` with plaintext credentials. Either use an `env_var()` template committed to the repo, or write the file to disk at runtime from a CI secret.
60+
</Warning>
61+
</Accordion>
62+
63+
<Accordion title='3. "Refresh dbt" button in the Lightdash UI'>
64+
Lightdash fetches the dbt code from the configured remote, compiles it server-side, and updates the project.
65+
66+
| Property | Value |
67+
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
68+
| dbt code source | Lightdash pulls from the remote git source configured in Project Settings (GitHub, GitLab, Azure DevOps, Bitbucket, dbt Cloud) |
69+
| Compile location | Lightdash servers |
70+
| `profiles.yml` source | Lightdash synthesises one from Project Settings → Connection Settings |
71+
| Target | UI Project Settings → Connection Settings → Target name |
72+
| UI Connection Settings respected? | Yes |
73+
| Auth | Logged-in user in the UI |
74+
| Prerequisites | Project must have a remote git source connected |
75+
| Remote git connection required? | Yes |
76+
77+
**Use cases:** ad-hoc refresh after a merge; quick UI-driven sync without leaving the browser.
78+
79+
**Trade-offs:** manual; relies on a human remembering to click it.
80+
</Accordion>
81+
82+
<Accordion title="4. lightdash refresh from CI">
83+
Calls the same endpoint as the "Refresh dbt" button, but programmatically from CI (GitHub Actions, GitLab CI, Azure Pipelines, Bitbucket Pipelines, etc.). Lightdash does the work server-side using the project's UI Connection Settings.
84+
85+
| Property | Value |
86+
| ------------------------------------- | ------------------------------------------------------------------------------------------- |
87+
| dbt code source | Lightdash pulls from the remote git source (configured in Project Settings) |
88+
| Compile location | Lightdash servers |
89+
| `profiles.yml` source | Synthesised by Lightdash from UI Connection Settings |
90+
| Target | UI Project Settings → Connection Settings → Target name |
91+
| UI Connection Settings respected? | Yes |
92+
| Auth | `LIGHTDASH_API_KEY` from CI secrets |
93+
| Prerequisites | Project must have a remote git source connected; Lightdash CLI installed in the workflow |
94+
| Remote git connection required? | Yes |
95+
| Workflow template | [`refresh.yml`](https://github.com/lightdash/cli-actions) in `lightdash/cli-actions` (GitHub Actions example) |
96+
97+
**Use cases:** automated post-merge sync to production when the project is already git-connected.
98+
99+
**Trade-offs:** can't push uncommitted changes (only what's in the connected branch); won't catch issues pre-merge — pair with a separate `lightdash validate` PR workflow if you want that.
100+
</Accordion>
101+
102+
<Accordion title="5. Pinging the refresh API directly from CI">
103+
Same as #4, but bypasses the Lightdash CLI entirely — just a `curl` to the API. Works from any CI system (GitHub Actions, GitLab CI, Azure Pipelines, Bitbucket Pipelines, etc.).
104+
105+
| Property | Value |
106+
| ------------------------------------- | --------------------------------------------------------------------------- |
107+
| dbt code source | Lightdash pulls from the remote git source (configured in Project Settings) |
108+
| Compile location | Lightdash servers |
109+
| `profiles.yml` source | Synthesised by Lightdash from UI Connection Settings |
110+
| Target | UI Project Settings → Connection Settings → Target name |
111+
| UI Connection Settings respected? | Yes |
112+
| Auth | `LIGHTDASH_API_KEY` from CI secrets |
113+
| Prerequisites | Project must have a remote git source connected |
114+
| Remote git connection required? | Yes |
115+
| Lightdash CLI required? | No |
116+
| Endpoint | `POST /api/v1/projects/{projectUuid}/refresh` |
117+
118+
**Use cases:** the simplest possible automated refresh — no dependencies to install. Useful for teams who don't want a Node toolchain in CI.
119+
120+
**Trade-offs:** same as #4. The refresh is async (returns a `jobUuid`); for stronger guarantees, poll `GET /api/v1/jobs/{jobUuid}` to confirm completion before exiting the workflow.
121+
</Accordion>
122+
</AccordionGroup>
123+
124+
### Decision matrix
125+
126+
| | #1 `lightdash deploy` — local CLI | #2 `lightdash deploy` — CI | #3 "Refresh dbt" — UI button | #4 `lightdash refresh` — CI | #5 `POST /refresh` API — CI |
127+
| ------------------------------------- | ------------------------------- | ---------------------------------- | ------------------ | ----------------------------- | ------------------------------ |
128+
| Automated? | No | Yes | No | Yes | Yes |
129+
| Pushes uncommitted local changes? | Yes | No | No | No | No |
130+
| Needs dbt + warehouse creds in CI? | N/A | Yes | No | No | No |
131+
| Needs Lightdash CLI installed? | Yes | Yes | No | Yes | No |
132+
| Respects UI Target name? | No | No | Yes | Yes | Yes |
133+
| Works without a remote git connection?| Yes | Yes | No | No | No |
134+
| Catches breaking changes pre-merge? | With `--select` + `validate` | If used in a PR workflow | No | No | No |
135+
| Best for | Initial setup, debugging | PR previews; CI deploys with no remote git connection | Ad-hoc UI refresh | Post-merge sync (recommended) | Post-merge sync, minimal-CI shops |
136+
137+
### Recommended setups
138+
139+
**Most teams** (Lightdash connected to GitHub, GitLab, Azure DevOps, Bitbucket, or dbt Cloud):
140+
141+
- **PR workflow** → `lightdash start-preview` + `lightdash validate` (mechanism #2 with `start-preview`)
142+
- **Post-merge to `main`** → `lightdash refresh` (#4) or the refresh API (#5)
143+
- **Manual UI refresh** (#3) available for ad-hoc syncs
144+
145+
**Teams without a remote git connection** (project loaded via local CLI only):
146+
147+
- **Production sync** → `lightdash deploy` in CI (#2)
148+
- **Previews** → `lightdash start-preview` in CI
12149

13-
You can sync your dbt project code with Lightdash in a few different ways. We recommend everyone set up continuous deployment, but you can also refresh in the Lightdash app or deploy from the CLI.
150+
**Teams just getting started:**
14151

15-
### 1. Set up continuous deployment
152+
- Local `lightdash deploy` (#1) for first-time setup, then graduate to one of the automated patterns above as soon as the project is stable.
16153

17-
[Read how to do that and check out our example workflow files](/guides/cli/how-to-use-lightdash-deploy#automatically-deploy-your-changes-to-lightdash-using-a-github-action).
154+
### Things to know
18155

19-
### 2. Click "Refresh dbt" in Lightdash
156+
<Info>
157+
**`lightdash deploy` does NOT use the UI "Target name" setting.** It uses the local `profiles.yml` default, or whatever `--target` overrides it with. Only mechanisms #3, #4, and #5 respect the UI Target name.
158+
</Info>
159+
160+
<Info>
161+
**`lightdash refresh` requires a remote git connection.** It will fail with a `ParameterError` if the project's `dbtConnection.type` is `NONE`.
162+
</Info>
163+
164+
<Info>
165+
**Refresh is async.** Both #4 and #5 return a `jobUuid` and the work continues server-side. If your CI needs to know the refresh succeeded, poll `GET /api/v1/jobs/{jobUuid}` before exiting the workflow.
166+
</Info>
167+
168+
<Warning>
169+
**Never commit `profiles.yml` with credentials.** For mechanism #2, use either a secret-written-to-file approach or an `env_var()` template — but never check in plaintext credentials.
170+
</Warning>
171+
172+
<Tip>
173+
Mechanisms #4 and #5 are underused. Most existing CI examples lead with `lightdash deploy`, which is the heaviest option in CI (requires dbt + warehouse credentials duplicated into the runner). The refresh-based patterns are simpler and often a better fit when the project is git-connected.
174+
</Tip>
20175

21-
The button can be found on the Query from tables page.
176+
### How to trigger each sync
177+
178+
#### Click "Refresh dbt" in Lightdash
179+
180+
The button can be found on the Query from tables page.
22181

23182
<Frame>
24183
![screenshot-refresh-dbt](/images/references/integrations/dbt-projects/dbt-refresh.png)
25184
</Frame>
26185

27-
_If you're using a git connection (like GitHub, Gitlab or Bitbucket), you'll need to push + merge your changes to the branch that your Lightdash project is connected to before you press `Refresh dbt`._
186+
_If you're using a git connection (like GitHub, GitLab, or Bitbucket), you'll need to push + merge your changes to the branch that your Lightdash project is connected to before you press `Refresh dbt`._
28187

29188
<Info>
30-
If you've made any changes to the underlying data (for example, adding a new column in your `model.sql` file or changing the SQL logic of a dimension), then you need to run: `dbt run -m yourmodel` before you click `Refresh dbt` in Lightdash.
189+
If you've made any changes to the underlying data (for example, adding a new column in your `model.sql` file or changing the SQL logic of a dimension), then you need to run `dbt run -m yourmodel` before you click `Refresh dbt` in Lightdash.
31190
</Info>
32191

33-
### 3. Push code from the CLI
192+
#### Push code from the CLI
34193

35194
If you're using the [Lightdash CLI](/guides/cli/how-to-install-the-lightdash-cli), you can use `lightdash deploy` to deploy your changes to Lightdash. [Read more about how to use `lightdash deploy`](/guides/cli/how-to-use-lightdash-deploy).
36195

37196
<Warning>
38197
We don't recommend using `lightdash deploy` from your local environment as the primary way you update Lightdash since small mistakes can lead to production issues.
39198
</Warning>
40199

200+
#### Set up continuous deployment
201+
202+
[Read how to do that and check out our example workflow files](/guides/cli/how-to-use-lightdash-deploy#automatically-deploy-your-changes-to-lightdash-using-a-github-action).
203+
41204
## dbt project settings
42205

43206
For more information about dbt connection types (Github, Gitlab, Bitbucket, etc.) and the fields required for each type, [read the dbt project section in our connection guide](/get-started/setup-lightdash/connect-project#2-import-a-dbt-project).

0 commit comments

Comments
 (0)