You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/quickstart/notebook.md
+31-40Lines changed: 31 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,8 +27,7 @@ Inform SQLMesh of the project location by setting a context with the `%context`
27
27
28
28
You can specify multiple directories in one call to `%context` if your SQLMesh project has [multiple repositories](../guides/multi_repo.md).
29
29
30
-
## 2. Plan and apply environments
31
-
### 2.1 Create a prod environment
30
+
## 2. Create a prod environment
32
31
33
32
SQLMesh's key actions are creating and applying *plans* to *environments*. At this point, the only environment is the empty `prod` environment.
34
33
@@ -57,28 +56,10 @@ The first output block shows the completion percentage and run time for each mod
57
56
58
57
You've now created a new production environment with all of history backfilled.
59
58
60
-
### 2.2 Create a dev environment
61
-
Now that you've created a production environment, it's time to create a development environment so that you can modify models without affecting production.
59
+
## 3. Update a model
62
60
63
-
Run `%plan dev` to create a development environment called `dev`. The following output will be displayed:
64
-
65
-
66
-
67
-
The output does not list any added or modified models because `dev` is being created from the existing `prod` environment without modification.
68
-
69
-
The `New environment` line shows that when you apply the plan creating the `dev` environment, it will only involve a Virtual Update. This is because SQLMesh is able to safely reuse the tables you've already backfilled in the `prod` environment.
70
-
71
-
Click the green button to perform the Virtual Update:
72
-
73
-
74
-
75
-
The output confirms that the `dev` environment has been updated successfully.
76
-
77
-
## 3. Make your first update
61
+
Now that we have have populated the `prod` environment, let's modify one of the SQL models.
78
62
79
-
Now that we have have populated both `prod` and `dev` environments, let's modify one of the SQL models, validate it in `dev`, and push it to `prod`.
80
-
81
-
### 3.1 Edit the configuration
82
63
We can modify the incremental SQL model using the `%model`*line* notebook magic (note the single `%`) and the model name:
83
64
84
65
@@ -91,59 +72,69 @@ We modify the incremental SQL model by adding a new column to the query. When we
91
72
92
73
93
74
94
-
## 4. Plan and apply updates
95
-
We can preview the impact of the change using the `%plan dev` command:
75
+
## 4. Work with a development environment
76
+
77
+
### 4.1 Create a dev environment
78
+
Now that you've modified a model, it's time to create a development environment so that you can validate the model change without affecting production.
96
79
97
-
80
+
Run `%plan dev` to create a development environment called `dev`. The following output will be displayed:
81
+
82
+
98
83
99
84
The first block of output notes that `%plan` successfully executed the project's test `tests/test_full_model.yaml` with duckdb.
100
85
101
-
The second block "Summary of differences against `dev`" summarizes the differences between the modified project components and the existing `dev` environment, detecting that we directly modified `incremental_model` and that `full_model` was indirectly modified because it selects from the incremental model.
86
+
The `New environment` line describes what environments the plan will affect when applied - a new `dev` environment will be created from the existing `prod` environment.
102
87
103
-
The third block shows the modifications we made to the rendered query.
88
+
The `Summary of differences` section summarizes the differences between the modified model and the new `dev` environment (right now just a copy of `prod`), detecting that we directly modified `incremental_model` and that `full_model` was indirectly modified because it selects from the incremental model. It shows a diff between the existing and updated model.
104
89
105
-
The next block shows that we directly modified the incremental model. SQLMesh understood that the change was additive (added a column not used by `full_model`) and automatically classified it as a non-breaking change.
90
+
SQLMesh automatically classified the change as `Non-breaking` because understood that the change was additive (added a column not used by `full_model`) and did not invalidate any data already in `prod`.
106
91
107
-
The final block describes the models requiring backfill, including the incremental model from our start date `2020-01-01`. We can modify the backfill start and end dates for the incremental model with the date picker widget, if desired.
92
+
The `Models needing backfill` section shows that only the directly modified `incremental_model` needs backfill and provides a date picker to specify the start and end dates for the backfill.
108
93
109
-
Click the green `Apply - Backfill Tables`button to apply the plan and execute the backfill:
94
+
Click the green button to perform the backfill:
110
95
111
-
96
+
97
+
98
+
The output shows that SQLMesh created a new model version in `dev`. The last line of the output shows that SQLMesh applied the change to `sqlmesh_example__dev.incremental_model`. In the model schema, the suffix "`__dev`" indicates that it is in the `dev` environment.
112
99
113
-
SQLMesh applies the change to `sqlmesh_example.incremental_model` and backfills the model. SQLMesh did not need to backfill `sqlmesh_example.full_model` since the change was `non-breaking`.
100
+
SQLMesh did not need to backfill anything for the `full_model` since the change was `Non-breaking`.
114
101
115
-
### 4.1 Validate updates in dev
102
+
### 4.2 Validate updates in dev
116
103
You can now view this change by querying data from `incremental_model` with the `%%fetchdf`*cell* magic (note the two `%` symbols) and the SQL query `select * from sqlmesh_example__dev.incremental_model`.
117
104
118
-
Note that the environment name `__dev` is appended to the schema namespace `sqlmesh_example` in the query:`select * from sqlmesh_example__dev.incremental_model`.
105
+
Note that the environment name `__dev` is appended to the schema namespace `sqlmesh_example` in the query:
119
106
120
107
121
108
122
109
You can see that `new_column` was added to the dataset.
123
110
124
-
The production table was not modified; you can validate this by querying the production table using `%%fetchdf` and the query `select * from sqlmesh_example.incremental_model`. Note that nothing has been appended to the schema namespace `sqlmesh_example` because `prod` is the default environment.
111
+
The production table was not modified; you can validate this by querying the production table using `%%fetchdf` and the query `select * from sqlmesh_example.incremental_model`.
112
+
113
+
Note that nothing has been appended to the schema namespace `sqlmesh_example` because `prod` is the default environment:
125
114
126
115
127
116
128
117
The production table does not have `new_column` because the changes to `dev` have not yet been applied to `prod`.
129
118
130
-
### 4.2 Apply updates to prod
131
-
Now that we've tested the changes in dev, it's time to move them to prod. Run `%plan` to plan and apply your changes to the prod environment.
119
+
## 5. Update the prod environment
120
+
Now that we've tested the changes in dev, it's time to move them to production. Run `%plan` to plan and apply your changes to the `prod` environment:
121
+
122
+
132
123
133
124
Click the green `Apply - Virtual Update` button to apply the plan and execute the backfill:
134
125
135
-
126
+
136
127
137
128
Note that a backfill was not necessary and only a Virtual Update occurred.
138
129
139
-
### 4.3. Validate updates in prod
130
+
### 5.2 Validate updates in prod
140
131
Double-check that the data updated in `prod` by running `%%fetchdf` with the SQL query `select * from sqlmesh_example.incremental_model`:
141
132
142
133
143
134
144
135
`new_column` is now present in the `prod` incremental model.
145
136
146
-
## 5. Next steps
137
+
## 6. Next steps
147
138
148
139
Congratulations, you've now conquered the basics of using SQLMesh!
0 commit comments