You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts/macros/sqlmesh_macros.md
+15-9Lines changed: 15 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,6 +27,7 @@ It uses the following five step approach to accomplish this:
27
27
1. Parse the text with the appropriate sqlglot SQL dialect (e.g., Postgres, BigQuery, etc.). During the parsing, it detects the special macro symbol `@` to differentiate non-SQL from SQL text. The parser builds a semantic representation of the SQL code's structure, capturing non-SQL text as "placeholder" values to use in subsequent steps.
28
28
29
29
2. Examine the placeholder values to classify them as one of the following types:
30
+
30
31
- Creation of user-defined macro variables with the `@DEF` operator (see more about [user-defined macro variables](#user-defined-variables))
31
32
- Macro variables, both [SQLMesh pre-defined](./macro_variables.md) and [user-defined](#User-defined-variables)
32
33
- Macro functions, both [SQLMesh's](#sqlmesh-macro-operators) and [user-defined](#user-defined-macro-functions)
@@ -154,6 +155,7 @@ The second argument `number -> number` tells SQLMesh what action should be taken
154
155
The right side of the arrow specifies what should be done to each item in the list. `number -> number` tells `@EACH` that for each item `number` it should return that item (e.g., `1`).
155
156
156
157
SQLMesh macros use their semantic understanding of SQL code to take automatic actions based on where in a SQL query macro variables are used. If `@EACH` is used in the `SELECT` clause of a SQL statement:
158
+
157
159
1. It prints the item
158
160
2. It knows fields are separated by commas in `SELECT`, so it automatically separates the printed items with commas
159
161
@@ -182,6 +184,7 @@ FROM table
182
184
```
183
185
184
186
In this code each number is being used in two distinct ways. For example, `4` is being used:
187
+
185
188
1. As a literal numeric value in `favorite_number = 4`
186
189
2. As part of a column name in `favorite_4`
187
190
@@ -230,6 +233,7 @@ FROM table
230
233
SQLMesh's `@IF` macro allows components of a SQL query to change based on the result of a logical condition.
231
234
232
235
It includes three elements:
236
+
233
237
1. A logical condition that evaluates to `TRUE` or `FALSE`
234
238
2. A value to return if the condition is `TRUE`
235
239
3. A value to return if the condition is `FALSE`[optional]
@@ -239,12 +243,14 @@ These elements are specified as `@IF([logical condition], [value if TRUE], [valu
239
243
The value to return if the condition is `FALSE` is optional - if it is not provided and the condition is `FALSE`, then the macro has no effect on the resulting query.
240
244
241
245
The logical condition should be written *in SQL* and is evaluated with [SQLGlot's](https://github.com/tobymao/sqlglot) SQL engine. It supports the following operators:
246
+
242
247
- Equality: `=` for equals, `!=` or `<>` for not equals
243
248
- Comparison: `<`, `>`, `<=`, `>=`,
244
249
- Between: `[number] BETWEEN [low number] AND [high number]`
245
250
- Membership: `[item] IN ([comma-separated list of items])`
246
251
247
252
For example, the following simple conditions are all valid SQL and evaluate to `TRUE`:
253
+
248
254
-`'a' = 'a'`
249
255
-`'a' != 'b'`
250
256
-`0 < 1`
@@ -280,11 +286,11 @@ Like [`@EACH`](#each-basics), they take two arguments: an array of items in brac
280
286
281
287
#### @FILTER
282
288
283
-
`@FILTER` is used to subset an input array of items to only those meeting the logical condition specified in the anonymous function. Its output can be consumed by other macro operators such as [`@EACH`](#each-basics) or `@REDUCE`.
289
+
`@FILTER` is used to subset an input array of items to only those meeting the logical condition specified in the anonymous function. Its output can be consumed by other macro operators such as [`@EACH`](#each-basics) or [`@REDUCE`](#reduce).
284
290
285
291
The user-specified anonymous function must evaluate to `TRUE` or `FALSE`. `@FILTER` applies the function to each item in the array, only including the item in the output array if it meets the condition.
286
292
287
-
The anonymous function should be written *in SQL* and is evaluated with [SQLGlot's](https://github.com/tobymao/sqlglot) SQL engine. It supports standard SQL equality and comparison operators - see [`@IF`](#if) above for more information.
293
+
The anonymous function should be written *in SQL* and is evaluated with [SQLGlot's](https://github.com/tobymao/sqlglot) SQL engine. It supports standard SQL equality and comparison operators - see [`@IF`](#if) above for more information about supported operators.
288
294
289
295
For example, consider this `@FILTER` call:
290
296
@@ -364,7 +370,7 @@ Each of these operators is used to dynamically add the code for its correspondin
364
370
#### How SQL clause operators work
365
371
The SQL clause operators take a single argument that determines whether the clause is generated.
366
372
367
-
If the argument is True the clause code is generated, if False the code is not. The argument should be written *in SQL* and its value is evaluated with [SQLGlot's](https://github.com/tobymao/sqlglot) SQL engine.
373
+
If the argument is `TRUE` the clause code is generated, if `FALSE` the code is not. The argument should be written *in SQL* and its value is evaluated with [SQLGlot's](https://github.com/tobymao/sqlglot) SQL engine.
368
374
369
375
Each SQL clause operator may only be used once in a query, but any common table expressions or subqueries may contain their own single use of the operator as well.
370
376
@@ -387,11 +393,11 @@ SELECT
387
393
count(distinct id) AS num_orders,
388
394
FROM
389
395
sqlmesh_example.incremental_model
390
-
@WHERE(True) item_id > @size
396
+
@WHERE(TRUE) item_id > @size
391
397
GROUP BY item_id
392
398
```
393
399
394
-
The `@WHERE` argument is set to `True`, so the WHERE code is included in the rendered model:
400
+
The `@WHERE` argument is set to `TRUE`, so the WHERE code is included in the rendered model:
395
401
396
402
```sql linenums="1"
397
403
SELECT
@@ -403,9 +409,9 @@ WHERE item_id > 1
403
409
GROUP BY item_id
404
410
```
405
411
406
-
If the `@WHERE` argument were instead set to `False` the `WHERE` clause would be omitted from the query.
412
+
If the `@WHERE` argument were instead set to `FALSE` the `WHERE` clause would be omitted from the query.
407
413
408
-
These operators aren't too useful if the argument's value is hard-coded. Instead, the argument can consist of code executable by Python.
414
+
These operators aren't too useful if the argument's value is hard-coded. Instead, the argument can consist of code executable by the SQLGlot SQL engine.
409
415
410
416
For example, the `WHERE` clause will be included in this query because 1 less than 2:
411
417
@@ -453,7 +459,7 @@ FROM
453
459
GROUP BY item_id
454
460
```
455
461
456
-
The argument to `@WHERE` will be "1 < 2" as in the previous example after the macro variables `left_number` and `right_number` are substituted in.
462
+
The argument to `@WHERE` will be "1 < 2" as in the previous hard-coded example after the macro variables `left_number` and `right_number` are substituted in.
457
463
458
464
### SQL clause operator examples
459
465
@@ -687,7 +693,7 @@ CASE WHEN vehicle = 'bus' THEN 'bus' ELSE NULL END AS vehicle_bus,
687
693
FROM table
688
694
```
689
695
690
-
Note that in the call `@make_indicators('vehicle', ['truck', 'bus'])`, all three strings are quoted. Those quotes do not propagate into the function body, `CASE WHEN vehicle` does not include quotes around `vehicle`. Because the quotes aren't propagated, we had to include the quotes around '{value}' in the function definition.
696
+
Note that in the call `@make_indicators('vehicle', ['truck', 'bus'])`, all three strings are quoted. Those quotes do not propagate into the function body, `CASE WHEN vehicle` does not include quotes around `vehicle`. Because the quotes aren't propagated, we had to include the quotes around `'{value}'` in the function definition.
0 commit comments