Skip to content

Commit 73eadd9

Browse files
erindrutreyspsungchun12
authored
Docs: Add docs for the Dagster facade (#3736)
Co-authored-by: Trey Spiller <1831878+treysp@users.noreply.github.com> Co-authored-by: Sung Won Chung <sungwonchung3@gmail.com>
1 parent 8f22a31 commit 73eadd9

15 files changed

Lines changed: 393 additions & 17 deletions

docs/cloud/features/scheduler/airflow.md

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,24 @@
22

33
Tobiko Cloud's Airflow integration allows you to combine Airflow system monitoring with the powerful debugging tools in Tobiko Cloud.
44

5+
![Airflow UI DAG view](./airflow/dag_view.png)
6+
7+
## How it works
8+
9+
Tobiko Cloud uses a custom approach to Airflow integration.
10+
11+
The Airflow DAG task mirrors the progress of the Tobiko Cloud scheduler run. Each local task reflects the outcome of its corresponding remote task.
12+
13+
This allows you to observe at a glance how your data pipeline is progressing, displayed alongside your other pipelines in Airflow. No need to context switch to Tobiko Cloud!
14+
15+
### Why a custom approach?
16+
17+
Tobiko Cloud's scheduler performs multiple optimizations to ensure that your pipelines run correctly and efficiently. Those optimizations are only possible within our SQLMesh-aware scheduler.
18+
19+
Our approach allows you to benefit from those optimizations while retaining the flexibility to attach extra tasks or logic to the DAG in your broader pipeline orchestration context.
20+
21+
Because `run`s are still triggered by the Tobiko Cloud scheduler and tasks in the local DAG just reflect their remote equivalent in Tobiko Cloud, we call our custom approach a *facade*.
22+
523
## Setup
624

725
Your SQLMesh project must be configured and connected to Tobiko Cloud before using the Airflow integration.
@@ -97,22 +115,6 @@ You can browse the DAG just like any other - each node is a SQLMesh model:
97115

98116
![Airflow UI DAG view](./airflow/dag_view.png)
99117

100-
## How it works
101-
102-
Tobiko Cloud uses a custom approach to Airflow integration - this section describes how it works.
103-
104-
The Airflow DAG task mirrors the progress of the Tobiko Cloud scheduler run. Each local task reflects the outcome of its corresponding remote task.
105-
106-
This allows you to observe at a glance how your data pipeline is progressing, displayed alongside your other pipelines in Airflow. No need to navigate to Tobiko Cloud!
107-
108-
### Why a custom approach?
109-
110-
Tobiko Cloud's scheduler performs multiple optimizations to ensure that your pipelines run correctly and efficiently. Those optimizations are only possible within our SQLMesh-aware scheduler.
111-
112-
Our approach allows you to benefit from those optimizations while retaining the flexibility to attach extra tasks or logic to the DAG in your broader pipeline orchestration context.
113-
114-
Because `run`s are still triggered by the Tobiko Cloud scheduler and tasks in the local DAG just reflect their remote equivalent in Tobiko Cloud, we call our custom approach a *facade*.
115-
116118
## Debugging
117119

118120
Each task in the local DAG writes logs that include a link to its corresponding remote task in Tobiko Cloud.

docs/cloud/features/scheduler/dagster.md

Lines changed: 375 additions & 1 deletion
Large diffs are not rendered by default.
47.1 KB
Loading
35 KB
Loading
85.7 KB
Loading
121 KB
Loading
14.1 KB
Loading
3.11 KB
Loading
23 KB
Loading
126 KB
Loading

0 commit comments

Comments
 (0)