|
2 | 2 |
|
3 | 3 | Tobiko Cloud's Airflow integration allows you to combine Airflow system monitoring with the powerful debugging tools in Tobiko Cloud. |
4 | 4 |
|
| 5 | + |
| 6 | + |
| 7 | +## How it works |
| 8 | + |
| 9 | +Tobiko Cloud uses a custom approach to Airflow integration. |
| 10 | + |
| 11 | +The Airflow DAG task mirrors the progress of the Tobiko Cloud scheduler run. Each local task reflects the outcome of its corresponding remote task. |
| 12 | + |
| 13 | +This allows you to observe at a glance how your data pipeline is progressing, displayed alongside your other pipelines in Airflow. No need to context switch to Tobiko Cloud! |
| 14 | + |
| 15 | +### Why a custom approach? |
| 16 | + |
| 17 | +Tobiko Cloud's scheduler performs multiple optimizations to ensure that your pipelines run correctly and efficiently. Those optimizations are only possible within our SQLMesh-aware scheduler. |
| 18 | + |
| 19 | +Our approach allows you to benefit from those optimizations while retaining the flexibility to attach extra tasks or logic to the DAG in your broader pipeline orchestration context. |
| 20 | + |
| 21 | +Because `run`s are still triggered by the Tobiko Cloud scheduler and tasks in the local DAG just reflect their remote equivalent in Tobiko Cloud, we call our custom approach a *facade*. |
| 22 | + |
5 | 23 | ## Setup |
6 | 24 |
|
7 | 25 | Your SQLMesh project must be configured and connected to Tobiko Cloud before using the Airflow integration. |
@@ -97,22 +115,6 @@ You can browse the DAG just like any other - each node is a SQLMesh model: |
97 | 115 |
|
98 | 116 |  |
99 | 117 |
|
100 | | -## How it works |
101 | | - |
102 | | -Tobiko Cloud uses a custom approach to Airflow integration - this section describes how it works. |
103 | | - |
104 | | -The Airflow DAG task mirrors the progress of the Tobiko Cloud scheduler run. Each local task reflects the outcome of its corresponding remote task. |
105 | | - |
106 | | -This allows you to observe at a glance how your data pipeline is progressing, displayed alongside your other pipelines in Airflow. No need to navigate to Tobiko Cloud! |
107 | | - |
108 | | -### Why a custom approach? |
109 | | - |
110 | | -Tobiko Cloud's scheduler performs multiple optimizations to ensure that your pipelines run correctly and efficiently. Those optimizations are only possible within our SQLMesh-aware scheduler. |
111 | | - |
112 | | -Our approach allows you to benefit from those optimizations while retaining the flexibility to attach extra tasks or logic to the DAG in your broader pipeline orchestration context. |
113 | | - |
114 | | -Because `run`s are still triggered by the Tobiko Cloud scheduler and tasks in the local DAG just reflect their remote equivalent in Tobiko Cloud, we call our custom approach a *facade*. |
115 | | - |
116 | 118 | ## Debugging |
117 | 119 |
|
118 | 120 | Each task in the local DAG writes logs that include a link to its corresponding remote task in Tobiko Cloud. |
|
0 commit comments