Skip to content

Commit 4c09cbc

Browse files
committed
docs: update openlineage doc
1 parent 0ebba77 commit 4c09cbc

6 files changed

Lines changed: 51 additions & 3 deletions

File tree

103 KB
Loading
51.7 KB
Loading
163 KB
Loading
84.3 KB
Loading
117 KB
Loading

docs/modules/demos/pages/openlineage-trino-airflow.adoc

Lines changed: 51 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,37 @@ include::partial$instance-hint.adoc[]
119119
Marquez is not an SDP product, so it does not appear in `stackablectl stacklet list`.
120120
====
121121

122-
== Explore the lineage in Marquez
122+
== Verify the demo worked
123+
124+
The demo automatically triggers the `openlineage_demo` DAG once during installation, so a completed run
125+
and its lineage should already be present. You can confirm the full flow - Airflow orchestrating, Trino
126+
executing, and Marquez collecting the lineage - across three UIs.
127+
128+
=== Airflow: the DAG tasks ran
129+
130+
Open the Airflow web UI using the `webserver-http` endpoint from the `stackablectl stacklet list` output
131+
above, and log in with the username `admin` and the password `adminadmin`.
132+
133+
Under *Dags* -> `openlineage_demo` -> *Task Instances* you should see the four tasks of the pipeline -
134+
`create_schema`, `load_customers`, `load_orders` and `customer_orders` - all in the `success` state:
135+
136+
image::openlineage/airflow.png[Airflow Task Instances view showing the four openlineage_demo tasks in the success state]
137+
138+
Each task is a `SQLExecuteQueryOperator` that submits a statement to Trino.
139+
140+
=== Trino: the queries finished
141+
142+
Open the Trino web UI using the `coordinator-https` endpoint from the `stackablectl stacklet list`
143+
output, and log in with the username `admin` and the password `adminadmin`.
144+
145+
The *Query Details* page lists the statements Airflow submitted - the `CREATE SCHEMA`, `DROP TABLE` and
146+
`CREATE TABLE ... AS SELECT` statements that build the `memory.demo.*` tables from the `tpch.tiny.*`
147+
source - all in the `FINISHED` state. Their *Source* is `airflow`, confirming they originated from the
148+
DAG rather than an interactive session:
149+
150+
image::openlineage/trino-queries.png[Trino Query Details page showing the finished CREATE TABLE queries submitted by Airflow]
151+
152+
=== Marquez: the lineage was collected
123153

124154
Marquez does not have an external listener. Forward its web UI to your machine:
125155

@@ -130,14 +160,32 @@ $ kubectl -n openlineage-demo port-forward svc/marquez-web 3000:3000
130160

131161
Then open http://localhost:3000 in your browser.
132162

133-
Use the namespace selector at the top to switch between the namespaces that produced the lineage:
163+
The landing page lists the OpenLineage *Jobs* that Trino and Airflow reported, together with the event,
164+
dataset and source counts. You should see the `openlineage_demo` DAG job and the per-query jobs such as
165+
`openlineage_demo.customer_orders.query.*`, all *COMPLETED*:
166+
167+
image::openlineage/marquez.png[Marquez home page listing the collected OpenLineage jobs as COMPLETED]
168+
169+
Selecting a job opens its details. For example, `openlineage_demo.load_orders.query.2` shows the exact
170+
`CREATE TABLE memory.demo.orders AS SELECT ...` statement along with its OpenLineage *job* and *run*
171+
facets rendered as JSON - including the `sql` facet, the `AIRFLOW` integration, and the `externalQuery`
172+
facet that links back to the originating Trino query ID:
173+
174+
image::openlineage/job-details-customer-orders-2.png[Marquez job details for load_orders showing the SQL statement and OpenLineage facets as JSON]
175+
176+
== Explore the lineage in Marquez
177+
178+
With the Marquez UI still open at http://localhost:3000, use the namespace selector at the top to switch
179+
between the namespaces that produced the lineage:
134180

135181
* the Trino namespace (named after `openlineage-event-listener.trino.uri`, i.e. `http://trino-coordinator:8080`),
136182
which contains the `tpch.tiny.*` and `memory.demo.*` datasets and the queries connecting them, and
137183
* `airflow-openlineage-demo`, which contains the `openlineage_demo` DAG and its tasks.
138184

139185
Selecting the `memory.demo.customer_orders` dataset and expanding upstream shows the full graph back to
140-
`tpch.tiny.customer` and `tpch.tiny.orders`, including column-level lineage for the individual queries.
186+
`tpch.tiny.customer` and `tpch.tiny.orders`, including column-level lineage for the individual queries:
187+
188+
image::openlineage/customer-orders-2.png[Marquez lineage graph tracing memory.demo.customer_orders upstream through the jobs to tpch.tiny.customer and tpch.tiny.orders]
141189

142190
[NOTE]
143191
====

0 commit comments

Comments
 (0)