Skip to content

Commit 4b357fd

Browse files
committed
fix: trino lineage namespaces
1 parent 4c09cbc commit 4b357fd

2 files changed

Lines changed: 28 additions & 9 deletions

File tree

docs/modules/demos/pages/openlineage-trino-airflow.adoc

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -80,10 +80,15 @@ event-listener.name=openlineage
8080
openlineage-event-listener.transport.type=HTTP
8181
openlineage-event-listener.transport.url=http://marquez:5000
8282
openlineage-event-listener.transport.endpoint=/api/v1/lineage
83-
openlineage-event-listener.trino.uri=http://trino-coordinator:8080
83+
openlineage-event-listener.trino.uri=https://trino-coordinator:8443
84+
openlineage-event-listener.namespace=trino-openlineage-demo
8485
----
8586

86-
See the {trino-openlineage-docs}[Trino OpenLineage event listener docs{external-link-icon}^] for details.
87+
`trino.uri` is Trino's own serving address; it names the *dataset* namespace, so pointing it at the same
88+
`trino-coordinator:8443` that Airflow uses lets both producers report the same tables under one namespace.
89+
`namespace` sets an explicit, colon-free *job* namespace - without it Marquez cannot parse the node ID of a
90+
Trino query job (its ID starts with a digit). See the {trino-openlineage-docs}[Trino OpenLineage event
91+
listener docs{external-link-icon}^] for details.
8792

8893
*Airflow* only needs the provider pointed at Marquez, which is done with `envOverrides`:
8994

@@ -178,9 +183,12 @@ image::openlineage/job-details-customer-orders-2.png[Marquez job details for loa
178183
With the Marquez UI still open at http://localhost:3000, use the namespace selector at the top to switch
179184
between the namespaces that produced the lineage:
180185

181-
* the Trino namespace (named after `openlineage-event-listener.trino.uri`, i.e. `http://trino-coordinator:8080`),
182-
which contains the `tpch.tiny.*` and `memory.demo.*` datasets and the queries connecting them, and
183-
* `airflow-openlineage-demo`, which contains the `openlineage_demo` DAG and its tasks.
186+
* `trino-openlineage-demo` - the Trino query jobs (one per executed statement), set via
187+
`openlineage-event-listener.namespace`, and
188+
* `trino://trino-coordinator:8443` - the datasets (`tpch.tiny.*` and `memory.demo.*`), named after the Trino
189+
coordinator's address. Both the Trino event listener and the Airflow SQL operator report the tables under
190+
this shared namespace, and
191+
* `airflow-openlineage-demo` - the `openlineage_demo` DAG and its tasks.
184192

185193
Selecting the `memory.demo.customer_orders` dataset and expanding upstream shows the full graph back to
186194
`tpch.tiny.customer` and `tpch.tiny.orders`, including column-level lineage for the individual queries:
@@ -189,9 +197,10 @@ image::openlineage/customer-orders-2.png[Marquez lineage graph tracing memory.de
189197

190198
[NOTE]
191199
====
192-
Trino and Airflow name datasets independently, so a given table may appear under both the Trino namespace (from the
193-
Trino event listener) and the Airflow namespace (from the SQL operator's extractor). This is expected - each producer
194-
reports lineage from its own perspective.
200+
Trino and Airflow report lineage independently but, because `trino.uri` matches the address Airflow uses for its
201+
Trino connection, both name the tables under the same `trino://trino-coordinator:8443` namespace. Their lineage
202+
therefore converges on one shared set of dataset nodes: Trino contributes column-level detail per query, while
203+
Airflow contributes the task/DAG structure around the same tables.
195204
====
196205

197206
== Summary

stacks/openlineage/trino.yaml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,17 @@ data:
4242
openlineage-event-listener.transport.type=HTTP
4343
openlineage-event-listener.transport.url=http://marquez:5000
4444
openlineage-event-listener.transport.endpoint=/api/v1/lineage
45-
openlineage-event-listener.trino.uri=http://trino-coordinator:8080
45+
# Must be Trino's real serving address (HTTPS on 8443 - TLS is enabled and 8080 is not exposed).
46+
# This is the naming string for the *dataset* namespace; matching the address Airflow uses in its
47+
# Trino connection makes both producers name the same tables `trino://trino-coordinator:8443`, so
48+
# their lineage attaches to one shared set of dataset nodes instead of two duplicate namespaces.
49+
openlineage-event-listener.trino.uri=https://trino-coordinator:8443
50+
# Set an explicit, colon-free job namespace. Without this the namespace defaults to the dataset
51+
# namespace (`trino://trino-coordinator:8443`), and Marquez's NodeId parser cannot split
52+
# `job:trino://trino-coordinator:8443:<queryId>` because Trino query IDs start with a digit -
53+
# its URI heuristic then treats the `:<queryId>` delimiter as a `host:port` colon and 500s when
54+
# a job is opened. A plain namespace keeps job NodeIds parseable.
55+
openlineage-event-listener.namespace=trino-openlineage-demo
4656
---
4757
apiVersion: trino.stackable.tech/v1alpha1
4858
kind: TrinoCluster

0 commit comments

Comments
 (0)