Skip to content

Commit 4dfcc22

Browse files
razvanrazvan
committed
feat: new openlineage demo (raw and untested)
1 parent dfd9cc9 commit 4dfcc22

9 files changed

Lines changed: 618 additions & 0 deletions

File tree

‎demos/demos-v2.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -313,3 +313,21 @@ demos:
313313
- name: branch
314314
description: Git branch to download demo files from
315315
default: main
316+
openlineage-trino-airflow:
317+
description: Showcase OpenLineage data lineage from Trino and Airflow, visualized in Marquez
318+
documentation: https://docs.stackable.tech/home/stable/demos/openlineage-trino-airflow
319+
stackableStack: openlineage
320+
labels:
321+
- trino
322+
- airflow
323+
- openlineage
324+
- marquez
325+
- data-lineage
326+
manifests:
327+
- plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/demos/openlineage-trino-airflow/rbac.yaml
328+
- plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/demos/openlineage-trino-airflow/enable-and-run-dag.yaml
329+
supportedNamespaces: []
330+
resourceRequests:
331+
cpu: 5000m
332+
memory: 12Gi
333+
pvc: 15Gi

‎demos/openlineage-trino-airflow/enable-and-run-dag.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
# Waits for Airflow to be ready, then un-pauses and triggers the `openlineage_demo` DAG via the
3+
# Airflow REST API. Running the DAG is what makes Trino execute the queries, so this is what
4+
# populates Marquez with lineage.
5+
apiVersion: batch/v1
6+
kind: Job
7+
metadata:
8+
name: enable-and-run-openlineage-dag
9+
spec:
10+
template:
11+
spec:
12+
containers:
13+
- name: enable-and-run-openlineage-dag
14+
image: oci.stackable.tech/sdp/tools:1.0.0-stackable0.0.0-dev
15+
command:
16+
- bash
17+
- -euo
18+
- pipefail
19+
- -c
20+
- |
21+
kubectl rollout status --watch statefulset/airflow-webserver-default
22+
kubectl rollout status --watch statefulset/airflow-scheduler-default
23+
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-credentials/adminUser.password)
24+
ACCESS_TOKEN=$(curl -XPOST http://airflow-webserver-default-headless:8080/auth/token -H 'Content-Type: application/json' -d '{"username": "admin", "password": "'$AIRFLOW_ADMIN_PASSWORD'"}' | jq -r .access_token)
25+
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPATCH http://airflow-webserver-default-headless:8080/api/v2/dags/openlineage_demo -d '{"is_paused": false}' | jq
26+
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPOST http://airflow-webserver-default-headless:8080/api/v2/dags/openlineage_demo/dagRuns -d '{"logical_date": null}' | jq
27+
volumeMounts:
28+
- name: airflow-credentials
29+
mountPath: /airflow-credentials
30+
volumes:
31+
- name: airflow-credentials
32+
secret:
33+
secretName: airflow-credentials
34+
restartPolicy: OnFailure
35+
backoffLimit: 20 # give some time for the Airflow cluster to be available

‎demos/openlineage-trino-airflow/rbac.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
# Allow the enable-and-run Job (running under the namespace's default ServiceAccount) to wait for the
3+
# Airflow StatefulSets to become ready via `kubectl rollout status`.
4+
apiVersion: rbac.authorization.k8s.io/v1
5+
kind: ClusterRole
6+
metadata:
7+
name: openlineage-demo-clusterrole
8+
rules:
9+
- apiGroups:
10+
- ""
11+
resources:
12+
- pods
13+
verbs:
14+
- get
15+
- list
16+
- watch
17+
- apiGroups:
18+
- apps
19+
resources:
20+
- statefulsets
21+
verbs:
22+
- get
23+
- list
24+
- watch
25+
---
26+
apiVersion: rbac.authorization.k8s.io/v1
27+
kind: ClusterRoleBinding
28+
metadata:
29+
name: openlineage-demo-clusterrolebinding
30+
roleRef:
31+
apiGroup: rbac.authorization.k8s.io
32+
kind: ClusterRole
33+
name: openlineage-demo-clusterrole
34+
subjects:
35+
- apiGroup: rbac.authorization.k8s.io
36+
kind: Group
37+
name: system:serviceaccounts

‎docs/modules/demos/pages/openlineage-trino-airflow.adoc‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
= openlineage-trino-airflow
2+
:description: Demo showcasing OpenLineage data lineage emitted by Trino and Airflow and visualized in Marquez, enabled with minimal configuration.
3+
4+
:openlineage: https://openlineage.io/
5+
:marquez: https://marquezproject.ai/
6+
:trino-openlineage-docs: https://trino.io/docs/current/admin/event-listeners-openlineage.html
7+
:airflow-openlineage-docs: https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html
8+
9+
This demo shows how two Stackable Data Platform (SDP) products - Apache Trino and Apache Airflow - can emit
10+
{openlineage}[OpenLineage{external-link-icon}^] events with a single configuration toggle each, and how those events
11+
come together in {marquez}[Marquez{external-link-icon}^], the OpenLineage reference metadata server and UI.
12+
13+
Install this demo on an existing Kubernetes cluster:
14+
15+
[NOTE]
16+
====
17+
The namespace `openlineage-demo` will be assumed in this guide.
18+
It will be created if it doesn't exist.
19+
====
20+
21+
[source,console]
22+
----
23+
$ stackablectl demo install openlineage-trino-airflow -n openlineage-demo
24+
----
25+
26+
[WARNING]
27+
====
28+
This demo should not be run alongside other demos.
29+
====
30+
31+
[#system-requirements]
32+
== System requirements
33+
34+
To run this demo, your system needs at least:
35+
36+
* 5 https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#cpu[cpu units{external-link-icon}^] (core/hyperthread)
37+
* 12GiB memory
38+
* 15GiB disk storage
39+
40+
== Overview
41+
42+
This demo will
43+
44+
* Install the required Stackable operators
45+
* Spin up the following products
46+
** *Trino*: A fast distributed SQL query engine. It is configured with the built-in, first-party `openlineage` event
47+
listener so that every query emits table- and column-level lineage.
48+
** *Airflow*: A workflow orchestrator. The native `apache-airflow-providers-openlineage` provider is enabled so that
49+
DAG and task runs - including the datasets they read and write - are reported as lineage.
50+
** *PostgreSQL*: Metadata database for Airflow.
51+
** *Marquez*: The OpenLineage reference server + web UI (with its own PostgreSQL), used here as the consuming
52+
application that collects and visualizes the lineage. Marquez is only part of this demo, not an SDP product.
53+
* Mount and trigger a single Airflow DAG (`openlineage_demo`) that runs a small ELT pipeline in Trino
54+
* Let you explore the resulting lineage graph in the Marquez UI
55+
56+
The DAG builds a layered set of tables using Trino's built-in `tpch` catalog as the source and the in-memory `memory`
57+
catalog as the sink:
58+
59+
[source]
60+
----
61+
tpch.tiny.customer ─┐
62+
├─> memory.demo.customer_orders
63+
tpch.tiny.orders ──┘
64+
----
65+
66+
Both products report to the *same* Marquez instance (`http://marquez:5000`): Trino emits the per-query dataset lineage,
67+
while Airflow emits the job/task run events (and dataset lineage via the Trino SQL operator).
68+
69+
== How OpenLineage is enabled
70+
71+
This demo deliberately uses only the existing SDP override mechanisms - there are *no operator or CRD changes*. This
72+
mirrors how a future "single toggle" feature would behave, while validating the approach today.
73+
74+
*Trino* loads the bundled `openlineage` event listener through a small properties file. A `ConfigMap` supplies the
75+
file, `podOverrides` mounts it, and `configOverrides` registers it in `config.properties`:
76+
77+
[source,properties]
78+
----
79+
event-listener.name=openlineage
80+
openlineage-event-listener.transport.type=HTTP
81+
openlineage-event-listener.transport.url=http://marquez:5000
82+
openlineage-event-listener.transport.endpoint=/api/v1/lineage
83+
openlineage-event-listener.trino.uri=http://trino-coordinator:8080
84+
----
85+
86+
See the {trino-openlineage-docs}[Trino OpenLineage event listener docs{external-link-icon}^] for details.
87+
88+
*Airflow* only needs the provider pointed at Marquez, which is done with `envOverrides`:
89+
90+
[source]
91+
----
92+
AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "http", "url": "http://marquez:5000", "endpoint": "api/v1/lineage"}'
93+
AIRFLOW__OPENLINEAGE__NAMESPACE="airflow-openlineage-demo"
94+
----
95+
96+
See the {airflow-openlineage-docs}[Airflow OpenLineage provider docs{external-link-icon}^] for details.
97+
98+
== List deployed Stackable services
99+
100+
To list the installed Stackable services run the following command:
101+
102+
[source,console]
103+
----
104+
$ stackablectl stacklet list
105+
106+
┌─────────┬─────────┬───────────────────┬──────────────────────────────────────────┬─────────────────────────────────┐
107+
│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
108+
╞═════════╪═════════╪═══════════════════╪══════════════════════════════════════════╪═════════════════════════════════╡
109+
│ airflow ┆ airflow ┆ openlineage-demo ┆ webserver-http http://172.19.0.3:31483 ┆ Available, Reconciling, Running │
110+
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
111+
│ trino ┆ trino ┆ openlineage-demo ┆ coordinator-https https://172.19.0.5:31087 ┆ Available, Reconciling, Running │
112+
└─────────┴─────────┴───────────────────┴──────────────────────────────────────────┴─────────────────────────────────┘
113+
----
114+
115+
include::partial$instance-hint.adoc[]
116+
117+
[NOTE]
118+
====
119+
Marquez is not an SDP product, so it does not appear in `stackablectl stacklet list`.
120+
====
121+
122+
== Explore the lineage in Marquez
123+
124+
Marquez does not have an external listener. Forward its web UI to your machine:
125+
126+
[source,console]
127+
----
128+
$ kubectl -n openlineage-demo port-forward svc/marquez-web 3000:3000
129+
----
130+
131+
Then open http://localhost:3000 in your browser.
132+
133+
Use the namespace selector at the top to switch between the namespaces that produced the lineage:
134+
135+
* the Trino namespace (named after `openlineage-event-listener.trino.uri`, i.e. `http://trino-coordinator:8080`),
136+
which contains the `tpch.tiny.*` and `memory.demo.*` datasets and the queries connecting them, and
137+
* `airflow-openlineage-demo`, which contains the `openlineage_demo` DAG and its tasks.
138+
139+
Selecting the `memory.demo.customer_orders` dataset and expanding upstream shows the full graph back to
140+
`tpch.tiny.customer` and `tpch.tiny.orders`, including column-level lineage for the individual queries.
141+
142+
[NOTE]
143+
====
144+
Trino and Airflow name datasets independently, so a given table may appear under both the Trino namespace (from the
145+
Trino event listener) and the Airflow namespace (from the SQL operator's extractor). This is expected - each producer
146+
reports lineage from its own perspective.
147+
====
148+
149+
== Summary
150+
151+
This demo showed that both Trino and Airflow can emit OpenLineage data to Marquez with only a small amount of
152+
configuration and no changes to the SDP operators or CRDs. It serves as a validation of the approach ahead of adding
153+
first-class OpenLineage support to the platform.
154+
155+
== Where to go from here
156+
157+
* Trigger the `openlineage_demo` DAG again from the Airflow UI and watch new runs appear in Marquez.
158+
* Run your own queries against Trino (for example via `stackablectl stacklet list` to find the coordinator endpoint)
159+
and see them show up as additional lineage.
160+
* Read more about {openlineage}[OpenLineage{external-link-icon}^] and {marquez}[Marquez{external-link-icon}^].

‎docs/modules/demos/partials/demos.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
* xref:logging.adoc[]
99
* xref:nifi-kafka-druid-earthquake-data.adoc[]
1010
* xref:nifi-kafka-druid-water-level-data.adoc[]
11+
* xref:openlineage-trino-airflow.adoc[]
1112
* xref:opensearch-rag.adoc[]
1213
* xref:signal-processing.adoc[]
1314
* xref:spark-k8s-anomaly-detection-taxi-data.adoc[]

‎stacks/_templates/marquez.yaml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
# Marquez is the OpenLineage reference metadata server + web UI, used here as the consuming
3+
# application for the lineage that Trino and Airflow emit. It is only part of this *demo* - it is
4+
# not an SDP product.
5+
#
6+
# Chart: https://github.com/MarquezProject/marquez/tree/main/chart (Apache-2.0)
7+
# NOTE: confirm the latest chart `version` and the resulting Service names before relying on this.
8+
# With `fullnameOverride: marquez` the API Service is `marquez:5000` and the web UI Service is
9+
# `marquez-web` (targetPort 3000), which is what Trino/Airflow point at.
10+
releaseName: marquez
11+
name: marquez
12+
repo:
13+
name: marquez
14+
url: https://marquezproject.github.io/marquez/
15+
version: 0.51.1 # verify latest
16+
options:
17+
fullnameOverride: marquez
18+
# Let the chart bring its own PostgreSQL so the demo needs no extra database wiring.
19+
postgresql:
20+
enabled: true
21+
auth:
22+
username: marquez
23+
password: marquez
24+
database: marquez
25+
global:
26+
security:
27+
allowInsecureImages: true
28+
image:
29+
repository: bitnamilegacy/postgresql
30+
marquez:
31+
labels:
32+
stackable.tech/vendor: Stackable
33+
stackable.tech/stack: {{ STACK }}
34+
# The following is a Jinja2 template directive, not a regular comment.
35+
# The `#` prefix is required to prevent YAML parsing errors, but Jinja2 still processes it.
36+
# {% if DEMO is defined %}
37+
stackable.tech/demo: {{ DEMO }}
38+
# {% endif %}
39+
web:
40+
enabled: true
41+
service:
42+
type: NodePort

0 commit comments

Comments
 (0)