|
| 1 | += OpenLineage |
| 2 | +:description: Emit OpenLineage data lineage events from Trino queries to an OpenLineage-compatible backend such as Marquez. |
| 3 | +:keywords: OpenLineage, data lineage, event listener, Marquez, Trino |
| 4 | + |
| 5 | +https://openlineage.io/[OpenLineage] is an open standard for data lineage collection. |
| 6 | +Trino ships an https://trino.io/docs/current/admin/event-listeners-openlineage.html[OpenLineage event listener] as a core plugin, which emits a lineage event for every executed query. |
| 7 | +The Trino operator can configure this listener automatically, so lineage from all queries run on the cluster is sent to an OpenLineage-compatible backend such as https://marquezproject.github.io/marquez/[Marquez], without any manual configuration. |
| 8 | + |
| 9 | +When configured, the operator: |
| 10 | + |
| 11 | +* writes an `event-listener.properties` file on the *coordinator* selecting the OpenLineage plugin and its HTTP transport, |
| 12 | +* points the transport at the backend endpoint (`<scheme>://<host>:<port>`), |
| 13 | +* records this cluster's `trino.uri` and a lineage namespace, and |
| 14 | +* (optionally) imports a backend CA certificate into the coordinator trust store and injects a bearer token for authenticated backends. |
| 15 | +
|
| 16 | +The event listener runs on the coordinator only; workers are not affected. |
| 17 | + |
| 18 | +== Enabling OpenLineage |
| 19 | + |
| 20 | +OpenLineage is configured through `clusterConfig.openLineage.connection`, which either inlines an OpenLineage connection or references a standalone `OpenLineageConnection` resource by name: |
| 21 | + |
| 22 | +[source,yaml] |
| 23 | +---- |
| 24 | +apiVersion: trino.stackable.tech/v1alpha1 |
| 25 | +kind: TrinoCluster |
| 26 | +metadata: |
| 27 | + name: trino |
| 28 | +spec: |
| 29 | + clusterConfig: |
| 30 | + openLineage: # <1> |
| 31 | + connection: # <2> |
| 32 | + inline: |
| 33 | + host: marquez |
| 34 | + port: 5000 |
| 35 | + # tls: |
| 36 | + # verification: |
| 37 | + # server: |
| 38 | + # caCert: |
| 39 | + # secretClass: marquez-ca |
| 40 | + # namespace: trino-lineage # <3> |
| 41 | + ... |
| 42 | +---- |
| 43 | +<1> Adding the `openLineage` block enables OpenLineage event emission. Omit it entirely to leave OpenLineage off (the default). |
| 44 | +<2> The OpenLineage backend connection. Use `inline` (shown here) to define the connection directly, or `reference: <name>` to point at an `OpenLineageConnection` resource in the same namespace. Required. |
| 45 | +<3> Optional. The OpenLineage namespace events are reported under. Defaults to the cluster's Kubernetes namespace. |
| 46 | + |
| 47 | +NOTE: The shared connection type also accepts an `appName` field. It is ignored for Trino, which is a long-running service rather than a single job and derives an OpenLineage job name per query. |
| 48 | + |
| 49 | +The `OpenLineageConnection` resource (`openlineage.stackable.tech`) is a reusable definition of a connection to an OpenLineage backend, provided by the Stackable operator library. |
| 50 | +Reference it to share one backend definition across multiple clusters: |
| 51 | + |
| 52 | +[source,yaml] |
| 53 | +---- |
| 54 | +apiVersion: openlineage.stackable.tech/v1alpha1 |
| 55 | +kind: OpenLineageConnection |
| 56 | +metadata: |
| 57 | + name: marquez |
| 58 | +spec: |
| 59 | + host: marquez |
| 60 | + port: 5000 |
| 61 | +---- |
| 62 | + |
| 63 | +The transport scheme is `https` when `tls.verification.server` is configured, otherwise `http`. |
| 64 | +When the connection verifies the server against a `secretClass` CA (as in the commented `tls` block above), the operator mounts that SecretClass certificate into the coordinator and adds it to the trust store, so the OpenLineage listener trusts the backend's certificate. |
| 65 | + |
| 66 | +[#authentication] |
| 67 | +== Authentication |
| 68 | + |
| 69 | +Backends that require authentication (for example DataHub, or a Marquez instance behind a gateway) expect a bearer token on each request. |
| 70 | +Configure this by referencing an xref:concepts:authentication.adoc[`AuthenticationClass`] from the connection via `authenticationClassRef`: |
| 71 | + |
| 72 | +[source,yaml] |
| 73 | +---- |
| 74 | +apiVersion: trino.stackable.tech/v1alpha1 |
| 75 | +kind: TrinoCluster |
| 76 | +metadata: |
| 77 | + name: trino |
| 78 | +spec: |
| 79 | + clusterConfig: |
| 80 | + openLineage: |
| 81 | + connection: |
| 82 | + inline: |
| 83 | + host: marquez |
| 84 | + port: 5000 |
| 85 | + authenticationClassRef: openlineage-auth # <1> |
| 86 | + ... |
| 87 | +--- |
| 88 | +apiVersion: authentication.stackable.tech/v1alpha1 |
| 89 | +kind: AuthenticationClass |
| 90 | +metadata: |
| 91 | + name: openlineage-auth |
| 92 | +spec: |
| 93 | + provider: |
| 94 | + static: # <2> |
| 95 | + userCredentialsSecret: |
| 96 | + name: openlineage-token # <3> |
| 97 | +--- |
| 98 | +apiVersion: v1 |
| 99 | +kind: Secret |
| 100 | +metadata: |
| 101 | + name: openlineage-token |
| 102 | + namespace: default # <4> |
| 103 | +stringData: |
| 104 | + apiKey: "s3cr3t-bearer-token" # <5> |
| 105 | +---- |
| 106 | +<1> References an `AuthenticationClass` by name (cluster-scoped). If omitted, no authentication is used. |
| 107 | +<2> Only the `static` provider is supported for OpenLineage. Other providers (LDAP, OIDC, TLS) are rejected with an error. |
| 108 | +<3> Name of the Secret holding the token. Must exist in the same namespace as the `TrinoCluster`. |
| 109 | +<4> The Secret must be in the cluster's namespace so it can be projected into the coordinator. |
| 110 | +<5> The token must be stored under the fixed key `apiKey`. The token is mounted as a file into the coordinator and resolved into the event listener configuration at startup, so it never appears in the ConfigMap or the pod's environment. |
| 111 | + |
| 112 | +The operator translates this into the OpenLineage HTTP transport's `api-key` (bearer) authentication. |
| 113 | +Combine it with the `tls` block above when the backend also needs TLS server verification. |
| 114 | + |
| 115 | +== Overriding the generated configuration |
| 116 | + |
| 117 | +The generated `event-listener.properties` can be extended or overridden per role group through xref:usage-guide/overrides.adoc[config overrides], using the `event-listener.properties` file name. |
| 118 | +Overrides are merged last and therefore take precedence over the operator-generated values: |
| 119 | + |
| 120 | +[source,yaml] |
| 121 | +---- |
| 122 | +spec: |
| 123 | + coordinators: |
| 124 | + configOverrides: |
| 125 | + event-listener.properties: |
| 126 | + openlineage-event-listener.trino.uri: "https://trino.example.com:8443" |
| 127 | +---- |
| 128 | + |
| 129 | +Event listeners run on the coordinator, so `event-listener.properties` overrides only take effect on the coordinator role. |
0 commit comments