refactor!: replace OL config map with host:port fields

razvan · razvan · commit 6cdec1ae0dd9 · 2026-07-16T14:10:15.000+02:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,7 +8,7 @@ All notable changes to this project will be documented in this file.
 
 - BREAKING: Add required CLI argument and env var to set the image repository used to construct final product image names: `IMAGE_REPOSITORY` (`--image-repository`), eg. `oci.example.org/my/namespace` ([#684]).
 - Support for Spark `4.1.2` ([#708])
-- Add `spec.openLineage` to enable OpenLineage lineage emission: injects the OpenLineage Spark listener, HTTP transport (endpoint resolved from a discovery ConfigMap), a stable job name, and the required `--add-opens` JVM flag ([#XXXX]).
+- Add `spec.openLineage` to enable OpenLineage lineage emission: injects the OpenLineage Spark listener, HTTP transport (endpoint built from the `host` and `port` fields), a stable job name, and the required `--add-opens` JVM flag ([#XXXX]).
 
 ### Fixed
 
diff --git a/docs/modules/spark-k8s/pages/usage-guide/openlineage.adoc b/docs/modules/spark-k8s/pages/usage-guide/openlineage.adoc
@@ -13,8 +13,8 @@ When configured, the operator injects everything required to make the https://op
 
 == Enabling OpenLineage
 
-The backend endpoint is not configured on the `SparkApplication` directly.
-Instead, mirroring the xref:usage-guide/logging.adoc[vector aggregator discovery], the operator reads it from a discovery ConfigMap referenced by `configMapName`:
+The backend endpoint is configured directly on the `SparkApplication` through the `host` and `port` fields.
+The operator points the HTTP transport at `http://<host>:<port>`:
 
 [source,yaml]
 ----
@@ -24,34 +24,22 @@ metadata:
   name: orders-by-nation
 spec:
   openLineage: # <1>
-    configMapName: marquez-discovery # <2>
-    # namespace: orders-lineage # <3>
-    # appName: orders-by-nation # <4>
+    host: marquez # <2>
+    port: 5000 # <3>
+    # namespace: orders-lineage # <4>
+    # appName: orders-by-nation # <5>
   ...
 ----
 <1> Adding the `openLineage` block enables OpenLineage event emission. Omit it entirely to leave OpenLineage off (the default), in which case the operator emits nothing OpenLineage-related and behaviour is unchanged.
-<2> Name of a discovery ConfigMap holding the backend endpoint (see below).
-    Must be in the same namespace as the `SparkApplication`.
-<3> Optional. The OpenLineage namespace events are reported under.
+<2> Host of the OpenLineage backend, for example the Marquez API service. Required.
+<3> Port of the OpenLineage backend. Required.
+<4> Optional. The OpenLineage namespace events are reported under.
     Defaults to the application's Kubernetes namespace.
-<4> Optional. The stable OpenLineage job name (see <<job-name>>).
+<5> Optional. The stable OpenLineage job name (see <<job-name>>).
 
-The discovery ConfigMap holds the backend base URL under the `ADDRESS` key:
+The transport protocol is always `http`; explicit `https` support will be added later.
 
-.Example discovery ConfigMap
-[source,yaml]
-----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: marquez-discovery
-data:
-  ADDRESS: http://marquez:5000 # <1>
-----
-<1> Base URL of the OpenLineage backend, for example the Marquez API service.
-
-Alternatively, you can set the endpoint directly through `sparkConf` (`spark.openlineage.transport.url`).
-If the `openLineage` block is present and no endpoint can be resolved from either the discovery ConfigMap or `sparkConf`, reconciliation fails with a clear error rather than emitting a config that silently drops events.
+You can still override any of the injected values — including the full `spark.openlineage.transport.url` — through `sparkConf`.
 
 [#job-name]
 == Job name
diff --git a/extra/crds.yaml b/extra/crds.yaml
@@ -1972,22 +1972,28 @@ spec:
                       `spark.app.name`, falling back to `metadata.name` (with a warning event).
                     nullable: true
                     type: string
-                  configMapName:
+                  host:
                     description: |-
-                      Name of the OpenLineage backend [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery).
-                      It must contain the key `ADDRESS` with the base URL of the OpenLineage backend
-                      (e.g. `http://marquez:5000`). Mirrors the `vectorAggregatorConfigMapName` field on this CRD.
-                    maxLength: 253
-                    minLength: 1
-                    nullable: true
-                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                      Host of the OpenLineage backend the HTTP transport points at (e.g. `marquez`).
+                      Combined with `port` into the transport URL `http://<host>:<port>`.
                     type: string
                   namespace:
                     description: |-
                       The OpenLineage namespace lineage is reported under.
                       Defaults to the application's Kubernetes namespace (`metadata.namespace`).
                     nullable: true
                     type: string
+                  port:
+                    description: |-
+                      Port of the OpenLineage backend (e.g. `5000`).
+                      Combined with `host` into the transport URL `http://<host>:<port>`.
+                    format: uint16
+                    maximum: 65535.0
+                    minimum: 0.0
+                    type: integer
+                required:
+                - host
+                - port
                 type: object
               s3connection:
                 description: |-
@@ -6610,22 +6616,28 @@ spec:
                       `spark.app.name`, falling back to `metadata.name` (with a warning event).
                     nullable: true
                     type: string
-                  configMapName:
+                  host:
                     description: |-
-                      Name of the OpenLineage backend [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery).
-                      It must contain the key `ADDRESS` with the base URL of the OpenLineage backend
-                      (e.g. `http://marquez:5000`). Mirrors the `vectorAggregatorConfigMapName` field on this CRD.
-                    maxLength: 253
-                    minLength: 1
-                    nullable: true
-                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                      Host of the OpenLineage backend the HTTP transport points at (e.g. `marquez`).
+                      Combined with `port` into the transport URL `http://<host>:<port>`.
                     type: string
                   namespace:
                     description: |-
                       The OpenLineage namespace lineage is reported under.
                       Defaults to the application's Kubernetes namespace (`metadata.namespace`).
                     nullable: true
                     type: string
+                  port:
+                    description: |-
+                      Port of the OpenLineage backend (e.g. `5000`).
+                      Combined with `host` into the transport URL `http://<host>:<port>`.
+                    format: uint16
+                    maximum: 65535.0
+                    minimum: 0.0
+                    type: integer
+                required:
+                - host
+                - port
                 type: object
               s3connection:
                 description: |-
diff --git a/rust/operator-binary/src/config/jvm.rs b/rust/operator-binary/src/config/jvm.rs
@@ -152,7 +152,9 @@ mod tests {
               mainApplicationFile: test.py
               sparkImage:
                 productVersion: 1.2.3
-              openLineage: {}
+              openLineage:
+                host: marquez
+                port: 5000
         "#;
 
         let deserializer = serde_yaml::Deserializer::from_str(input);
diff --git a/rust/operator-binary/src/crd/constants.rs b/rust/operator-binary/src/crd/constants.rs
@@ -118,10 +118,6 @@ pub const OPENLINEAGE_LISTENER_CLASS: &str = "io.openlineage.spark.agent.OpenLin
 pub const OPENLINEAGE_JAR_LOCAL_URI: &str =
     "local:///stackable/spark/openlineage/openlineage-spark_2.13-1.51.0.jar";
 
-/// The key the OpenLineage discovery ConfigMap must hold, carrying the backend base URL. Mirrors the
-/// `ADDRESS` contract of the existing `vectorAggregatorConfigMapName` discovery ConfigMap.
-pub const OPENLINEAGE_CONFIG_MAP_ADDRESS_KEY: &str = "ADDRESS";
-
 /// Java module-system flag OpenLineage requires on Spark 4.x: without it the driver throws a
 /// non-fatal `InaccessibleObjectException` and silently degrades extension-interface lineage.
 /// Appended to both driver and executor `extraJavaOptions`.
diff --git a/rust/operator-binary/src/crd/mod.rs b/rust/operator-binary/src/crd/mod.rs
@@ -9,7 +9,7 @@ use constants::*;
 use history::LogFileDirectorySpec;
 use logdir::ResolvedLogDir;
 use serde::{Deserialize, Serialize};
-use snafu::{OptionExt, ResultExt, Snafu, ensure};
+use snafu::{OptionExt, ResultExt, Snafu};
 use stackable_operator::{
     builder::pod::volume::{
         SecretFormat, SecretOperatorVolumeSourceBuilder, SecretOperatorVolumeSourceBuilderError,
@@ -115,13 +115,6 @@ pub enum Error {
 
     #[snafu(display("failed to configure log directory"))]
     ConfigureLogDir { source: logdir::Error },
-
-    #[snafu(display(
-        "OpenLineage is enabled but no backend endpoint could be resolved: set \
-         `spec.openLineage.configMapName` (a discovery ConfigMap holding the `ADDRESS` key) or \
-         `spark.openlineage.transport.url` in `sparkConf`"
-    ))]
-    MissingOpenLineageEndpoint,
 }
 
 pub type SparkApplicationJobRoleType =
@@ -576,7 +569,6 @@ impl v1alpha1::SparkApplication {
         s3conn: &Option<s3::v1alpha1::ConnectionSpec>,
         log_dir: &Option<ResolvedLogDir>,
         spark_image: &str,
-        openlineage_endpoint: Option<&str>,
     ) -> Result<Vec<String>, Error> {
         // mandatory properties
         let mode = &self.spec.mode;
@@ -754,26 +746,14 @@ impl v1alpha1::SparkApplication {
         // (`spark.jars`, `spark.extraListeners`) are handled AFTER the merge so a user value is
         // combined with — not clobbered by — ours. See the OpenLineage usage guide.
         if let Some(open_lineage) = &self.spec.open_lineage {
-            // Fail fast rather than emit a transport that silently drops every event.
-            let has_url_override = self
-                .spec
-                .spark_conf
-                .contains_key("spark.openlineage.transport.url");
-            ensure!(
-                openlineage_endpoint.is_some() || has_url_override,
-                MissingOpenLineageEndpointSnafu
-            );
-
             submit_conf.insert(
                 "spark.openlineage.transport.type".to_string(),
                 "http".to_string(),
             );
-            if let Some(endpoint) = openlineage_endpoint {
-                submit_conf.insert(
-                    "spark.openlineage.transport.url".to_string(),
-                    endpoint.to_string(),
-                );
-            }
+            submit_conf.insert(
+                "spark.openlineage.transport.url".to_string(),
+                open_lineage.transport_url(),
+            );
 
             let namespace = match &open_lineage.namespace {
                 Some(namespace) => namespace.clone(),
@@ -1640,13 +1620,11 @@ spec:
         assert_eq!(got, expected);
     }
 
-    /// Builds a `SparkApplication` from YAML and returns the assembled spark-submit command string,
-    /// with `openlineage_endpoint` standing in for the value the controller resolves from the
-    /// discovery ConfigMap.
-    fn build_command_with_openlineage(yaml: &str, openlineage_endpoint: Option<&str>) -> String {
+    /// Builds a `SparkApplication` from YAML and returns the assembled spark-submit command string.
+    fn build_command_with_openlineage(yaml: &str) -> String {
         let spark_application = serde_yaml::from_str::<v1alpha1::SparkApplication>(yaml).unwrap();
         spark_application
-            .build_command(&None, &None, "test-image", openlineage_endpoint)
+            .build_command(&None, &None, "test-image")
             .unwrap()
             .join(" ")
     }
@@ -1663,13 +1641,14 @@ spec:
           mainApplicationFile: test.py
           sparkImage:
             productVersion: 1.2.3
-          openLineage: {}
+          openLineage:
+            host: marquez
+            port: 5000
     "#};
 
     #[test]
     fn test_openlineage_injects_conf_when_enabled() {
-        let command =
-            build_command_with_openlineage(OPENLINEAGE_ENABLED_APP, Some("http://marquez:5000"));
+        let command = build_command_with_openlineage(OPENLINEAGE_ENABLED_APP);
 
         assert!(command.contains(r#"--conf "spark.openlineage.transport.type=http""#));
         assert!(
@@ -1709,8 +1688,7 @@ spec:
               sparkImage:
                 productVersion: 1.2.3
         "#};
-        // Endpoint is supplied to prove it is ignored, not that it is simply missing.
-        let command = build_command_with_openlineage(yaml, Some("http://marquez:5000"));
+        let command = build_command_with_openlineage(yaml);
 
         assert!(!command.contains("openlineage"));
         assert!(!command.contains("OpenLineageSparkListener"));
@@ -1732,11 +1710,13 @@ spec:
               mainApplicationFile: test.py
               sparkImage:
                 productVersion: 1.2.3
-              openLineage: {}
+              openLineage:
+                host: marquez
+                port: 5000
               sparkConf:
                 spark.extraListeners: com.example.CustomListener
         "#};
-        let command = build_command_with_openlineage(yaml, Some("http://marquez:5000"));
+        let command = build_command_with_openlineage(yaml);
 
         assert!(command.contains(&format!(
             r#"--conf "spark.extraListeners=com.example.CustomListener,{OPENLINEAGE_LISTENER_CLASS}""#
@@ -1757,52 +1737,20 @@ spec:
               mainApplicationFile: test.py
               sparkImage:
                 productVersion: 1.2.3
-              openLineage: {}
+              openLineage:
+                host: marquez
+                port: 5000
               sparkConf:
                 spark.openlineage.transport.url: http://custom:1234
                 spark.openlineage.namespace: custom-ns
         "#};
-        let command = build_command_with_openlineage(yaml, Some("http://marquez:5000"));
+        let command = build_command_with_openlineage(yaml);
 
         assert!(command.contains(r#"--conf "spark.openlineage.transport.url=http://custom:1234""#));
         assert!(!command.contains("http://marquez:5000"));
         assert!(command.contains(r#"--conf "spark.openlineage.namespace=custom-ns""#));
     }
 
-    #[test]
-    fn test_openlineage_fails_fast_without_endpoint() {
-        // Enabled, but neither a resolved endpoint nor a sparkConf transport.url override.
-        let spark_application =
-            serde_yaml::from_str::<v1alpha1::SparkApplication>(OPENLINEAGE_ENABLED_APP).unwrap();
-        let result = spark_application.build_command(&None, &None, "test-image", None);
-
-        assert!(matches!(result, Err(Error::MissingOpenLineageEndpoint)));
-    }
-
-    #[test]
-    fn test_openlineage_endpoint_from_spark_conf_satisfies_fail_fast() {
-        // No resolved endpoint, but a sparkConf override → must NOT fail.
-        let yaml = indoc! {r#"
-            ---
-            apiVersion: spark.stackable.tech/v1alpha1
-            kind: SparkApplication
-            metadata:
-              name: spark-examples
-              namespace: default
-            spec:
-              mode: cluster
-              mainApplicationFile: test.py
-              sparkImage:
-                productVersion: 1.2.3
-              openLineage: {}
-              sparkConf:
-                spark.openlineage.transport.url: http://custom:1234
-        "#};
-        let command = build_command_with_openlineage(yaml, None);
-
-        assert!(command.contains(r#"--conf "spark.openlineage.transport.url=http://custom:1234""#));
-    }
-
     #[rstest]
     #[case::explicit_app_name(
         indoc! {r#"
@@ -1818,6 +1766,8 @@ spec:
               sparkImage:
                 productVersion: 1.2.3
               openLineage:
+                host: marquez
+                port: 5000
                 appName: explicit-name
               sparkConf:
                 spark.app.name: spark-conf-name
@@ -1838,7 +1788,9 @@ spec:
               mainApplicationFile: test.py
               sparkImage:
                 productVersion: 1.2.3
-              openLineage: {}
+              openLineage:
+                host: marquez
+                port: 5000
               sparkConf:
                 spark.app.name: spark-conf-name
         "#},
@@ -1858,7 +1810,9 @@ spec:
               mainApplicationFile: test.py
               sparkImage:
                 productVersion: 1.2.3
-              openLineage: {}
+              openLineage:
+                host: marquez
+                port: 5000
         "#},
         "metadata-name",
         true
diff --git a/rust/operator-binary/src/openlineage.rs b/rust/operator-binary/src/openlineage.rs
diff --git a/rust/operator-binary/src/spark_k8s_controller.rs b/rust/operator-binary/src/spark_k8s_controller.rs
