feat: harmonise OpenLineage jar name via a stable image symlink

Author: sweb

CHANGELOG.md

@@ -8,7 +8,7 @@ All notable changes to this project will be documented in this file.
 
 - BREAKING: Add required CLI argument and env var to set the image repository used to construct final product image names: `IMAGE_REPOSITORY` (`--image-repository`), eg. `oci.example.org/my/namespace` ([#684]).
 - Support for Spark `4.1.2` ([#708])
-- Add `spec.openLineage` to enable OpenLineage lineage emission: injects the OpenLineage Spark listener, HTTP transport (endpoint resolved from a discovery ConfigMap) and a stable job name. The Scala build of the listener jar and the `--add-opens java.base/java.security` JVM flag are selected by the Spark version: the `--add-opens` flag is emitted only on Spark 4.x (it is unnecessary on the Spark 3.5.x images and must not be set there) ([#XXXX]).
+- Add `spec.openLineage` to enable OpenLineage lineage emission: injects the OpenLineage Spark listener, HTTP transport (endpoint resolved from a discovery ConfigMap) and a stable job name. The listener jar is referenced through a stable, version- and Scala-independent symlink maintained by the Spark image, so the operator tracks neither. The `--add-opens java.base/java.security` JVM flag is emitted only on Spark 4.x (it is unnecessary on the Spark 3.5.x images and must not be set there) ([#XXXX]).
 
 ### Fixed
 

docs/modules/spark-k8s/pages/usage-guide/openlineage.adoc

@@ -6,7 +6,7 @@ The Spark operator can automatically emit lineage events for a `SparkApplication
 When enabled, the operator injects everything required to make the https://openlineage.io/docs/integrations/spark/[OpenLineage Spark listener] work:
 
 * the OpenLineage Spark listener (appended to `spark.extraListeners`, so any listener you configure yourself is preserved),
-* the OpenLineage jar baked into the Spark image, referenced via `spark.jars` so it shares the same classloader as connectors pulled in through xref:usage-guide/job-dependencies.adoc[`deps.packages`] (for example Apache Iceberg). The operator selects the jar built for the image's Scala version automatically (Scala 2.13 for Spark 4.x, Scala 2.12 for Spark 3.5.x),
+* the OpenLineage jar baked into the Spark image, referenced via `spark.jars` so it shares the same classloader as connectors pulled in through xref:usage-guide/job-dependencies.adoc[`deps.packages`] (for example Apache Iceberg). The operator references a stable symlink maintained by the image, which points at the correct build for that image's Spark version (Scala 2.13 for Spark 4.x, Scala 2.12 for Spark 3.5.x), so neither the jar version nor its Scala suffix has to be tracked in the operator,
 * an HTTP transport pointing at the backend endpoint,
 * a stable lineage namespace and job name (see <<job-name>>), and
 * on Spark 4.x only, the `--add-opens java.base/java.security=ALL-UNNAMED` JVM flag the listener requires on the driver and executors. This flag is not added on the Spark 3.5.x images, where it is unnecessary (and, on their JVM, potentially harmful).

rust/operator-binary/src/crd/constants.rs

@@ -108,25 +108,17 @@ pub const DEFAULT_SUBMIT_JOB_RETRY_ON_FAILURE_COUNT: u16 = 0;
 /// The OpenLineage Spark listener, appended to `spark.extraListeners` when OpenLineage is enabled.
 pub const OPENLINEAGE_LISTENER_CLASS: &str = "io.openlineage.spark.agent.OpenLineageSparkListener";
 
-/// Version of the OpenLineage Spark listener jar baked into the Spark images.
-///
-/// IMPORTANT: MUST stay in sync with the `openlineage-spark-version` build-argument in
-/// `docker-images/spark-k8s/boil-config.toml` (the image is what places the jar on disk).
-pub const OPENLINEAGE_SPARK_JAR_VERSION: &str = "1.51.0";
-
 /// `local://` URI of the OpenLineage jar baked into the Spark image, referenced via `spark.jars` so
 /// it shares the (child) classloader with `--packages` connectors. Delivering it this way — rather
 /// than on `extraClassPath` (system classloader) — is what lets OpenLineage's connector probes
 /// succeed; see the classpath discussion in the OpenLineage usage guide.
 ///
-/// The jar's Scala suffix depends on the Spark image: Stackable ships Spark 4.x as Scala 2.13 and
-/// Spark 3.5.x as Scala 2.12, so the operator selects the matching jar by Spark major version.
-/// `scala_binary_version` is e.g. `"2.13"` or `"2.12"`.
-pub fn openlineage_jar_local_uri(scala_binary_version: &str) -> String {
-    format!(
-        "local:///stackable/spark/openlineage/openlineage-spark_{scala_binary_version}-{OPENLINEAGE_SPARK_JAR_VERSION}.jar"
-    )
-}
+/// This is a stable, version- and Scala-independent symlink the image maintains (mirroring the
+/// `jmx_prometheus_javaagent.jar` pattern): the image bakes the correct `openlineage-spark_<scala>-<version>.jar`
+/// for its Spark build and symlinks it here. Referencing the symlink decouples the operator from the
+/// jar's version and Scala suffix, so bumping either only touches `docker-images/spark-k8s`.
+pub const OPENLINEAGE_JAR_LOCAL_URI: &str =
+    "local:///stackable/spark/openlineage/openlineage-spark.jar";
 
 /// The key the OpenLineage discovery ConfigMap must hold, carrying the backend base URL. Mirrors the
 /// `ADDRESS` contract of the existing `vectorAggregatorConfigMapName` discovery ConfigMap.

rust/operator-binary/src/crd/mod.rs

@@ -892,20 +892,9 @@ impl v1alpha1::SparkApplication {
                 "spark.extraListeners",
                 OPENLINEAGE_LISTENER_CLASS,
             );
-            // Select the jar built for the image's Scala version: Stackable ships Spark 4.x as
-            // Scala 2.13 and Spark 3.5.x as Scala 2.12. Referencing the wrong build would leave a
-            // non-existent path in `spark.jars`, which Spark ignores silently (no listener, no
-            // events), so this must track the actual image.
-            let scala_binary_version = if spark_major_version(product_version)? >= 4 {
-                "2.13"
-            } else {
-                "2.12"
-            };
-            append_conf_csv(
-                &mut submit_conf,
-                "spark.jars",
-                &openlineage_jar_local_uri(scala_binary_version),
-            );
+            // Reference the stable symlink the image maintains; it points at the correct
+            // Scala/version build for this image, so the operator needs to know neither.
+            append_conf_csv(&mut submit_conf, "spark.jars", OPENLINEAGE_JAR_LOCAL_URI);
         }
 
         // ...before being added to the command collection
@@ -1836,10 +1825,9 @@ spec:
         assert!(command.contains(&format!(
             r#"--conf "spark.extraListeners={OPENLINEAGE_LISTENER_CLASS}""#
         )));
-        // Spark 4.x is Scala 2.13, so the 2.13 build of the jar is referenced.
+        // The jar is referenced via the stable, Scala/version-independent image symlink.
         assert!(command.contains(&format!(
-            r#"--conf "spark.jars={}""#,
-            openlineage_jar_local_uri("2.13")
+            r#"--conf "spark.jars={OPENLINEAGE_JAR_LOCAL_URI}""#
         )));
         // --add-opens reaches both driver and executor on Spark 4.x.
         assert_eq!(
@@ -1850,18 +1838,17 @@ spec:
     }
 
     #[test]
-    fn test_openlineage_selects_scala_2_12_jar_and_no_add_opens_on_spark_3() {
-        // On the Scala 2.12 / JDK 17 Spark 3.5.x images, the operator must reference the 2.12 build
-        // of the jar (the 2.13 path would not exist in the image) and must NOT emit `--add-opens`.
+    fn test_openlineage_stable_jar_uri_and_no_add_opens_on_spark_3() {
+        // On the JDK 17 Spark 3.5.x images the operator references the same stable jar symlink (the
+        // image points it at the Scala 2.12 build) and must NOT emit `--add-opens`.
         let command = build_command_with_openlineage(
             OPENLINEAGE_ENABLED_APP,
             "3.5.8",
             Some("http://marquez:5000"),
         );
 
         assert!(command.contains(&format!(
-            r#"--conf "spark.jars={}""#,
-            openlineage_jar_local_uri("2.12")
+            r#"--conf "spark.jars={OPENLINEAGE_JAR_LOCAL_URI}""#
         )));
         assert!(
             !command.contains(OPENLINEAGE_ADD_OPENS),