Add Spark 4.0 support via deequ:2.0.14-spark-4.0#11
Conversation
…rail false positive
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| def empty_scala_seq(jvm): | ||
| """ | ||
| Returns an empty Scala immutable List (Nil), usable as Seq[_]. | ||
| Uses JavaConverters.toList() to produce an immutable.List rather than |
There was a problem hiding this comment.
JavaConverters.iterableAsScalaIterableConverter may not exist in Scala 2.12 (Spark 3.x) as a static method accessible via Py4J. In Scala 2.12, JavaConverters exposes implicit conversions, not static converter methods like iterableAsScalaIterableConverter(...).asScala(). This pattern works in Scala 2.13+ but may fail at runtime on Spark 3.x. You should test this on all supported Spark versions (3.1, 3.2, 3.3, 3.5) or add a version-conditional fallback to JavaConversions for Spark 3.x.
|
|
||
| def scala_map_to_dict(jvm, scala_map): | ||
| return dict(jvm.scala.collection.JavaConversions.mapAsJavaMap(scala_map)) | ||
| return dict(jvm.scala.collection.JavaConverters.mapAsJavaMapConverter(scala_map).asJava()) |
There was a problem hiding this comment.
Same concern as to_scala_seq: JavaConverters.mapAsJavaMapConverter(scala_map).asJava() relies on Scala 2.13-style static method access. On Scala 2.12 (Spark 3.x), mapAsJavaMapConverter is an implicit class, not a directly callable static method via Py4J. This may break on Spark 3.x at runtime.
|
|
||
| def scala_map_to_java_map(jvm, scala_map): | ||
| return jvm.scala.collection.JavaConversions.mapAsJavaMap(scala_map) | ||
| return jvm.scala.collection.JavaConverters.mapAsJavaMapConverter(scala_map).asJava() |
There was a problem hiding this comment.
Same Scala 2.12 vs 2.13 compatibility issue as scala_map_to_dict above.
| """ | ||
| self._run_result = run | ||
| profile_map = self._jvm.scala.collection.JavaConversions.mapAsJavaMap(run.profiles()) # TODO from ScalaUtils | ||
| profile_map = self._jvm.scala.collection.JavaConverters.mapAsJavaMapConverter(run.profiles()).asJava() # TODO from ScalaUtils |
There was a problem hiding this comment.
Same JavaConverters.mapAsJavaMapConverter(...).asJava() Scala 2.12 compatibility concern. This was changed from JavaConversions but the existing helper scala_map_to_java_map in scala_utils.py does the same thing — use that helper instead of duplicating the conversion logic inline.
|
|
||
|
|
||
| SPARK_TO_DEEQU_COORD_MAPPING = { | ||
| "4.0": "com.amazon.deequ:deequ:2.0.14-spark-4.0", |
There was a problem hiding this comment.
The _get_deequ_maven_config function at line 35 slices spark_version[:3] which for "4.0" yields "4.0" (correct), but this truncation approach is fragile — for any version like "4.01" it would yield "4.0" but _extract_major_minor_versions already returns "4.0". The [:3] slice is redundant with the regex extraction and could mask bugs.
| env: | ||
| SPARK_VERSION: ${{matrix.PYSPARK_VERSION}} | ||
| PANDAS_VERSION: ${{matrix.PANDAS_VERSION}} | ||
| run: | |
There was a problem hiding this comment.
poetry run pip install pyspark==$SPARK_VERSION bypasses Poetry's dependency resolver and can cause inconsistencies between the lock file and the actual environment. The previous approach poetry add pyspark==$SPARK_VERSION was more correct. If poetry add doesn't work for Spark 4.0, document why this workaround is needed.
| numpy = ">=1.14.1" | ||
| pandas = ">=0.23.0" | ||
| pyspark = { version = ">=2.4.7,<3.4.0", optional = true } | ||
| pyspark = [ |
There was a problem hiding this comment.
The two pyspark version constraints split by Python version create a gap: Python 3.8 users are limited to <4.0.0 but Python 3.9+ users get <5.0.0. However, the python = ">=3.8,<4" top-level constraint combined with the CI matrix only testing Python 3.8 for Spark 3.x and 3.9 for Spark 4.0 means Python 3.9 users on Spark 3.x could accidentally resolve pyspark 4.x. Consider tightening the constraints or documenting the expected combinations.
1 participant
Issue #, if available:
Description of changes:
Adds Spark 4.0 compatibility by migrating from deprecated JavaConversions to JavaConverters, adding empty_scala_seq helper, and updating configs/CI matrix.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.