Skip to content

Commit 3e4e9fc

Browse files
m-aciekclaude
andcommitted
fix: probe CollectionConverters with a call to detect Scala version correctly
On Scala 2.12, jvm.scala.jdk.javaapi.CollectionConverters resolves to a JavaPackage placeholder rather than raising an exception, so the previous try/except always selected the jdk path and broke Spark 3.x. Verify usability by actually calling asScala() as a one-time probe. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 7275937 commit 3e4e9fc

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

pydeequ/scala_utils.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,12 @@ def _get_converters(jvm):
8181
key = id(jvm)
8282
if key not in _jvm_converters_cache:
8383
try:
84-
_jvm_converters_cache[key] = ("jdk", jvm.scala.jdk.javaapi.CollectionConverters)
84+
converters = jvm.scala.jdk.javaapi.CollectionConverters
85+
# On Scala 2.12, the path resolves to a JavaPackage placeholder (no class
86+
# exists), so attribute access succeeds but any method call raises TypeError.
87+
# Probe with an actual call to confirm the class is genuinely usable.
88+
converters.asScala(jvm.java.util.ArrayList())
89+
_jvm_converters_cache[key] = ("jdk", converters)
8590
except Exception:
8691
_jvm_converters_cache[key] = ("legacy", jvm.scala.collection.JavaConverters)
8792
return _jvm_converters_cache[key]

0 commit comments

Comments
 (0)