TypeError: 'JavaPackage' object is not callable when running pydeequ

[sudsali]

I installed pydeequ with pip and I'm getting this error when trying to run VerificationSuite:

from pyspark.sql import SparkSession, Row                                                                                                                                        
import pydeequ                                                                                                                                                                   
from pydeequ.checks import Check, CheckLevel                                                                                                                                     
from pydeequ.verification import VerificationSuite, VerificationResult                                                                                                           
                                                                                                                                                                                 
spark = SparkSession.builder.master("local").appName("test").getOrCreate()                                                                                                       
                                                                                                                                                                                 
df = spark.createDataFrame([Row(a="foo", b=1), Row(a="bar", b=2)])                                                                                                               
                                                                                                                                                                                 
check = Check(spark, CheckLevel.Warning, "test")                                                                                                                                 
result = VerificationSuite(spark).onData(df).addCheck(check.isComplete("a")).run()                                                                                               
                                                                                                                                                                                 
Error: TypeError: 'JavaPackage' object is not callable                                                                                                                           
                                                                                                                                                                                 
Using Spark 3.3, Python 3.9, pydeequ 1.3.0 

[github-actions]

The TypeError: 'JavaPackage' object is not callable error means the Deequ JAR is not on Spark's classpath. You're missing the spark.jars.packages configuration when building your SparkSession. Also make sure you set the SPARK_VERSION environment variable before importing pydeequ.

Fix your setup like this:

import os
os.environ["SPARK_VERSION"] = "3.3"

from pyspark.sql import SparkSession, Row
import pydeequ
from pydeequ.checks import Check, CheckLevel
from pydeequ.verification import VerificationSuite, VerificationResult

spark = (SparkSession
    .builder
    .master("local")
    .appName("test")
    .config("spark.jars.packages", pydeequ.deequ_maven_coord)
    .config("spark.jars.excludes", pydeequ.f2j_maven_coord)
    .getOrCreate())

df = spark.createDataFrame([Row(a="foo", b=1), Row(a="bar", b=2)])

check = Check(spark, CheckLevel.Warning, "test")
result = VerificationSuite(spark).onData(df).addCheck(check.isComplete("a")).run()

The two key pieces are:

1. SPARK_VERSION env var — must be set before import pydeequ
2. spark.jars.packages — tells Spark to download and load the Deequ JAR at session creation

Without the JAR, Py4J can't find the Deequ classes and returns a JavaPackage stub instead of the actual class, which causes your error.

---

This response was generated using AI and may not be fully accurate. If this doesn't help, please reply and a maintainer will follow up.

[sudsali]

Thanks, but I'm running on EMR and can't set SPARK_VERSION before the session starts. Is there a way to pass the Deequ JAR coordinates manually?

[github-actions]

Yes, on EMR you can handle this a couple of ways:

1. Set SPARK_VERSION early in your script (before importing pydeequ):

Even on EMR, you can set the env var at the top of your script before any pydeequ import:

import os
os.environ["SPARK_VERSION"] = "3.3"  # match your EMR Spark version

import pydeequ
from pydeequ.checks import Check, CheckLevel
from pydeequ.verification import VerificationSuite, VerificationResult

This doesn't require setting it before the session starts — it just needs to be set before import pydeequ, since pydeequ/configs.py reads it at import time to resolve the correct Deequ Maven coordinate.

2. Pass the Deequ JAR coordinate manually (bypassing pydeequ.deequ_maven_coord):

If you prefer not to use the env var at all, you can hardcode the Maven coordinate directly:

spark = (SparkSession
    .builder
    .config("spark.jars.packages", "com.amazon.deequ:deequ:2.0.8-spark-3.3")
    .config("spark.jars.excludes", "net.sourceforge.f2j:arpack_combined_all")
    .getOrCreate())

But note that you'll still need SPARK_VERSION set before import pydeequ, because pydeequ/configs.py reads it at module load time and will raise a RuntimeError if it's missing. So you'd do:

import os
os.environ["SPARK_VERSION"] = "3.3"

before any pydeequ imports regardless.

3. On EMR specifically, if the SparkSession is already created for you (e.g., in an EMR notebook), you can add the Deequ JAR via --packages in your EMR step configuration or cluster bootstrap, and then just set the env var at the top of your notebook/script before importing pydeequ.

The supported Spark→Deequ mappings are:

Spark	Deequ Maven Coordinate
3.1	com.amazon.deequ:deequ:2.0.8-spark-3.1
3.2	com.amazon.deequ:deequ:2.0.8-spark-3.2
3.3	com.amazon.deequ:deequ:2.0.8-spark-3.3
3.5	com.amazon.deequ:deequ:2.0.8-spark-3.5

Pick the one matching your EMR Spark version.

---

This response was generated using AI and may not be fully accurate. If this doesn't help, please reply and a maintainer will follow up.