How do I use PyDeequ with Spark 3.5 on EMR?

[sudsali]

I'm trying to run PyDeequ on EMR with Spark 3.5 but getting a RuntimeError about SPARK_VERSION. What's the correct setup?

from pydeequ.checks import Check, CheckLevel                                                                                                                                     
from pydeequ.verification import VerificationSuite                                                                                                                               

Error: RuntimeError: SPARK_VERSION environment variable is required

[github-actions]

Thank you for reporting this.

This has been flagged for review by our maintainer team. We'll get back to you as soon as possible.

---

This response was generated using AI and may not be fully accurate. If this doesn't help, please reply and a maintainer will follow up.