Is PyDeequ 2.0 still being actively developed? State of PyDeequ in general

[billpratt]

We’re currently running PyDeequ on Spark 3.5 and planning a future migration to Spark 4.1 / Delta Lake 4.1 / Python 3.13. Before we commit further to PyDeequ, we’d like to understand the project’s current state.

The 2.0.0b1 beta was announced in January 2025 with Spark Connect support — great direction, since _jvm bridge calls aren’t supported under Spark Connect which is the default in Spark 4.x. However, the last PyPI release was v1.5 in April 2025 and there’s been no follow-up since.

Two questions:

1. Is the project in active development or maintenance mode?

2. Is there a plan to support Spark 4.1 and get 2.0 published to PyPI?

This will directly inform whether we continue building on PyDeequ or evaluate alternatives ahead of our migration. Thanks for any visibility.

[github-actions]

Thanks for raising this — it's a fair and important question for teams planning migrations.

To be transparent: PyDeequ development has slowed significantly. The core maintainers have had limited bandwidth, and while the 2.0.0b1 beta with Spark Connect support was a step in the right direction, there is currently no published timeline for a stable 2.0 release on PyPI or official Spark 4.x support.

The project is not formally abandoned, but it would be accurate to describe it as in maintenance mode rather than active feature development at this time. Community contributions are welcome and will be reviewed.

Regarding your specific questions:

1. Active development vs. maintenance mode — Closer to maintenance mode currently. Bug fixes and community PRs are still accepted, but there's no dedicated team driving new features on a regular cadence.

2. Spark 4.1 / PyPI 2.0 plan — There is no concrete roadmap for Spark 4.1 support at this time. The _jvm bridge dependency is indeed the core blocker for Spark Connect compatibility, which the 2.0 beta began addressing. If your migration timeline is near-term, I'd recommend evaluating alternatives in parallel (e.g., native Deequ on Scala/Spark, or other Python data quality frameworks) rather than blocking on a PyDeequ 2.0 GA release.

If you or your team are interested in contributing toward Spark 4.x support, we'd welcome the collaboration. Otherwise, I understand if you need to hedge your bets given the uncertainty.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.

[billpratt]

@chenliu0831

[github-actions]

This issue is tagging a maintainer (@chenliu0831) directly for visibility on the project's roadmap and development status. This requires an official response from the PyDeequ maintainers regarding Spark 4.x support plans and the 2.0 release timeline — I cannot speak authoritatively on behalf of the project's direction.

This has also been flagged for our maintainer team to review.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 67928783) — may not be fully accurate. Reply if this doesn't help.

[chenliu0831]

I'm still pushing a few updates on Spark Connect support recently and working with rest of the maintaining team in AWS to make sure they are comfortable. Deequ & its Python binding (PyDeequ) will still be maintained. As far as I know, Spark connect and Spark 4+ support is desired by the internal owning team towards the second half of this year.

[billpratt]

Great to hear and thanks for the update. Let me know if I can help. Also this PR was recently merged: #262. Do we need to apply it to the 2.0 branch as well? What's the release cycle for these types of changes for 1.X?

[chenliu0831]

Do we need to apply it to the 2.0 branch as well? What's the release cycle for these types of changes for 1.X?

It would depend on how long we need to keep Py4J bridge approach (could be forever). The version strategy is yet to be aligned with the maintaining team. If you feel the need to cherry pick some changes to V2 branch, shoot me a PR.

[github-actions]

I've reached the limit of what I can assist with on this issue. The maintainer team has been notified and will take over.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: n/a) — may not be fully accurate. Reply if this doesn't help.