AnalysisRunner groupBy throws UNRESOLVED_COLUMN error when DataFrame has a 'count' column

[sudsali]

I'm using PyDeequ with Spark 3.4 and getting this error when running hasNumberOfDistinctValues on a DataFrame that has a column named "count":

AnalysisException: [UNRESOLVEDCOLUMN.WITHSUGGESTION] A column or function parameter with name com_amazon_deequ_dq_metrics_count cannot be resolved

check = Check(spark, CheckLevel.Error, "test")                                                                                                                                   
check.hasNumberOfDistinctValues("count", lambda x: x == 3)                                                                                                                       
result = VerificationSuite(spark).onData(df).addCheck(check).run()                                                                                                               

This seems like a bug in how Deequ handles column name conflicts internally. The Histogram analyzer renames the column but then can't find it.

[github-actions]

This issue may not be related to the PyDeequ data quality library. The maintainer team has been notified and will review.

---

This response was generated using AI and may not be fully accurate. If this doesn't help, please reply and a maintainer will follow up.