Cannot generate correct golden files for TPC-DS q9

[andygrove]

Describe the bug

#2492 re-enabled the plan stability checks. I had to work around an issue with the expected plan for q9 on Spark 3.5 by manually editing the approved plan.

There is a difference between the explain plan for q9 depending on whether SPARK_GENERATE_GOLDEN_FILES is specified.

Generating new golden files for Spark 3.5 after #2492 is merged will always generate a different approved plan for q9.

The difference is related to a CometHashAggregateExec and the output contains one of the following, depending on the value of regenerateGoldenFiles:

count#34, sum#35, count#36, sum#37, count#38

or

count#34, sum#35, count#36, sum#15, count#37

The suite runs for a different set of queries depending on whether regenerateGoldenFiles is true or false, so perhaps that has something to do with the issue, but I am not sure.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

After more testing, it appears that the explain plan for q9 is non-deterministic.

[andygrove]

I can no longer reproduce this