[SPARK-56355][SQL] Improve join stats estimation when equi-join keys lack column statistics

[wecharyu]

What changes were proposed in this pull request?

Improve join estimation in case where equal-join's join key column stats are missing. Currently, the estimator could fall back to a Cartesian product T(A) * T(B), which often leads to severe overestimation.

We can instead use max(T(A), T(B)) as a more accurate estimate.

Why are the changes needed?

To improve the reliability of join estimation especially in AQE re-optimize.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Add unit test, and test on the prod Spark job.

TPC-DS benchmark shows no performance regression:

• Before

[info] TPCDS:                                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] q24a                                             237404         237715         440          1.3         775.7       1.0X

• After

[info] Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
[info] TPCDS:                                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] q24a                                             236725         237440        1011          1.3         773.5       1.0X

Was this patch authored or co-authored using generative AI tooling?

Cursor / Opus 4.6

[wecharyu]

cc @cloud-fan @wangyum

[lakechd]

why leftouter not inner

[wecharyu]

test for both inner and leftouter

[wecharyu]

gentle ping @cloud-fan , could you help take a look? The original over-estimation could lead Gluten choose wrong build side and cause OOM, details in apache/gluten#11774