SQL Subquery Decorrelation

[damccorm]

Adds a custom subquery decorrelation pass to Beam SQL to handle complex correlated subqueries.

The Problem

Calcite's default SqlToRelConverter decorrelates most queries, but some complex correlated shapes (like correlated EXISTS or IN subqueries inside PROJECT or JOIN conditions) survive as RexSubQuery or residual LogicalCorrelate nodes.

Beam's Volcano planner ruleset has no physical converter rules for a general LogicalCorrelate node. When these residues survive, the planner fails with a CannotPlanException.

The Fix

Implemented a pre-Volcano normalization pass in CalciteQueryPlanner.convertToBeamRel:

1. normalizeForVolcano: This pass runs a short-lived HepPlanner with Calcite's *SUB_QUERY_TO_CORRELATE rules to turn any un-expanded RexSubQuery into a LogicalCorrelate.
2. It then calls RelDecorrelator.decorrelateQuery to lower these correlates into standard relational shapes (Join, Aggregate, Project, Filter) which existing Beam rules already know how to handle.
3. This pass is strictly gated on the tree actually referencing a correlation variable (using RelOptUtil.getVariablesUsed), making it a no-op for shapes like UNNEST (LogicalCorrelate(_, Uncollect)) which define but do not reference a correlation variable, preserving them for BeamUnnestRule.
4. It runs before the Volcano planner and cost-based metadata provider swap, keeping it off the recursive cost path.

Added a detailed design document in decorrelation_design.md explaining the approach and the Volcano planner convergence safety.

[codecov]

Codecov Report

❌ Patch coverage is 37.30159% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.32%. Comparing base (d9117f4) to head (27b2d0b).
⚠️ Report is 55 commits behind head on master.

Files with missing lines	Patch %	Lines
...m/sdk/extensions/sql/impl/CalciteQueryPlanner.java	36.36%	64 Missing and 6 partials ⚠️
...pache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java	30.00%	7 Missing ⚠️
...che/beam/sdk/extensions/sql/impl/QueryPlanner.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #38929      +/-   ##
============================================
- Coverage     57.77%   54.32%   -3.46%     
+ Complexity    12969     1707   -11262     
============================================
  Files          2509     1062    -1447     
  Lines        260525   166535   -93990     
  Branches      10658     1252    -9406     
============================================
- Hits         150516    90462   -60054     
+ Misses       104318    73860   -30458     
+ Partials       5691     2213    -3478     

Flag	Coverage Δ
java	67.05% <37.30%> (+2.87%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.