feat: introduce LateralJoinRel for a correlated subquery evaluation

[yongchul]

Background

SQL or relational query plan may have subqueries. Some subqueries can be easily decorrelated in terms of joins but in other cases it is non-trivial to do such rewriting. Some queries can be de-correlated (i.e., rewritten as a join) but sometimes it is non-trivial to decorrelate complex subqueries.

ApplyRel is one of the way to model a generic correlated subquery execution and implemented in multiple systems (e.g., Google Spanner, Oracle, SQL Server family).

The semantic of apply is that the subquery is executed per row from the input rather than operating on set of the rows at a time like join. The subquery can reference columns in the input row as outer references (thus each subquery is correlated to each input row).

Apply operation is inherently expensive thus typically used in a special context such as index lookup join or in a pinch where a system couldn't decorrelate the operation.

Proposal

Extend JoinRel with lateral flag to represent lateral join.
LATERAL keyword was introduced in SQL 99, as part of table reference. When the table source is wrapped with LATERAL, the subquery MAY reference columns and tables preceding the LATERAL table reference, and allows correlated subquery.

By this nature of scoping of tables and references, we propose to extend JoinRel with lateral flag to denote lateral join, and let right input as the correlated subquery, which can reference the fields from left input.

In this way, we reuse all the join semantics automatically without introducing a dedicated Rel.

Same change was made to CrossRel.

We decided to introduce a new LateralJoinRel for clarity and to ease handling the new semantic in existing consumers.

The change is backward compatible.

This PR depends on #1031 .

AI disclaimer

This PR was assisted by Claude Opus 4.6.

---

This change is 

[vbarua]

I'm not super familiar with LATERAL in SQL, but if I was to rephrase the problem we're trying to solve it's that to express LATERAL-style joins in a JOIN like:

  JoinRel
  /     \
 LHS    RHS

we need to be able to reference LHS outputs in the RHS (or vice-versa).

This type capability has come up before for me in the context of dynamic filtering, where we need to be able to reference fields from other join inputs in the

substrait/proto/substrait/algebra.proto

Line 109 in dc27654

Expression best_effort_filter = 11;

field to express a (potential) data dependency.

I'm a little wary of re-using Outer References like you've suggested, because as I understand it nothing about the construction were talking prevents from putting actual subqueries in the subtree between the Join relation and where the field is referenced.

To avoid this ambiguity, would it make sense to have a specialized reference type like JoinReference in the same vein as OuterReference and LambdaParameterReference?

[yongchul]

I'm not super familiar with LATERAL in SQL, but if I was to rephrase the problem we're trying to solve it's that to express LATERAL-style joins in a JOIN like:

  JoinRel
  /     \
 LHS    RHS

we need to be able to reference LHS outputs in the RHS (or vice-versa).

No, not vice-versa. The correlation is typically one direction (from left to right) due to scoping.

This type capability has come up before for me in the context of dynamic filtering, where we need to be able to reference fields from other join inputs in the

substrait/proto/substrait/algebra.proto

Line 109 in dc27654

Expression best_effort_filter = 11;

field to express a (potential) data dependency.

I'm a little wary of re-using Outer References like you've suggested, because as I understand it nothing about the construction were talking prevents from putting actual subqueries in the subtree between the Join relation and where the field is referenced.

I wish we just ditch the current outer reference honestly and do it differently -- more like global id reference to avoid hair-pulling wiring outer references in the other PR. I don't understand "putting actual subqueries in the subtree join relation"... could you elaborate your concern? Or, it is actually better to do a quick call if you have time.

Anyhow, each subquery would put the "depth" -- only in the context of set predicate today-- and this PR proposing lateral join will put another depth in the nested stack of schema. So I don't see any problem...

To avoid this ambiguity, would it make sense to have a specialized reference type like JoinReference in the same vein as OuterReference and LambdaParameterReference?

I don't know. OuterReference is OuterReference, you are referencing something outside of your usual column resolution scope. LambdaParameterReference is making sense because there is a clear binding point. OuterReference is generic, and based on the "correlation" boundary -- either the Rel containing set predicate or the left input Rel of lateral join in this PR.

If you prefer more explicit like LambdaParameterReference, we could explore id based declaration, and binding, something like

JoinRel {
  repeated CorrelatedFields correlated_left_fields;

  message CorrelatedField
  {
      int anchor = 1;  // plan level unique. or we can give an id to this joinrel, then joinrel id with field reference identify unique reference.
      FieldReference correlated_field = 2;    // correlated field
  }
}

message CorrelatedFieldReference
{
    int correlated_reference = 1;
}

Basically, the idea is for outer references, going with more like name or identifier based approach rather than offset based approach. Yes, it is departure from the basic referencing scheme but this is much simpler to reuse expressions and binding the correct columns/fields without going through the hassles of massaging offsets and depth, which depends on context.

I can draft this as well and we can see how it look like.

[nielspardon]

It took me a while to wrap my ahead around this. I think I understand the idea now.

What makes me feel a little uneasy is that the lateral flag changes quite a bit of the JoinRel and CrossRel semantics like restricting the supported join types. This makes me think that having a separate relation would be easier to understand. We would have to re-declare some of the join semantics on the new relation but we would have less conditional semantics baked into JoinRel / CrossRel. If we had a separate relation which only had the join types that make sense for a lateral join that would be easier to understand than the conditional logic of the current proposal.

I was curious how Calcite deals with lateral joins and they have a separate Correlate operator which they also chose since it was easier to distinguish between Join and Correlate in optimization rules since they are claiming most join optimization rules shouldn't be applied to correlates (if I understand it correctly).

https://github.com/apache/calcite/blob/17bf870d32d1362055e1df0050eec4e7b0f7e03c/core/src/main/java/org/apache/calcite/rel/core/Correlate.java#L54

[nielspardon]

LGTM

[nielspardon]

I think we would also need to update the Substrait dialect schema to allow implementations to declare their support for the new relation.

[vbarua]

Left some very minor suggestions, but otherwise LGTM!

[vbarua]

👿 Hat: One could make the case that this should be it's own enum with only the variants supported by LateralJoinRel. Which is what we've done in the past.

Taking the 👿 off, that has historically made the libraries more complicated because we end up with a bunch of custom enum wrappers. Dialects feel like a better solution and story for validations in going forwards as well.

I like this unification on the JoinRel enum here, and would be interested in updating the other Join relations do this if possible.

[yongchul]

Either this (use JoinRel.JoinType) or just lift the join type to top level as you did in one of the PR. This is something we should clean up IMO.

[benbellick]

Apologies for such a late review. a few minor things but otherwise looks great! Thanks for your patience. 🚀