feat: clarify post_join_filter. add residual_expressions to hash join and merge join (#1044)

# Background

JoinRel, HashJoinRel, and MergeJoinRel have a field
named`post_join_filter`, causing confusion. The confusing point is
whether `post_join_filter` is part of the join predicate (i.e., two rows
are **matched** when it evaluates to `true`) or not (i.e., as the name
says, this is **post** join filter, thus `filter` evaluates after the
join operation).

Substrait
[FAQ](https://substrait.io/faq/#what-is-the-purpose-of-the-post-join-filter-field-on-join-relations)
calls out that the `post_join_filter` is part of join predicate but
@yongchul was deeply discontent about the explanation and the naming.

After lengthy discussion in the Slack channel, #807 was created. After
more back-and-forth, the original intent of `post_join_filter` was
indeed [**post** join filter, not part of join
predicate](#807 (comment)).

Also, it appears that Calcite has precisely the same name and same
semantic, post join filter. Following code point shows placing the
filter on top of the join.

https://github.com/apache/calcite/blob/0a39568b167592ded8db1128b5838982ffe264f3/core/src/main/java/org/apache/calcite/rel/rules/LoptOptimizeJoinRule.java#L553-L559

# What this changes do?

TL;DR; Make `post_join_filter` as `post_join_filter` again.

1. Documents `post_join_filter` is not join condition but semantically
filter on top of join.
2. Introduce `residual_expression` to HashJoinRel and MergeJoinRel so
that they can express non-equality join predicate -- I guess this was
the confusion who tried to shove non-equality join predicate to
`HashJoin` and `MergeJoin` somewhere and distorted the meaning without
much thought.
3. Drop `equi` from hash join and merge join documentation as with
`residual_expression` they are capable of supporting arbitrary join
predicate.
4. Fix the FAQ.

# Breaking change?

This is debatable. According to previous FAQ, `post_join_filter` was
`residual_expression` introduced in this PR. If we take the FAQ as part
of the "spec", then this is a breaking change.

If we don't consider FAQ as a spec, then this is not a breaking change
but clarification and adding a new feature -- HashJoinRel, MergeJoinRel
now supports arbitrary join predicate.

# AI disclaimer

The PR is assisted by Claud Opus 4.6.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg"
height="34" align="absmiddle"
alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1044)
<!-- Reviewable:end -->

Author: yongchul

proto/substrait/algebra.proto

@@ -261,12 +261,18 @@ message ProjectRel {
   substrait.extensions.AdvancedExtension advanced_extension = 10;
 }
 
-// The binary JOIN relational operator left-join-right, including various join types, a join condition and post_join_filter expression
+// The binary JOIN relational operator left-join-right, including various join
+// types, a join condition, and an optional filter on the joined output.
 message JoinRel {
   RelCommon common = 1;
   Rel left = 2;
   Rel right = 3;
+  // A boolean condition evaluated over the left and right inputs that
+  // determines whether a pair of records is a match.
   Expression expression = 4;
+  // An optional boolean filter applied to each output record after
+  // join-type-specific output formation is complete.
+  // Semantically equivalent to placing a FilterRel directly above this join.
   Expression post_join_filter = 5;
 
   JoinType type = 6;
@@ -837,10 +843,12 @@ message ComparisonJoinKey {
   }
 }
 
-// The hash equijoin operator will build a hash table out of one input (default `right`) based on a set of join keys.
-// It will then probe that hash table for the other input (default `left`), finding matches.
+// The hash join operator will build a hash table out of one input (default
+// `right`) based on a set of join keys. It will then probe that hash table for
+// the other input (default `left`), finding matches.
 //
-// Two rows are a match if the comparison function returns true for all keys
+// Two rows are a match if every key comparison returns true and, when
+// specified, `residual_expression` also evaluates to true.
 message HashJoinRel {
   RelCommon common = 1;
   Rel left = 2;
@@ -864,12 +872,23 @@ message HashJoinRel {
   // hash function for a given comparsion function or to reject the plan if it cannot
   // do so.
   repeated ComparisonJoinKey keys = 8;
+  // An optional boolean filter applied to each output record after
+  // join-type-specific output formation is complete.
+  // Semantically equivalent to placing a FilterRel directly above this join.
   Expression post_join_filter = 6;
 
   JoinType type = 7;
 
-  // Specifies which side of input to build the hash table for this hash join. Default is `BUILD_INPUT_RIGHT`.
+  // Specifies which side of input to build the hash table for this hash join.
+  // Defaults to `BUILD_INPUT_RIGHT`.
   BuildInput build_input = 9;
+  // An optional boolean expression evaluated on each candidate key-match to
+  // determine whether it is a true match. A candidate is only considered a
+  // match when all `keys` comparisons AND this expression evaluate to true.
+  // This expression interacts with join-type semantics: for example, in a
+  // left outer join, a left row that has key matches but no candidate
+  // satisfying this expression will be emitted with nulls for the right side.
+  Expression residual_expression = 11;
 
   enum JoinType {
     JOIN_TYPE_UNSPECIFIED = 0;
@@ -896,8 +915,9 @@ message HashJoinRel {
   substrait.extensions.AdvancedExtension advanced_extension = 10;
 }
 
-// The merge equijoin does a join by taking advantage of two sets that are sorted on the join keys.
-// This allows the join operation to be done in a streaming fashion.
+// The merge join does a join by taking advantage of two sets that are
+// sorted on the join keys. This allows the join operation to be done in a
+// streaming fashion.
 message MergeJoinRel {
   RelCommon common = 1;
   Rel left = 2;
@@ -923,9 +943,19 @@ message MergeJoinRel {
   // is free to do so as well).  If possible, the consumer should verify the sort
   // order and reject invalid plans.
   repeated ComparisonJoinKey keys = 8;
+  // An optional boolean filter applied to each output record after
+  // join-type-specific output formation is complete.
+  // Semantically equivalent to placing a FilterRel directly above this join.
   Expression post_join_filter = 6;
 
   JoinType type = 7;
+  // An optional boolean expression evaluated on each candidate key-match to
+  // determine whether it is a true match. A candidate is only considered a
+  // match when all `keys` comparisons AND this expression evaluate to true.
+  // This expression interacts with join-type semantics: for example, in a
+  // left outer join, a left row that has key matches but no candidate
+  // satisfying this expression will be emitted with nulls for the right side.
+  Expression residual_expression = 9;
 
   enum JoinType {
     JOIN_TYPE_UNSPECIFIED = 0;

site/docs/faq.md

@@ -6,9 +6,12 @@ title: FAQ
 
 ## What is the purpose of the post-join filter field on Join relations?
 
-The post-join filter on the various Join relations is not always equivalent to an explicit Filter relation AFTER the Join.
+`post_join_filter` is an optional filter on the output records of a join. It is applied after matching and any join-type–specific post-processing (e.g., null-padding unmatched rows for outer joins, emitting unmatched rows for anti joins) are complete. It is semantically equivalent to placing a separate `Filter` relation directly above the join. Because it does not participate in matching, it has no effect on which rows are considered matched or unmatched for the purpose of outer, anti, semi, or mark join semantics. When omitted it defaults to `true`.
 
-See the example [here](https://facebookincubator.github.io/velox/develop/joins.html#hash-join-implementation) that highlights how the post-join filter behaves differently than a Filter relation in the case of a left join.
+This is distinct from predicates that *do* participate in match determination:
+
+- In `JoinRel`, all match predicates belong in `expression`.
+- In `HashJoinRel` and `MergeJoinRel`, equijoin predicates go in `keys` and any remaining match predicates go in `residual_expression`.
 
 ## Why does the project relation keep existing columns?
 

site/docs/relations/logical_relations.md

@@ -237,8 +237,8 @@ The join operation will combine two separate inputs into a single output, based
 | Inputs               | 2                                                            |
 | Outputs              | 1                                                            |
 | Property Maintenance | Distribution is maintained. Orderedness is empty post operation. Physical relations may provide better property maintenance. |
-| Input Order          | The input order is the left input followed by the right input. All field references of [Join Properties](#join-properties) are over this order. |
-| Direct Output Order  | For semi joins and anti joins, the emit order is either left or right only. For mark joins, the emit order is either left or right with a "mark" column appended at the end. See [Join Types](#join-types) for detail. Otherwise, the same as `Input Order`. |
+| Input Order          | The input order is the left input followed by the right input. All field references of [Join Properties](#join-properties) except Post-Join Filter are over this order. |
+| Direct Output Order  | For semi joins and anti joins, the emit order is either left or right only. For mark joins, the emit order is either left or right with a "mark" column appended at the end. See [Join Types](#join-types) for detail. Otherwise, the same as `Input Order`. All field references of Post-Join Filter are over this order. |
 
 ### Join Properties
 
@@ -247,7 +247,7 @@ The join operation will combine two separate inputs into a single output, based
 | Left Input       | A relational input.                                          | Required                           |
 | Right Input      | A relational input.                                          | Required                           |
 | Join Expression  | A boolean condition that describes whether each record from the left set "match" the record from the right set. Field references correspond to the input order of the data. | Required. Can be the literal True. |
-| Post-Join Filter | A boolean condition to be applied to each result record after the inputs have been joined, yielding only the records that satisfied the condition. | Optional                           |
+| Post-Join Filter | An optional boolean condition applied to the output of the join. Semantically equivalent to placing a [Filter](#filter-operation) directly above the join. Does not influence which rows are considered matches. Field references correspond to the direct output order of the join operation. | Optional, defaults to True.        |
 | Join Type        | One of the join types defined below.                         | Required                           |
 
 ### Join Types

site/docs/relations/physical_relations.md

@@ -2,9 +2,9 @@
 
 There is no true distinction between logical and physical operations in Substrait. By convention, certain operations are classified as physical, but all operations can be potentially used in any kind of plan. A particular set of transformations or target operators may (by convention) be considered the "physical plan" but this is a characteristic of the system consuming substrait as opposed to a definition within Substrait.
 
-## Hash Equijoin Operator
+## Hash Join Operator
 
-The hash equijoin join operator will build a hash table out of one input (default `right`) based on a set of join keys. It will then probe that hash table for the other input (default `left`), finding matches.
+The hash join operator will build a hash table out of one input (default `right`) based on a set of join keys. It will then probe that hash table for the other input (default `left`), finding matches, then use `residual_expression` to determine whether the matches are true matches.
 
 | Signature            | Value                                                             |
 | -------------------- | ----------------------------------------------------------------- |
@@ -14,7 +14,7 @@ The hash equijoin join operator will build a hash table out of one input (defaul
 | Input Order          | Same as the [Join](logical_relations.md#join-operation) operator. |
 | Direct Output Order  | Same as the [Join](logical_relations.md#join-operation) operator. |
 
-### Hash Equijoin Properties
+### Hash Join Properties
 
 | Property            | Description                                                                                                                                                                                                             | Required                                           |
 | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
@@ -23,7 +23,8 @@ The hash equijoin join operator will build a hash table out of one input (defaul
 | Build Input         | Specifies which input is the `Build`.                                                                                                                                                                                   | Optional, defaults to build `Right`, probe `Left`. |
 | Left Keys           | References to the fields to join on in the left input.                                                                                                                                                                  | Required                                           |
 | Right Keys          | References to the fields to join on in the right input.                                                                                                                                                                 | Required                                           |
-| Post Join Predicate | An additional expression that can be used to reduce the output of the join operation post the equality condition. Minimizes the overhead of secondary join conditions that cannot be evaluated using the equijoin keys. | Optional, defaults true.                           |
+| Residual Expression | An optional boolean expression evaluated on each candidate key-match to determine whether it is a true match. Use this for join predicates that cannot be expressed as equijoin keys (e.g., range or inequality conditions). This expression participates in join-type semantics (outer, anti, mark, etc.). | Optional, defaults to True. |
+| Post-Join Filter    | An optional boolean condition applied to the output of the join. Semantically equivalent to placing a [Filter](logical_relations.md#filter-operation) directly above the join. Does not influence which rows are considered matches. All field references are over direct output order. | Optional, defaults to True. |
 | Join Type           | One of the join types defined in the Join operator.                                                                                                                                                                     | Required                                           |
 
 ## NLJ (Nested Loop Join) Operator
@@ -47,9 +48,9 @@ The nested loop join operator does a join by holding the entire right input and
 | Join Expression | A boolean condition that describes whether each record from the left set "match" the record from the right set. | Optional. Defaults to true (a Cartesian join). |
 | Join Type       | One of the join types defined in the Join operator.                                                             | Required                                       |
 
-## Merge Equijoin Operator
+## Merge Join Operator
 
-The merge equijoin does a join by taking advantage of two sets that are sorted on the join keys. This allows the join operation to be done in a streaming fashion.
+The merge join does a join by taking advantage of two sets that are sorted on the join keys. This allows the join operation to be done in a streaming fashion. Once the join keys are matched, then use `residual_expression` to determine whether the matches are true matches.
 
 | Signature            | Value                                                             |
 | -------------------- | ----------------------------------------------------------------- |
@@ -67,7 +68,8 @@ The merge equijoin does a join by taking advantage of two sets that are sorted o
 | Right Input         | A relational input.                                                                                                                                                                                                     | Required                 |
 | Left Keys           | References to the fields to join on in the left input.                                                                                                                                                                  | Required                 |
 | Right Keys          | References to the fields to join on in the right input.                                                                                                                                                                 | Required                 |
-| Post Join Predicate | An additional expression that can be used to reduce the output of the join operation post the equality condition. Minimizes the overhead of secondary join conditions that cannot be evaluated using the equijoin keys. | Optional, defaults true. |
+| Residual Expression | An optional boolean expression evaluated on each candidate key-match to determine whether it is a true match. Use this for join predicates that cannot be expressed as equijoin keys (e.g., range or inequality conditions). This expression participates in join-type semantics (outer, anti, mark, etc.). | Optional, defaults to True. |
+| Post-Join Filter    | An optional boolean condition applied to the output of the join. Semantically equivalent to placing a [Filter](logical_relations.md#filter-operation) directly above the join. Does not influence which rows are considered matches. All field references are over direct output order. | Optional, defaults to True. |
 | Join Type           | One of the join types defined in the Join operator.                                                                                                                                                                     | Required                 |
 
 ## Exchange Operator