Address review comments on #5488

Per @penghuo review:

PlanUtils.java
- Revert mayBeFilterFromBucketNonNull to the original ternary form; drop the
  early-return refactor (no behavior change, just cleaner).
- Drop the !rexCall.getOperands().isEmpty() guard before .get(0): IS NOT NULL
  is always unary in Calcite, so the check is dead.
- Trim the JavaDoc to the essentials (un-merged vs merged-AND shape; concrete
  partition-key match happens in PPLSimplifyDedupRule#apply).
- Promote isNotNullOnRef from package-private to public so the dedup rule can
  reuse it from a different package.

PPLSimplifyDedupRule.java
- isNotNullOnPartitionKey now delegates the IS NOT NULL($ref) structural check
  to PlanUtils.isNotNullOnRef and adds the partition-key index check on top.

CalciteExplainIT.java
- Add testDedupAfterWherePushDown: an end-to-end regression that runs the
  shape `... | where <pred> | dedup <field>` and asserts (a) LogicalDedup
  appears in the explain output (PPLSimplifyDedupRule fired even after
  FilterMergeRule had a chance to merge the two filters), and (b)
  EnumerableWindow does NOT appear (the in-memory ROW_NUMBER fallback the
  bug caused is gone).

Signed-off-by: Jialiang Liang <jiallian@amazon.com>

Author: RyanL1997

core/src/main/java/org/opensearch/sql/calcite/plan/rule/PPLSimplifyDedupRule.java

@@ -172,11 +172,11 @@ protected void apply(
   }
 
   private static boolean isNotNullOnPartitionKey(RexNode rex, Set<Integer> partitionKeyIndices) {
-    return rex instanceof RexCall rexCall
-        && rexCall.isA(SqlKind.IS_NOT_NULL)
-        && !rexCall.getOperands().isEmpty()
-        && rexCall.getOperands().get(0) instanceof RexInputRef ref
-        && partitionKeyIndices.contains(ref.getIndex());
+    if (!PlanUtils.isNotNullOnRef(rex)) {
+      return false;
+    }
+    RexInputRef ref = (RexInputRef) ((RexCall) rex).getOperands().get(0);
+    return partitionKeyIndices.contains(ref.getIndex());
   }
 
   private static @Nullable RelCollation extractCollationFromWindow(RexWindow window) {

core/src/main/java/org/opensearch/sql/calcite/utils/PlanUtils.java

@@ -786,44 +786,22 @@ static Mapping mapping(List<RexNode> rexNodes, RelDataType schema) {
   }
 
   /**
-   * Loose structural check for the bucket-non-null filter PPL emits as part of the {@code dedup}
-   * pattern. Returns true for either:
-   *
-   * <ul>
-   *   <li>a pure {@code IS NOT NULL($ref)}, or
-   *   <li>any {@code AND} whose conjuncts contain at least one {@code IS NOT NULL($ref)}.
-   * </ul>
-   *
-   * <p>This intentionally accepts the un-merged shape ({@code IS NOT NULL} alone) and the merged
-   * shape that {@link org.apache.calcite.rel.rules.FilterMergeRule} produces when a user {@code
-   * where} clause precedes the dedup ({@code AND(IS NOT NULL($pk), <user predicate>)}). The
-   * concrete partition-key match — and the split-out of any user predicate that was folded in —
-   * happens inside {@code PPLSimplifyDedupRule#apply}, which has access to the partition keys
-   * declared by the {@code ROW_NUMBER} window above this filter. Keeping the operand predicate
-   * order-independent means the simplify rule fires regardless of whether {@code FilterMergeRule}
-   * has already run, so dedup pushdown is no longer load-bearing on HEP rule ordering.
-   *
-   * <p>Misclassification is bounded by the surrounding four-level operand chain: a project that
-   * does not contain the {@code _row_number_dedup_} column, above a filter whose condition is
-   * {@code _row_number_ <= N} and which does contain {@code _row_number_dedup_}, above a project
-   * that does contain {@code _row_number_dedup_} (with a {@code ROW_NUMBER} window function), above
-   * this filter. That signature is unique to the PPL-emitted dedup pattern, so a user-written
-   * filter such as {@code IS NOT NULL(x) AND ...} cannot accidentally match.
+   * Accepts the un-merged {@code IS NOT NULL($ref)} shape and the merged-{@code AND} shape that
+   * {@link org.apache.calcite.rel.rules.FilterMergeRule} produces when a user {@code where}
+   * precedes the dedup. The concrete partition-key match — and the split-out of any user predicate
+   * folded into the AND — happens in {@link PPLSimplifyDedupRule#apply}.
    */
   static boolean mayBeFilterFromBucketNonNull(LogicalFilter filter) {
     RexNode condition = filter.getCondition();
-    if (isNotNullOnRef(condition)) {
-      return true;
-    }
-    return condition instanceof RexCall rexCall
-        && rexCall.getOperator().equals(SqlStdOperatorTable.AND)
-        && rexCall.getOperands().stream().anyMatch(PlanUtils::isNotNullOnRef);
+    return isNotNullOnRef(condition)
+        || (condition instanceof RexCall rexCall
+            && rexCall.getOperator().equals(SqlStdOperatorTable.AND)
+            && rexCall.getOperands().stream().anyMatch(PlanUtils::isNotNullOnRef));
   }
 
-  static boolean isNotNullOnRef(RexNode rex) {
+  public static boolean isNotNullOnRef(RexNode rex) {
     return rex instanceof RexCall rexCall
         && rexCall.isA(SqlKind.IS_NOT_NULL)
-        && !rexCall.getOperands().isEmpty()
         && rexCall.getOperands().get(0) instanceof RexInputRef;
   }
 

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java

@@ -2153,6 +2153,29 @@ public void testTransposeExplain() throws IOException {
                 + "|  transpose 4 column_name='column_names'"));
   }
 
+  /**
+   * Regression test for #5482: a user {@code where} clause adjacent to dedup's bucket-non-null
+   * filter must still let {@code PPLSimplifyDedupRule} fire even after {@code FilterMergeRule} has
+   * merged the two filters into a single AND. Without the fix, the plan keeps the in-memory {@code
+   * ROW_NUMBER} window over a row-fetching scan and dedup pushdown is silently disabled.
+   */
+  @Test
+  public void testDedupAfterWherePushDown() throws IOException {
+    enabledOnlyWhenPushdownIsEnabled();
+    String result =
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account | where age > 25 | dedup gender");
+    assertTrue(
+        "Expected LogicalDedup in the plan (simplify rule should fire after FilterMergeRule):\n"
+            + result,
+        result.contains("LogicalDedup"));
+    assertFalse(
+        "Unexpected EnumerableWindow in the plan — dedup fell back to the in-memory ROW_NUMBER"
+            + " form, which means PPLSimplifyDedupRule did not fire:\n"
+            + result,
+        result.contains("EnumerableWindow"));
+  }
+
   public void testComplexDedup() throws IOException {
     enabledOnlyWhenPushdownIsEnabled();
     String expected = loadExpectedPlan("explain_dedup_complex1.yaml");