feat(gfql/cypher): execute WHERE NOT (pattern) via anti-semi row filtering (#1031 slice 2 phase 2b) (#1238)

lmeyerov · web-flow · commit bd420d6b1add · 2026-04-29T19:26:29.000-07:00
* feat(gfql/cypher): execute WHERE NOT pattern via anti-semi row filtering (#1031 #1235) * test(gfql/cypher): amplify OPTIONAL MATCH NOT-pattern coverage (#1235)
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -30,6 +30,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **Polars support**: `polars.DataFrame` and `polars.LazyFrame` now work in `plot()`, `materialize_nodes()`, `get_degrees()`, `get_indegrees()`, `get_outdegrees()`, and `hypergraph()`. Polars is an optional dependency — no behavior change when not installed. Upload path uses efficient Arrow conversion (`to_arrow()` with schema-metadata stripping and memoization); compute/hypergraph paths coerce to pandas at entry. `LazyFrame` is materialized via `.collect()` at each boundary. Adds `test_polars.py` with 17 tests; skips gracefully when polars is absent (#1133).
 
 ### Internal
+- **GFQL / Cypher lowering — WHERE NOT (pattern) anti-semi execution (#1031 slice 2 phase 2b)**: `WherePatternPredicate(negated=True)` no longer hard-fails at lowering. The compiler now emits a row pre-filter call (`anti_semi_apply`) for negated pattern predicates in both general MATCH lowering and connected OPTIONAL MATCH clause lowering, with per-predicate validation (must include relationship, must not introduce new aliases, must share bound aliases). Added `anti_semi_apply` row-pipeline runtime operation + call-safelist entry, plus row-table base-graph context preservation needed for correlated bindings execution. New tests cover compile shape (`row_pre_filters` emission) and runtime behavior for `MATCH (n) WHERE NOT (n)-[:R]->()` (including mixed row-expression + NOT-pattern filtering), plus connected `OPTIONAL MATCH ... WHERE NOT (pattern)` filtering/null-fill semantics. Full `cypher/test_lowering.py` suite passes (756 passed / 66 skipped) and touched-module mypy is clean.
 - **GFQL / Cypher parser + ast_normalizer — NOT-pattern AST plumbing (#1031 slice 2 phase 2a)**: Top-level `WHERE NOT (pattern)` shapes (e.g. `WHERE NOT (n)-[:R]->()`) now parse cleanly and lift into `WhereClause.predicates` as `WherePatternPredicate(negated=True)` entries instead of tripping the legacy "cannot yet be mixed with generic row expressions" E108.  `_split_top_level_and_pattern_leaves` adds a top-level `not(pattern_atom)` case that strips the NOT and emits the inner pattern as a negated leaf; `_build_where_with_pattern_lift` accepts both positive and negated leaves and emits one `WherePatternPredicate` per leaf with the matching `negated` flag.  ast_normalizer's `_rewrite_where_pattern_predicates_to_matches` partitions into positive (rewrites to appended MatchClause as before) vs negated (passes through to lowering).  Lowering now distinguishes the two cases: positive `WherePatternPredicate` still raises "must be rewritten before lowering" (defensive — slice 3 already rewrites all positives in ast_normalizer); negated raises a scoped "Cypher WHERE NOT (pattern) anti-semi-join lowering is not yet supported" pointing the way for the engine half (path-C row-pipeline anti-join, see `plans/1031-slices-2-3-4/findings/slice-2-scope.md`).  Adds `test_parse_lifts_top_level_not_pattern_to_negated_predicate` and `test_string_cypher_failfast_rejects_negated_pattern_until_slice2_lowering`.  De Morgan compositions, OR-around-pattern, and double-NOT remain rejected at the lift step (slice 4 / future).  Phase 2a only — runtime (anti-semi-join lowering) ships in a follow-up sub-PR (#1031).
 - **GFQL / Cypher row-boolean residual matrix + guardrails (#1219 hardening)**: Locks compositional row-boolean WHERE shapes that #1217's Earley swap admitted but its initial test surface didn't cover.  Adds 11 native tests: nullable NOT/OR over a 4-row 3VL fixture (`NULL OR T = T`); N-ary OR (3 branches) + duplicate-branch companion isolating rightmost-drop associativity bugs; De Morgan equivalences (`NOT (A OR B)` ≡ `NOT A AND NOT B`; `NOT (A AND B)` ≡ `NOT A OR NOT B`) parametrized to assert both per-form expected rows AND the form-equivalence; double negation; XOR symmetric difference + XOR with NULL preserving 3VL; mixed-string-numeric AND inside OR exercising `_StringAllowingComparisonMixin` GT path; unit test locking `boolean_expr_to_text(BooleanExpr(op="pattern", ...))` round-trip for the (currently unreachable) defensive branch.  Three docstring guardrails: `expr_split.split_top_level_and` documents AND-only intent + the `_combine_conjuncts` AND-recombine mechanism that makes a hypothetical `split_top_level_or` silently incorrect; `predicate_pushdown._split_conjuncts` mirrored guard naming the failure mode; `_boolean_expr_text.boolean_expr_to_text` explicit `op == "pattern"` branch with both unreachability paths documented.  No production-code behavior change.  Closes the residual-frontier portion of #1219; deeper compositional shapes beyond current fixtures remain tracked under that issue (#1219, #1227).
 - **GFQL / Cypher parser + ast_normalizer — multi-positive WHERE pattern predicates (#1031 slice 3)**: AND-joined positive WHERE pattern predicates (`WHERE (n)-[:R]->() AND (n)-[:T]->()`) now lift into structured `WhereClause.predicates` as N `WherePatternPredicate` entries.  The ast_normalizer packs them into a single appended `MatchClause` whose `patterns: Tuple[Tuple[PatternElement, ...], ...]` carries one tuple per predicate (multi-pattern cartesian within MATCH), preserving the lowering invariant that only the FINAL match is connected — pre-binding seeds remain node-only.  Per-predicate validation (must include a relationship; cannot introduce new aliases) runs independently before the lift.  Removes the legacy `len(pattern_leaves) > 1` gate in `parser.py::_build_where_with_pattern_lift` and the corresponding gate in `ast_normalizer._rewrite_where_pattern_predicates_to_matches`.  Refactors `pattern_atom` to split the greedy `WHERE_PATTERN` lexer token (which gobbles `pattern AND pattern AND ...` chains as a single match) back into individual pattern-item texts via `_WHERE_PATTERN_ITEM_RE.finditer` and emit one `BooleanExpr(op="pattern")` per item, joined by an AND-tree via `_rebuild_and_tree`.  Adds `test_gfql_executes_multi_positive_where_pattern_predicates_as_intersected_seed` and updates the legacy rejection test to assert the new lift + compile shape.  Closes #1031 slice 3 (#1031).
diff --git a/graphistry/compute/ast.py b/graphistry/compute/ast.py
@@ -1745,6 +1745,25 @@ def where_rows(
     return ASTCall("where_rows", params)
 
 
+def anti_semi_apply(
+    *,
+    binding_ops: List[Dict[str, Any]],
+    join_aliases: Sequence[str],
+) -> ASTCall:
+    """Filter active rows by removing rows matching a correlated pattern.
+
+    ``binding_ops`` encodes the pattern to evaluate as bindings rows.
+    ``join_aliases`` names shared aliases used as anti-join keys.
+    """
+    return ASTCall(
+        "anti_semi_apply",
+        {
+            "binding_ops": binding_ops,
+            "join_aliases": list(join_aliases),
+        },
+    )
+
+
 def order_by(keys: Iterable[Tuple[Any, str]]) -> ASTCall:
     """Create an ORDER BY operation for GFQL row pipelines."""
     return ASTCall("order_by", {"keys": list(keys)})
diff --git a/graphistry/compute/gfql/call/validation.py b/graphistry/compute/gfql/call/validation.py
@@ -260,6 +260,17 @@ def _group_by_requires_node_cols(params: Dict[str, object]) -> List[str]:
         schema_effects=_schema_effects(requires_node_cols=_where_rows_requires_node_cols),
     ),
 
+    'anti_semi_apply': _method_entry(
+        allowed_params={'binding_ops', 'join_aliases'},
+        required_params={'binding_ops', 'join_aliases'},
+        param_validators={
+            'binding_ops': is_list_of_dicts,
+            'join_aliases': is_non_empty_list_of_strings,
+        },
+        description='Filter active rows by anti-semi joining against correlated binding rows',
+        schema_effects=NO_SCHEMA_EFFECTS,
+    ),
+
     'order_by': _method_entry(
         allowed_params={'keys'},
         required_params={'keys'},
diff --git a/graphistry/compute/gfql/cypher/lowering.py b/graphistry/compute/gfql/cypher/lowering.py
@@ -12,6 +12,7 @@
     ASTEdge,
     ASTObject,
     ASTNode,
+    anti_semi_apply,
     distinct,
     drop_cols,
     e_forward,
@@ -125,6 +126,7 @@ class LoweredCypherMatch:
     query: List[ASTObject]
     where: List[WhereComparison]
     row_where: Optional[ExpressionText] = None
+    row_pre_filters: Tuple[ASTCall, ...] = ()
 
 
 _CYPHER_INT64_MIN = -(2**63)
@@ -4466,6 +4468,9 @@ def _append_match_row_where(
     allowed_match_aliases: Optional[AbstractSet[str]],
     params: Optional[Mapping[str, Any]],
 ) -> None:
+    if lowered.row_pre_filters:
+        row_steps.extend(lowered.row_pre_filters)
+
     expr = lowered.row_where
     if expr is None:
         return
@@ -4544,7 +4549,7 @@ def _lower_projection_chain(
         if plan.all_source_aliases is not None
         else binding_row_aliases
     )
-    if plan.all_source_aliases is not None or binding_row_aliases:
+    if plan.all_source_aliases is not None or binding_row_aliases or lowered.row_pre_filters:
         row_steps: List[ASTObject] = [rows(binding_ops=serialize_binding_ops(lowered.query))]
     else:
         row_steps = [rows(table=plan.table, source=plan.source_alias)]
@@ -4573,7 +4578,7 @@ def _lower_projection_chain(
             )
         )
     _append_page_ops(row_steps, query=query, params=params)
-    if binding_row_aliases:
+    if binding_row_aliases or lowered.row_pre_filters:
         return row_steps
     return lowered.query + row_steps
 
@@ -4635,7 +4640,7 @@ def _build_initial_row_scope(
     if active_match_alias is None:
         row_steps: List[ASTObject] = [rows(table="nodes")]
         scope_mode: Literal["match_alias", "row_columns"] = "row_columns"
-    elif binding_row_aliases:
+    elif binding_row_aliases or lowered.row_pre_filters:
         row_steps = [rows(binding_ops=serialize_binding_ops(lowered.query))]
         scope_mode = "match_alias"
     else:
@@ -6332,8 +6337,82 @@ def _check_projection_clause(clause: ReturnClause) -> None:
                     item.expression.text,
                     field="order_by",
                     line=item.span.line,
-                    column=item.span.column,
-                )
+                        column=item.span.column,
+                    )
+
+
+def _predicate_pattern_aliases(predicate: WherePatternPredicate) -> List[str]:
+    aliases: List[str] = []
+    seen: Set[str] = set()
+    for element in predicate.pattern:
+        alias = getattr(element, "variable", None)
+        if alias is None:
+            continue
+        alias_name = cast(str, alias)
+        if alias_name in seen:
+            continue
+        seen.add(alias_name)
+        aliases.append(alias_name)
+    return aliases
+
+
+def _lower_negated_pattern_predicate_to_row_filter(
+    predicate: WherePatternPredicate,
+    *,
+    alias_targets: Mapping[str, ASTObject],
+    params: Optional[Mapping[str, Any]],
+) -> ASTCall:
+    if len(predicate.pattern) < 3:
+        raise _unsupported(
+            "Cypher WHERE pattern predicates must include a relationship",
+            field="where",
+            value=None,
+            line=predicate.span.line,
+            column=predicate.span.column,
+        )
+
+    predicate_aliases = _predicate_pattern_aliases(predicate)
+    if not predicate_aliases:
+        raise _unsupported(
+            "Cypher WHERE NOT (pattern) currently requires at least one shared bound alias",
+            field="where",
+            value=None,
+            line=predicate.span.line,
+            column=predicate.span.column,
+        )
+
+    introduced_aliases = sorted(alias for alias in predicate_aliases if alias not in alias_targets)
+    if introduced_aliases:
+        raise _unsupported(
+            "Cypher WHERE pattern predicates cannot introduce new aliases in this phase",
+            field="where",
+            value=introduced_aliases,
+            line=predicate.span.line,
+            column=predicate.span.column,
+        )
+
+    shared_aliases = [alias for alias in predicate_aliases if alias in alias_targets]
+    if not shared_aliases:
+        raise _unsupported(
+            "Cypher WHERE NOT (pattern) currently requires at least one shared bound alias",
+            field="where",
+            value=predicate_aliases,
+            line=predicate.span.line,
+            column=predicate.span.column,
+        )
+
+    pattern_clause = MatchClause(
+        patterns=(predicate.pattern,),
+        span=predicate.span,
+        optional=False,
+        pattern_aliases=(None,),
+        pattern_alias_kinds=("pattern",),
+    )
+    pattern_ops = lower_match_clause(pattern_clause, params=params)
+    return anti_semi_apply(
+        binding_ops=serialize_binding_ops(pattern_ops),
+        join_aliases=shared_aliases,
+    )
 
 
 def lower_match_query(
@@ -6365,6 +6444,7 @@ def lower_match_query(
         params=params,
     )
     where_out.extend(dynamic_where_out)
+    row_pre_filters: List[ASTCall] = []
 
     row_where: Optional[ExpressionText] = None
     row_where_predicates: List[str] = list(dynamic_row_where_predicates)
@@ -6390,17 +6470,14 @@ def lower_match_query(
         for predicate in query.where.predicates:
             if isinstance(predicate, WherePatternPredicate):
                 if predicate.negated:
-                    # #1031 slice 2: NOT-pattern parses + survives ast_normalizer
-                    # but lowering for anti-semi-join is not yet implemented.
-                    # See plans/1031-slices-2-3-4/findings/slice-2-scope.md
-                    # for the path-C (row-pipeline anti-join) approach.
-                    raise _unsupported(
-                        "Cypher WHERE NOT (pattern) anti-semi-join lowering is not yet supported",
-                        field="where",
-                        value=None,
-                        line=predicate.span.line,
-                        column=predicate.span.column,
+                    row_pre_filters.append(
+                        _lower_negated_pattern_predicate_to_row_filter(
+                            predicate,
+                            alias_targets=alias_targets,
+                            params=params,
+                        )
                     )
+                    continue
                 raise _unsupported(
                     "Cypher WHERE pattern predicates must be rewritten before lowering",
                     field="where",
@@ -6439,7 +6516,12 @@ def lower_match_query(
             span=query.where.span if query.where is not None else merged_match.span,
         )
 
-    return LoweredCypherMatch(query=ops, where=where_out, row_where=row_where)
+    return LoweredCypherMatch(
+        query=ops,
+        where=where_out,
+        row_where=row_where,
+        row_pre_filters=tuple(row_pre_filters),
+    )
 
 
 def _fresh_temp_name(existing: Set[str], prefix: str) -> str:
@@ -8174,7 +8256,7 @@ def _apply_where_to_ops(
     alias_targets: Dict[str, ASTObject],
     *,
     params: Optional[Mapping[str, Any]],
-) -> Tuple[List[WhereComparison], List[ExpressionText]]:
+) -> Tuple[List[WhereComparison], List[ExpressionText], List[ASTCall]]:
     """Apply a WHERE clause's predicates to already-lowered ops.
 
     Label predicates mutate the ASTNode filter in *alias_targets* (in-place).
@@ -8185,8 +8267,9 @@ def _apply_where_to_ops(
     """
     where_out: List[WhereComparison] = []
     row_expr_filters: List[ExpressionText] = []
+    row_pre_filters: List[ASTCall] = []
     if where is None:
-        return where_out, row_expr_filters
+        return where_out, row_expr_filters, row_pre_filters
     where_expr = _where_clause_expr_text(where)
     if where_expr is not None:
         type_where = _extract_relationship_type_where(
@@ -8211,13 +8294,14 @@ def _apply_where_to_ops(
     for predicate in where.predicates:
         if isinstance(predicate, WherePatternPredicate):
             if predicate.negated:
-                raise _unsupported(
-                    "Cypher WHERE NOT (pattern) anti-semi-join lowering is not yet supported",
-                    field="where",
-                    value=None,
-                    line=predicate.span.line,
-                    column=predicate.span.column,
+                row_pre_filters.append(
+                    _lower_negated_pattern_predicate_to_row_filter(
+                        predicate,
+                        alias_targets=alias_targets,
+                        params=params,
+                    )
                 )
+                continue
             raise _unsupported(
                 "Cypher WHERE pattern predicates must be rewritten before lowering",
                 field="where",
@@ -8256,7 +8340,7 @@ def _apply_where_to_ops(
             right=cast(Optional[CypherLiteral], predicate.right),
             params=params,
         )
-    return where_out, row_expr_filters
+    return where_out, row_expr_filters, row_pre_filters
 
 
 def _compile_connected_optional_match(
@@ -8275,8 +8359,13 @@ def _compile_connected_optional_match(
     base_ops = lower_match_clause(base_clause, params=params)
     base_alias_targets = _alias_target(base_ops)
     base_aliases = _match_clause_aliases(base_clause)
-    base_where, base_row_expr_filters = _apply_where_to_ops(base_clause.where, base_alias_targets, params=params)
+    base_where, base_row_expr_filters, base_row_pre_filters = _apply_where_to_ops(
+        base_clause.where,
+        base_alias_targets,
+        params=params,
+    )
     base_chain_ops: List[ASTObject] = list(base_ops)
+    base_chain_ops.extend(base_row_pre_filters)
     for expr in base_row_expr_filters:
         base_chain_ops.append(
             where_rows(
@@ -8326,12 +8415,13 @@ def _compile_connected_optional_match(
                 column=opt_clause.span.column,
             )
 
-        opt_where, opt_row_expr_filters = _apply_where_to_ops(
+        opt_where, opt_row_expr_filters, opt_row_pre_filters = _apply_where_to_ops(
             opt_clause.where,
             opt_alias_targets,
             params=params,
         )
         opt_chain_ops: List[ASTObject] = list(opt_ops)
+        opt_chain_ops.extend(opt_row_pre_filters)
         for expr in opt_row_expr_filters:
             opt_chain_ops.append(
                 where_rows(
diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py
diff --git a/graphistry/tests/compute/gfql/cypher/test_lowering.py b/graphistry/tests/compute/gfql/cypher/test_lowering.py
