feat(gfql/cypher): NOT-pattern AST plumbing (#1031 slice 2 phase 2a) (#1233)

Top-level `WHERE NOT (pattern)` shapes (e.g. `WHERE NOT (n)-[:R]->()`)
now parse cleanly and lift into `WhereClause.predicates` as
`WherePatternPredicate(negated=True)` entries instead of tripping the
legacy "cannot yet be mixed with generic row expressions" E108.  This
is the AST half of slice 2; the runtime half (anti-semi-join lowering)
ships in a follow-up sub-PR.

Changes:

- `ast.py`: `WherePatternPredicate.negated: bool = False` field.
  Default keeps existing single-positive / multi-positive callers
  unchanged.
- `parser.py::_split_top_level_and_pattern_leaves`: add top-level
  `not(pattern_atom)` case that strips the NOT and emits the inner
  pattern as a negated leaf.  Returns a 4-tuple `(positive, negated,
  others, has_nested)`.  Patterns nested deeper (under OR/XOR or
  double-NOT) still trip the legacy E108 reject so slice 4 / De-Morgan-
  NOT compositions stay deferred.
- `parser.py::_build_where_with_pattern_lift`: accept both positive
  and negated leaves; emit one `WherePatternPredicate` per leaf with
  the matching `negated` flag.
- `ast_normalizer::_rewrite_where_pattern_predicates_to_matches`:
  partition into positive (appended MatchClause as before) vs negated
  (passes through to lowering).
- `lowering.py`: `WherePatternPredicate` checks now distinguish the
  two cases.  Negated raises a scoped "Cypher WHERE NOT (pattern)
  anti-semi-join lowering is not yet supported" pointing at the
  engine-half follow-up.  Positive case unchanged.

Tests:

- `test_parser.py`: replaces `test_parse_rejects_mixed_where_pattern_predicates_as_unsupported`
  parametrize with two parametrize blocks — OR cases (still rejected,
  slice 4) and NOT cases (now lift, new test
  `test_parse_lifts_top_level_not_pattern_to_negated_predicate`).
- `test_lowering.py`: splits the legacy `_failfast_rejects_unsupported_mixed_variable_length_where_pattern_predicates`
  test — drops NOT cases, adds `_failfast_rejects_negated_pattern_until_slice2_lowering`
  locking the precise new error message.

Verified:
- 1585/1585 GFQL tests pass.
- mypy clean on all 4 touched cypher modules.

Out of scope (engine half / phase 2b+):
- Anti-semi-join runtime lowering itself.  Path C (row-pipeline anti-join
  via Let + NotIn) recommended in `plans/1031-slices-2-3-4/findings/slice-2-scope.md`.
  Will land as a follow-up PR.
- Bound-aliases NOT-pattern (`MATCH (a)-[:R]->(b) WHERE NOT (b)-[:R]-(a)`).
  AST plumbing already handles it via the negated flag; runtime needs
  the same engine work.
- IC10 benchmark unblock.  Composes the engine work above with row-NOT
  (already supported via expr_tree).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: lmeyerov

CHANGELOG.md

@@ -30,6 +30,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **Polars support**: `polars.DataFrame` and `polars.LazyFrame` now work in `plot()`, `materialize_nodes()`, `get_degrees()`, `get_indegrees()`, `get_outdegrees()`, and `hypergraph()`. Polars is an optional dependency — no behavior change when not installed. Upload path uses efficient Arrow conversion (`to_arrow()` with schema-metadata stripping and memoization); compute/hypergraph paths coerce to pandas at entry. `LazyFrame` is materialized via `.collect()` at each boundary. Adds `test_polars.py` with 17 tests; skips gracefully when polars is absent (#1133).
 
 ### Internal
+- **GFQL / Cypher parser + ast_normalizer — NOT-pattern AST plumbing (#1031 slice 2 phase 2a)**: Top-level `WHERE NOT (pattern)` shapes (e.g. `WHERE NOT (n)-[:R]->()`) now parse cleanly and lift into `WhereClause.predicates` as `WherePatternPredicate(negated=True)` entries instead of tripping the legacy "cannot yet be mixed with generic row expressions" E108.  `_split_top_level_and_pattern_leaves` adds a top-level `not(pattern_atom)` case that strips the NOT and emits the inner pattern as a negated leaf; `_build_where_with_pattern_lift` accepts both positive and negated leaves and emits one `WherePatternPredicate` per leaf with the matching `negated` flag.  ast_normalizer's `_rewrite_where_pattern_predicates_to_matches` partitions into positive (rewrites to appended MatchClause as before) vs negated (passes through to lowering).  Lowering now distinguishes the two cases: positive `WherePatternPredicate` still raises "must be rewritten before lowering" (defensive — slice 3 already rewrites all positives in ast_normalizer); negated raises a scoped "Cypher WHERE NOT (pattern) anti-semi-join lowering is not yet supported" pointing the way for the engine half (path-C row-pipeline anti-join, see `plans/1031-slices-2-3-4/findings/slice-2-scope.md`).  Adds `test_parse_lifts_top_level_not_pattern_to_negated_predicate` and `test_string_cypher_failfast_rejects_negated_pattern_until_slice2_lowering`.  De Morgan compositions, OR-around-pattern, and double-NOT remain rejected at the lift step (slice 4 / future).  Phase 2a only — runtime (anti-semi-join lowering) ships in a follow-up sub-PR (#1031).
 - **GFQL / Cypher row-boolean residual matrix + guardrails (#1219 hardening)**: Locks compositional row-boolean WHERE shapes that #1217's Earley swap admitted but its initial test surface didn't cover.  Adds 11 native tests: nullable NOT/OR over a 4-row 3VL fixture (`NULL OR T = T`); N-ary OR (3 branches) + duplicate-branch companion isolating rightmost-drop associativity bugs; De Morgan equivalences (`NOT (A OR B)` ≡ `NOT A AND NOT B`; `NOT (A AND B)` ≡ `NOT A OR NOT B`) parametrized to assert both per-form expected rows AND the form-equivalence; double negation; XOR symmetric difference + XOR with NULL preserving 3VL; mixed-string-numeric AND inside OR exercising `_StringAllowingComparisonMixin` GT path; unit test locking `boolean_expr_to_text(BooleanExpr(op="pattern", ...))` round-trip for the (currently unreachable) defensive branch.  Three docstring guardrails: `expr_split.split_top_level_and` documents AND-only intent + the `_combine_conjuncts` AND-recombine mechanism that makes a hypothetical `split_top_level_or` silently incorrect; `predicate_pushdown._split_conjuncts` mirrored guard naming the failure mode; `_boolean_expr_text.boolean_expr_to_text` explicit `op == "pattern"` branch with both unreachability paths documented.  No production-code behavior change.  Closes the residual-frontier portion of #1219; deeper compositional shapes beyond current fixtures remain tracked under that issue (#1219, #1227).
 - **GFQL / Cypher parser + ast_normalizer — multi-positive WHERE pattern predicates (#1031 slice 3)**: AND-joined positive WHERE pattern predicates (`WHERE (n)-[:R]->() AND (n)-[:T]->()`) now lift into structured `WhereClause.predicates` as N `WherePatternPredicate` entries.  The ast_normalizer packs them into a single appended `MatchClause` whose `patterns: Tuple[Tuple[PatternElement, ...], ...]` carries one tuple per predicate (multi-pattern cartesian within MATCH), preserving the lowering invariant that only the FINAL match is connected — pre-binding seeds remain node-only.  Per-predicate validation (must include a relationship; cannot introduce new aliases) runs independently before the lift.  Removes the legacy `len(pattern_leaves) > 1` gate in `parser.py::_build_where_with_pattern_lift` and the corresponding gate in `ast_normalizer._rewrite_where_pattern_predicates_to_matches`.  Refactors `pattern_atom` to split the greedy `WHERE_PATTERN` lexer token (which gobbles `pattern AND pattern AND ...` chains as a single match) back into individual pattern-item texts via `_WHERE_PATTERN_ITEM_RE.finditer` and emit one `BooleanExpr(op="pattern")` per item, joined by an AND-tree via `_rebuild_and_tree`.  Adds `test_gfql_executes_multi_positive_where_pattern_predicates_as_intersected_seed` and updates the legacy rejection test to assert the new lift + compile shape.  Closes #1031 slice 3 (#1031).
 - **GFQL / Cypher lowering**: Connected `MATCH + OPTIONAL MATCH` compilation now supports row-boolean `WHERE` expressions (`OR`/`NOT`/`XOR` and mixed row predicates) by carrying non-lowerable expressions into post-binding `where_rows(...)` filters for base and optional arms, preserving null-extension behavior while expanding supported disjunction shapes (#1219, #1224).

graphistry/compute/gfql/cypher/ast.py

@@ -140,6 +140,10 @@ class WherePredicate:
 class WherePatternPredicate:
     pattern: Tuple[PatternElement, ...]
     span: SourceSpan
+    # #1031 slice 2: True when lifted from a `WHERE NOT (...)` shape, signaling
+    # anti-semi-join lowering instead of intersect-MATCH.  Default False keeps
+    # all existing single-positive / multi-positive callers unchanged.
+    negated: bool = False
 
 
 WhereTerm = Union[WherePredicate, WherePatternPredicate]

graphistry/compute/gfql/cypher/ast_normalizer.py

@@ -476,6 +476,13 @@ def _rewrite_where_pattern_predicates_to_matches(query: CypherQuery) -> CypherQu
     pattern_preds = [predicate for predicate in query.where.predicates if isinstance(predicate, WherePatternPredicate)]
     if not pattern_preds:
         return query
+    # Slice 2 of #1031: WherePatternPredicate.negated stays in place — the
+    # ast_normalizer rewriter only handles positive predicates (which compile
+    # to MatchClause append).  Negated predicates are passed through to
+    # lowering, which emits an anti-semi-join row-pipeline step.
+    positive_preds = [p for p in pattern_preds if not p.negated]
+    if not positive_preds:
+        return query
     # Slice 3 of #1031: support N positive pattern predicates by emitting one
     # appended ``MatchClause`` per predicate.  Each predicate is independently
     # validated (must include a relationship; cannot introduce new aliases).
@@ -486,12 +493,13 @@ def _rewrite_where_pattern_predicates_to_matches(query: CypherQuery) -> CypherQu
         for element in pattern
         if getattr(element, "variable", None) is not None
     }
-    # Validate every pattern; collect into a single appended MatchClause whose
-    # ``patterns`` tuple holds N patterns (multi-positive WHERE pattern via
-    # cartesian-style multi-pattern MATCH; #1031 slice 3).  Packing into one
-    # MatchClause (rather than N appended MatchClauses) preserves the lowering
-    # invariant that only the FINAL match is connected — seeds remain node-only.
-    for pred in pattern_preds:
+    # Validate every positive pattern; collect into a single appended
+    # MatchClause whose ``patterns`` tuple holds N patterns (multi-positive
+    # WHERE pattern via cartesian-style multi-pattern MATCH; #1031 slice 3).
+    # Packing into one MatchClause (rather than N appended MatchClauses)
+    # preserves the lowering invariant that only the FINAL match is
+    # connected — seeds remain node-only.
+    for pred in positive_preds:
         if len(pred.pattern) < 3:
             raise _unsupported(
                 "Cypher WHERE pattern predicates must include a relationship",
@@ -514,16 +522,21 @@ def _rewrite_where_pattern_predicates_to_matches(query: CypherQuery) -> CypherQu
                 column=pred.span.column,
             )
 
-    first = pattern_preds[0]
+    first = positive_preds[0]
     extra_match = MatchClause(
-        patterns=tuple(pred.pattern for pred in pattern_preds),
+        patterns=tuple(pred.pattern for pred in positive_preds),
         span=first.span,
         optional=False,
-        pattern_aliases=tuple(None for _ in pattern_preds),
-        pattern_alias_kinds=tuple("pattern" for _ in pattern_preds),
+        pattern_aliases=tuple(None for _ in positive_preds),
+        pattern_alias_kinds=tuple("pattern" for _ in positive_preds),
     )
 
-    remaining = tuple(predicate for predicate in query.where.predicates if not isinstance(predicate, WherePatternPredicate))
+    # Keep negated WherePatternPredicates in `remaining` so lowering sees them
+    # for anti-semi-join emission (#1031 slice 2).
+    remaining = tuple(
+        predicate for predicate in query.where.predicates
+        if (not isinstance(predicate, WherePatternPredicate)) or predicate.negated
+    )
     remaining_where = None
     if remaining or query.where.expr_tree is not None:
         remaining_where = WhereClause(

graphistry/compute/gfql/cypher/lowering.py

@@ -6389,6 +6389,18 @@ def lower_match_query(
                 )
         for predicate in query.where.predicates:
             if isinstance(predicate, WherePatternPredicate):
+                if predicate.negated:
+                    # #1031 slice 2: NOT-pattern parses + survives ast_normalizer
+                    # but lowering for anti-semi-join is not yet implemented.
+                    # See plans/1031-slices-2-3-4/findings/slice-2-scope.md
+                    # for the path-C (row-pipeline anti-join) approach.
+                    raise _unsupported(
+                        "Cypher WHERE NOT (pattern) anti-semi-join lowering is not yet supported",
+                        field="where",
+                        value=None,
+                        line=predicate.span.line,
+                        column=predicate.span.column,
+                    )
                 raise _unsupported(
                     "Cypher WHERE pattern predicates must be rewritten before lowering",
                     field="where",
@@ -8198,6 +8210,14 @@ def _apply_where_to_ops(
                 row_expr_filters.append(rewritten)
     for predicate in where.predicates:
         if isinstance(predicate, WherePatternPredicate):
+            if predicate.negated:
+                raise _unsupported(
+                    "Cypher WHERE NOT (pattern) anti-semi-join lowering is not yet supported",
+                    field="where",
+                    value=None,
+                    line=predicate.span.line,
+                    column=predicate.span.column,
+                )
             raise _unsupported(
                 "Cypher WHERE pattern predicates must be rewritten before lowering",
                 field="where",

graphistry/compute/gfql/cypher/parser.py

@@ -347,17 +347,37 @@ def _lift_label_only_and_spine(
 
 def _split_top_level_and_pattern_leaves(
     expr: BooleanExpr,
-) -> Tuple[List[BooleanExpr], List[BooleanExpr], bool]:
-    """Split *expr* at top-level AND boundaries into (patterns, others, has_nested_pattern)."""
+) -> Tuple[List[BooleanExpr], List[BooleanExpr], List[BooleanExpr], bool]:
+    """Split *expr* at top-level AND boundaries.
+
+    Returns ``(positive_patterns, negated_patterns, others, has_nested_pattern)``.
+
+    - ``positive_patterns``: leaf ``BooleanExpr(op="pattern")`` nodes at top-level
+      AND positions.  Lifted to ``WherePatternPredicate(negated=False)``.
+    - ``negated_patterns``: leaf ``BooleanExpr(op="pattern")`` nodes wrapped in a
+      single top-level ``not`` (i.e. ``NOT (n)-[:R]->()``).  Lifted to
+      ``WherePatternPredicate(negated=True)`` for #1031 slice 2 anti-semi-join
+      lowering.  Returns the inner ``pattern`` leaf (the ``not`` is consumed).
+    - ``others``: non-pattern conjuncts that should remain in ``expr_tree``.
+    - ``has_nested_pattern``: True when a pattern atom appears in a deeper
+      non-AND/non-direct-NOT context (e.g. ``OR`` with a pattern leaf, or
+      ``NOT (and-tree-of-patterns)``).  Triggers the legacy E108 reject so
+      slice 4 / De-Morgan-NOT compositions stay deferred.
+    """
     if expr.op == "and":
         if expr.left is None or expr.right is None:
-            return [], [expr], False
-        left_pat, left_other, left_bad = _split_top_level_and_pattern_leaves(expr.left)
-        right_pat, right_other, right_bad = _split_top_level_and_pattern_leaves(expr.right)
-        return left_pat + right_pat, left_other + right_other, left_bad or right_bad
+            return [], [], [expr], False
+        l_pos, l_neg, l_oth, l_bad = _split_top_level_and_pattern_leaves(expr.left)
+        r_pos, r_neg, r_oth, r_bad = _split_top_level_and_pattern_leaves(expr.right)
+        return l_pos + r_pos, l_neg + r_neg, l_oth + r_oth, l_bad or r_bad
     if expr.op == "pattern":
-        return [expr], [], False
-    return [], [expr], _has_pattern_descendant(expr)
+        return [expr], [], [], False
+    if expr.op == "not" and expr.left is not None and expr.left.op == "pattern":
+        # `WHERE NOT (pattern)` — slice 2 anti-semi-join target.  Strip the NOT
+        # and emit the inner pattern leaf as a negated pattern.  No nested
+        # pattern; rest of the tree continues structural traversal as usual.
+        return [], [expr.left], [], False
+    return [], [], [expr], _has_pattern_descendant(expr)
 
 
 def _has_pattern_descendant(expr: BooleanExpr) -> bool:
@@ -394,6 +414,7 @@ def _rebuild_and_tree(conjuncts: List[BooleanExpr]) -> Optional[BooleanExpr]:
 def _build_where_with_pattern_lift(
     *,
     pattern_leaves: List[BooleanExpr],
+    negated_pattern_leaves: List[BooleanExpr],
     other_conjuncts: List[BooleanExpr],
     nested_pattern: bool,
     expr_text: str,
@@ -410,13 +431,18 @@ def _build_where_with_pattern_lift(
             column=span.column,
             language="cypher",
         )
-    # Multi-positive (slice 3 of #1031): emit one ``WherePatternPredicate`` per
-    # leaf; downstream ``_rewrite_where_pattern_predicates_to_matches`` lifts
-    # each into its own appended ``MatchClause``.
+    # Slice 3 (#1031): N positive patterns each become a WherePatternPredicate
+    # (negated=False).  Slice 2 (#1031): N NOT-patterns each become a
+    # WherePatternPredicate (negated=True) for downstream anti-semi-join
+    # lowering.  Both groups travel together in WhereClause.predicates;
+    # ast_normalizer dispatches by the negated flag.
     pattern_preds: List[WherePatternPredicate] = []
     for leaf in pattern_leaves:
         assert leaf.pattern is not None, "pattern_atom invariant: pattern payload always set"
-        pattern_preds.append(WherePatternPredicate(pattern=leaf.pattern, span=leaf.span))
+        pattern_preds.append(WherePatternPredicate(pattern=leaf.pattern, span=leaf.span, negated=False))
+    for leaf in negated_pattern_leaves:
+        assert leaf.pattern is not None, "pattern_atom invariant: pattern payload always set"
+        pattern_preds.append(WherePatternPredicate(pattern=leaf.pattern, span=leaf.span, negated=True))
     new_expr_tree = _rebuild_and_tree(other_conjuncts)
     if new_expr_tree is None:
         return WhereClause(predicates=tuple(pattern_preds), expr_tree=None, span=span)
@@ -1199,14 +1225,15 @@ def generic_where_clause(self, meta: Any, items: Sequence[Any]) -> WhereClause:
             # top-level AND positions; extract them as
             # ``WherePatternPredicate`` entries so the existing AST-
             # normalizer step (``_rewrite_where_pattern_predicates_to_matches``)
-            # can lift them to a separate ``MatchClause`` later.  Pattern
-            # leaves nested under non-AND ops (NOT/OR/XOR) and multiple
-            # positive patterns are rejected here with shape-specific
-            # E108 errors — slice 2/3/4 territory.
-            pattern_leaves, other_conjuncts, nested_pattern = _split_top_level_and_pattern_leaves(expr_tree)
-            if pattern_leaves or nested_pattern:
+            # can lift them.  Slice 2 (#1031): top-level ``NOT (pattern)``
+            # leaves are also lifted, marked ``negated=True`` for anti-semi-
+            # join lowering.  Patterns nested deeper (under OR/XOR or
+            # double-NOT) trip the legacy E108 reject.
+            pos_leaves, neg_leaves, other_conjuncts, nested_pattern = _split_top_level_and_pattern_leaves(expr_tree)
+            if pos_leaves or neg_leaves or nested_pattern:
                 return _build_where_with_pattern_lift(
-                    pattern_leaves=pattern_leaves,
+                    pattern_leaves=pos_leaves,
+                    negated_pattern_leaves=neg_leaves,
                     other_conjuncts=other_conjuncts,
                     nested_pattern=nested_pattern,
                     expr_text=expr_text,

graphistry/tests/compute/gfql/cypher/test_lowering.py

@@ -5145,7 +5145,6 @@ def test_string_cypher_executes_where_pattern_predicate_and_expr_mix(
     "query",
     [
         "MATCH (n) WHERE (n)-[:R*]->() OR n.id = 'z' RETURN n",
-        "MATCH (n) WHERE NOT (n)-[:R*]->() RETURN n",
     ],
 )
 def test_string_cypher_failfast_rejects_unsupported_mixed_variable_length_where_pattern_predicates(query: str) -> None:
@@ -5158,6 +5157,27 @@ def test_string_cypher_failfast_rejects_unsupported_mixed_variable_length_where_
     assert "mixed with generic row expressions" in exc_info.value.message
 
 
+@pytest.mark.parametrize(
+    "query",
+    [
+        "MATCH (n) WHERE NOT (n)-[:R*]->() RETURN n",
+        "MATCH (n) WHERE NOT (n)-[:R]->() RETURN n",
+    ],
+)
+def test_string_cypher_failfast_rejects_negated_pattern_until_slice2_lowering(query: str) -> None:
+    # #1031 slice 2 plumbing: parser lifts NOT-pattern to
+    # ``WherePatternPredicate(negated=True)``; lowering raises a scoped error
+    # until the anti-semi-join lowering lands.  Locks the precise message
+    # so future engine work knows where to plug in.
+    graph = _mk_empty_graph()
+
+    with pytest.raises(GFQLValidationError) as exc_info:
+        graph.gfql(query)
+
+    assert exc_info.value.code == ErrorCode.E108
+    assert "anti-semi-join" in exc_info.value.message
+
+
 def test_string_cypher_failfast_rejects_multi_alias_return_star_projection() -> None:
     graph = _mk_empty_graph()