Merge pull request #1247 from graphistry/issue-1071-with-multi-alias-carry

feat(cypher): support multi-alias carry through WITH before MATCH re-entry (#1071)

Author: lmeyerov

CHANGELOG.md

@@ -30,6 +30,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **Polars support**: `polars.DataFrame` and `polars.LazyFrame` now work in `plot()`, `materialize_nodes()`, `get_degrees()`, `get_indegrees()`, `get_outdegrees()`, and `hypergraph()`. Polars is an optional dependency — no behavior change when not installed. Upload path uses efficient Arrow conversion (`to_arrow()` with schema-metadata stripping and memoization); compute/hypergraph paths coerce to pandas at entry. `LazyFrame` is materialized via `.collect()` at each boundary. Adds `test_polars.py` with 17 tests; skips gracefully when polars is absent (#1133).
 
 ### Internal
+- **GFQL / Cypher lowering — multi-alias carry through `WITH` before MATCH re-entry (#1071)**: Lifted the residual constraint that `WITH` before a MATCH re-entry must project exactly one whole-row node alias. The local Cypher compiler now accepts patterns like `MATCH (p)-[:KNOWS]-(friend) WITH p, friend MATCH (friend)-[:IS_LOCATED_IN]->(c) RETURN p.firstName, c.name` (LDBC SNB IC1 shape). Implementation pre-rewrites the prefix `WITH` in `_compile_bounded_reentry_query`: the trailing-MATCH primary alias is preserved as the sole whole-row carry; secondary aliases are demoted to scalar property carries via synthesized `S.X AS __cypher_reentry_<S>_<X>__` items, with downstream `S.X` references rewritten to bare hidden identifiers that compose with the existing single-alias carried-scalar machinery (#1047 / #1068). Returning a secondary alias as a whole-row entity (`RETURN s`) and re-binding a secondary alias as a node variable in the trailing MATCH remain unsupported with precise errors. Two prior failfast tests retargeted to assert correct multi-alias behavior; new tests cover IC1-shape, three-alias carry, and the secondary-whole-row-return rejection.
 - **GFQL / Cypher lowering — OR/XOR around pattern predicates (#1236, #1031 slice 4)**: Local Cypher `WHERE` boolean trees now support disjunctive compositions that mix pattern-existence leaves with row expressions (for example `pattern OR expr`, `pattern XOR expr`). Lowering rewrites pattern leaves into correlated `semi_apply_mark(...)` row pre-filters that emit boolean marker columns, then evaluates the rewritten boolean expression in `where_rows(...)`. Added direct row-pipeline/safelist coverage for `semi_apply_mark` plus parser/lowering runtime regression tests for OR/XOR-around-pattern query shapes.
 - **GFQL / Cypher residual cleanup (#1226, #1219)**: Retired `graphistry/compute/gfql/expr_split.py` by migrating top-level-AND splitting to `predicate_pushdown.py` (`split_top_level_and_conjuncts`) and updating conformance coverage to target the migrated splitter path. Fixed primitive literal atom fallback text in parser boolean-tree construction so `atom_text` now preserves Cypher keyword casing (`true`/`false`/`null`) instead of Python casing. Tightened optional-arm pushdown null-safety guardrails by treating OR-compound predicates as conservatively null-rejecting (disjunct-level alias analysis remains out of scope), with new regression coverage for mixed-alias `IS NOT NULL OR ...` forms.
 - **GFQL / Cypher lowering — WHERE NOT (pattern) anti-semi execution (#1031 slice 2 phase 2b)**: `WherePatternPredicate(negated=True)` no longer hard-fails at lowering. The compiler now emits a row pre-filter call (`anti_semi_apply`) for negated pattern predicates in both general MATCH lowering and connected OPTIONAL MATCH clause lowering, with per-predicate validation (must include relationship, must not introduce new aliases, must share bound aliases). Added `anti_semi_apply` row-pipeline runtime operation + call-safelist entry, plus row-table base-graph context preservation needed for correlated bindings execution. New tests cover compile shape (`row_pre_filters` emission) and runtime behavior for `MATCH (n) WHERE NOT (n)-[:R]->()` (including mixed row-expression + NOT-pattern filtering), plus connected `OPTIONAL MATCH ... WHERE NOT (pattern)` filtering/null-fill semantics. Full `cypher/test_lowering.py` suite passes (756 passed / 66 skipped) and touched-module mypy is clean.

graphistry/compute/gfql/cypher/lowering.py

@@ -3,7 +3,7 @@
 from dataclasses import dataclass, field, replace
 import math
 import re
-from typing import AbstractSet, Any, Callable, Dict, Iterable, List, Mapping, Optional, Sequence, Set, Tuple, Union, cast
+from typing import AbstractSet, Any, Callable, Dict, FrozenSet, Iterable, List, Mapping, Optional, Sequence, Set, Tuple, Union, cast
 from typing_extensions import Literal
 import pandas as pd
 
@@ -7468,6 +7468,285 @@ def _first_pattern_node_alias(clause: MatchClause) -> Optional[str]:
     return pattern[0].variable
 
 
+def _secondary_reentry_hidden_column_name(alias: str, prop: str) -> str:
+    """Hidden carry column name for a secondary whole-row alias's property access.
+
+    Distinct from `_reentry_hidden_column_name` (which is keyed only by the
+    output name) so secondary `<S>.<X>` carries cannot collide with user-named
+    scalar carries on the primary alias (#1071).
+    """
+    return f"__cypher_reentry_{alias}_{prop}__"
+
+
+_BARE_IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+
+
+def _is_whole_row_with_item(item: ReturnItem, *, match_node_aliases: Set[str]) -> bool:
+    """A WITH item is a whole-row carry when it is a bare identifier referencing
+    a node alias bound by the prior MATCH and has no rename alias (or aliases
+    to itself)."""
+    text = item.expression.text
+    if not _BARE_IDENT_RE.match(text):
+        return False
+    if item.alias is not None and item.alias != text:
+        return False
+    return text in match_node_aliases
+
+
+def _collect_secondary_property_refs(
+    expr: ExpressionText,
+    *,
+    secondary_aliases: Set[str],
+    field: str,
+) -> Tuple[ExpressionText, Set[Tuple[str, str]], Set[str]]:
+    """Walk one ExpressionText, replacing PropertyAccessExpr(Identifier(S), X)
+    with Identifier(__cypher_reentry_<S>_<X>__) for each S in
+    ``secondary_aliases``. Reports bare Identifier(S) usages too.
+
+    Returns (rewritten_expr, refs, bare_alias_uses).
+    """
+    if not secondary_aliases:
+        return expr, set(), set()
+    if not any(re.search(rf"(?<![A-Za-z0-9_]){re.escape(name)}(?![A-Za-z0-9_])", expr.text) for name in secondary_aliases):
+        return expr, set(), set()
+    try:
+        node = parse_expr(expr.text)
+    except (GFQLExprParseError, ImportError) as exc:
+        raise _unsupported(
+            "Cypher MATCH after WITH multi-alias carry rewrite requires a locally supported scalar expression",
+            field=field,
+            value=expr.text,
+            line=expr.span.line,
+            column=expr.span.column,
+        ) from exc
+    refs: Set[Tuple[str, str]] = set()
+    bare: Set[str] = set()
+    rewritten = _rewrite_secondary_alias_property_refs(
+        node,
+        secondary_aliases=secondary_aliases,
+        refs=refs,
+        bare=bare,
+    )
+    if not refs and not bare:
+        return expr, set(), set()
+    new_text = _render_expr_node(rewritten)
+    if new_text == expr.text:
+        return expr, refs, bare
+    return ExpressionText(text=new_text, span=expr.span), refs, bare
+
+
+def _rewrite_secondary_alias_property_refs(
+    node: ExprNode,
+    *,
+    secondary_aliases: Set[str],
+    refs: Set[Tuple[str, str]],
+    bare: Set[str],
+    shadowed: Optional[FrozenSet[str]] = None,
+) -> ExprNode:
+    """AST walk that rewrites secondary `S.X` to a bare hidden identifier and
+    flags bare `S` references. Quantifier/ListComprehension binders shadow
+    matching alias names within their scope (mirrors
+    ``_rewrite_expr_identifiers``)."""
+    active_shadow = shadowed or frozenset()
+    if isinstance(node, PropertyAccessExpr) and isinstance(node.value, Identifier):
+        alias_name = node.value.name
+        if alias_name in secondary_aliases and alias_name not in active_shadow:
+            refs.add((alias_name, node.property))
+            return Identifier(_secondary_reentry_hidden_column_name(alias_name, node.property))
+    if isinstance(node, Identifier):
+        if node.name in secondary_aliases and node.name not in active_shadow:
+            bare.add(node.name)
+        return node
+    if isinstance(node, QuantifierExpr):
+        next_shadow = active_shadow | {node.var}
+        return QuantifierExpr(
+            node.fn,
+            node.var,
+            _rewrite_secondary_alias_property_refs(
+                node.source, secondary_aliases=secondary_aliases, refs=refs, bare=bare, shadowed=next_shadow,
+            ),
+            _rewrite_secondary_alias_property_refs(
+                node.predicate, secondary_aliases=secondary_aliases, refs=refs, bare=bare, shadowed=next_shadow,
+            ),
+        )
+    if isinstance(node, ListComprehension):
+        next_shadow = active_shadow | {node.var}
+        return ListComprehension(
+            node.var,
+            _rewrite_secondary_alias_property_refs(
+                node.source, secondary_aliases=secondary_aliases, refs=refs, bare=bare, shadowed=next_shadow,
+            ),
+            predicate=None if node.predicate is None else _rewrite_secondary_alias_property_refs(
+                node.predicate, secondary_aliases=secondary_aliases, refs=refs, bare=bare, shadowed=next_shadow,
+            ),
+            projection=None if node.projection is None else _rewrite_secondary_alias_property_refs(
+                node.projection, secondary_aliases=secondary_aliases, refs=refs, bare=bare, shadowed=next_shadow,
+            ),
+        )
+    return _rebuild_expr_node(
+        node,
+        rewrite=lambda child: _rewrite_secondary_alias_property_refs(
+            child, secondary_aliases=secondary_aliases, refs=refs, bare=bare, shadowed=active_shadow,
+        ),
+        error_context="secondary alias rewrite",
+    )
+
+
+def _all_match_node_aliases(query: CypherQuery) -> Set[str]:
+    out: Set[str] = set()
+    for clause in query.matches:
+        for pattern in clause.patterns:
+            out.update(_pattern_node_aliases(pattern))
+        for element in _match_pattern_elements(clause):
+            if isinstance(element, NodePattern) and element.variable is not None:
+                out.add(element.variable)
+    return out
+
+
+def _demote_secondary_whole_row_aliases(
+    query: CypherQuery,
+    *,
+    prefix_stage: ProjectionStage,
+    primary_alias: Optional[str],
+) -> Tuple[CypherQuery, ProjectionStage, Tuple[str, ...]]:
+    """Rewrite ``query`` to demote any secondary whole-row alias in the prefix
+    ``WITH`` to a synthesized scalar property carry (#1071).
+
+    Returns ``(rewritten_query, rewritten_prefix_stage, secondary_aliases)``.
+    When no demotion is needed, returns the inputs unchanged with empty
+    ``secondary_aliases``.
+
+    Why: the existing MATCH-after-WITH machinery requires exactly one whole-row
+    alias in the prefix projection. Other carried aliases need only support
+    property access (``S.X``) in subsequent clauses. By rewriting ``S.X`` to a
+    bare hidden identifier and synthesizing a new prefix item ``S.X AS
+    __cypher_reentry_<S>_<X>__``, multi-alias carry reduces to the existing
+    single-alias-plus-scalars path.
+    """
+    if not query.reentry_matches:
+        return query, prefix_stage, ()
+    match_node_aliases = _all_match_node_aliases(query)
+    whole_row_items: List[Tuple[int, ReturnItem]] = [
+        (idx, item)
+        for idx, item in enumerate(prefix_stage.clause.items)
+        if _is_whole_row_with_item(item, match_node_aliases=match_node_aliases)
+    ]
+    if len(whole_row_items) <= 1:
+        return query, prefix_stage, ()
+    if primary_alias is None:
+        # Existing downstream check at the trailing-MATCH start raises a clearer
+        # error; bail out so it fires.
+        return query, prefix_stage, ()
+
+    primary_indices = {idx for idx, item in whole_row_items if item.expression.text == primary_alias}
+    if not primary_indices:
+        # Trailing MATCH starts from an alias not carried by the prefix: existing
+        # check at line ~7779 raises "must start from the same carried node alias".
+        return query, prefix_stage, ()
+    secondary_items = [(idx, item) for idx, item in whole_row_items if idx not in primary_indices]
+    secondary_aliases: Set[str] = {item.expression.text for _idx, item in secondary_items}
+
+    # Reject re-binding a secondary alias as a node variable in any trailing MATCH.
+    for trailing in (*query.reentry_matches,):
+        trailing_aliases: Set[str] = set()
+        for pattern in trailing.patterns:
+            trailing_aliases.update(_pattern_node_aliases(pattern))
+        rebound = sorted(trailing_aliases & secondary_aliases)
+        if rebound:
+            raise _unsupported_at_span(
+                "Cypher MATCH after WITH does not yet support re-binding a carried secondary alias as a node variable in the trailing MATCH",
+                field="match",
+                value=rebound,
+                span=trailing.span,
+            )
+
+    refs_collected: Set[Tuple[str, str]] = set()
+    bare_collected: Set[str] = set()
+
+    def rewrite_text(expr: ExpressionText, field: str) -> ExpressionText:
+        rewritten, refs, bare = _collect_secondary_property_refs(
+            expr,
+            secondary_aliases=secondary_aliases,
+            field=field,
+        )
+        refs_collected.update(refs)
+        bare_collected.update(bare)
+        return rewritten
+
+    # Rewrite trailing MATCH expressions (node/edge property maps via WHERE
+    # comes through reentry_wheres / clause.where).
+    rewritten_reentry_matches = tuple(
+        _rewrite_reentry_match_clause(clause, rewrite_expr=rewrite_text)
+        for clause in query.reentry_matches
+    )
+    rewritten_reentry_wheres = tuple(
+        where_clause if where_clause is None else _rewrite_where_clause_and_resync(where_clause, rewrite_text, "where")
+        for where_clause in query.reentry_wheres
+    )
+    rewritten_with_stages_tail = tuple(
+        _rewrite_reentry_projection_stage(stage, rewrite_expr=rewrite_text)
+        for stage in query.with_stages[1:]
+    )
+    rewritten_unwinds = tuple(
+        replace(unwind, expression=rewrite_text(unwind.expression, "unwind"))
+        for unwind in query.reentry_unwinds
+    )
+    rewritten_return = _rewrite_reentry_projection_clause(query.return_, rewrite_expr=rewrite_text)
+    rewritten_order_by = (
+        None
+        if query.order_by is None
+        else replace(
+            query.order_by,
+            items=tuple(
+                replace(item, expression=rewrite_text(item.expression, "order_by"))
+                for item in query.order_by.items
+            ),
+        )
+    )
+
+    if bare_collected:
+        raise _unsupported_at_span(
+            "Cypher MATCH after WITH does not yet support carrying secondary whole-row aliases as whole-row outputs; reference them by property only",
+            field="return",
+            value=sorted(bare_collected),
+            span=query.return_.span,
+        )
+
+    # Synthesize prefix WITH items: drop the secondaries, append S.X AS hidden
+    # for each unique referenced (S, X) pair.
+    new_items: List[ReturnItem] = []
+    secondary_drop_indices = {idx for idx, _item in secondary_items}
+    for idx, item in enumerate(prefix_stage.clause.items):
+        if idx in secondary_drop_indices:
+            continue
+        new_items.append(item)
+    template_span = prefix_stage.span
+    for alias_name, prop in sorted(refs_collected):
+        hidden_alias = _secondary_reentry_hidden_column_name(alias_name, prop)
+        new_items.append(
+            ReturnItem(
+                expression=ExpressionText(text=f"{alias_name}.{prop}", span=template_span),
+                alias=hidden_alias,
+                span=template_span,
+            )
+        )
+    rewritten_prefix_stage = replace(
+        prefix_stage,
+        clause=replace(prefix_stage.clause, items=tuple(new_items)),
+    )
+
+    rewritten_query = replace(
+        query,
+        with_stages=(rewritten_prefix_stage,) + rewritten_with_stages_tail,
+        reentry_matches=rewritten_reentry_matches,
+        reentry_wheres=rewritten_reentry_wheres,
+        reentry_unwinds=rewritten_unwinds,
+        return_=rewritten_return,
+        order_by=rewritten_order_by,
+    )
+    return rewritten_query, rewritten_prefix_stage, tuple(sorted(secondary_aliases))
+
+
 def _map_terminal_reentry_query(
     compiled_query: CompiledCypherQuery,
     *,
@@ -7673,14 +7952,20 @@ def _compile_bounded_reentry_query(
             span=query.return_.span,
         )
     prefix_stage = query.with_stages[0]
-    projection_items = [item.expression.text for item in prefix_stage.clause.items]
     if prefix_stage.where is not None:
         raise _unsupported_at_span(
             "Cypher MATCH after WITH does not yet support WITH ... WHERE in the prefix stage",
             field="with.where",
             value=prefix_stage.where.text,
             span=prefix_stage.span,
         )
+    primary_alias_hint = _first_pattern_node_alias(query.reentry_matches[0])
+    query, prefix_stage, _demoted_secondary_aliases = _demote_secondary_whole_row_aliases(
+        query,
+        prefix_stage=prefix_stage,
+        primary_alias=primary_alias_hint,
+    )
+    projection_items = [item.expression.text for item in prefix_stage.clause.items]
     prefix_query = replace(
         query,
         call=None,