Merge pull request #874 from graphistry/feat/collections-support

feat: add collections validation and GFQL support

Author: lmeyerov

CHANGELOG.md

@@ -52,6 +52,13 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL / Cypher**: Extracted `ASTNormalizer` into `graphistry/compute/gfql/cypher/ast_normalizer.py` and moved shortestPath + WHERE-pattern-predicate rewrite ownership out of `lowering.py`, with parity-preserving wiring in compile/lowering flows and focused regression coverage for rewrite behavior and invocation order (#1117).
 - **GFQL / Cypher compiler**: Lowering now functionally consumes `BoundIR` metadata for the M1 integration slice: binder-provided params are merged into effective lowering params (runtime overrides preserved) with binder metadata keys filtered out of runtime-param resolution, scope membership narrowing uses the active scope frame for WITH-boundary correctness, semantic-table entity kinds inform alias table routing, and nullable alias metadata is wired into optional-only alias detection. `_StageScope` duplicated table bookkeeping was reduced, binder now runs pre- and post-normalization in compile flow, and binder-path regression tests were added for these code paths (#1116).
 
+### Changed
+- **Collections**: Autofix validation now drops invalid collections (e.g., invalid GFQL ops) and non-string collection color fields instead of string-coercing them; warnings still emit when `warn=True`.
+- **Collections**: `collections(...)` now always canonicalizes to URL-encoded JSON (string inputs are parsed + re-encoded); the `encode` parameter was removed to avoid ambiguous behavior.
+- **Collections**: Set collections now require an `id` field (server requires it for subgraph storage); missing IDs are warned and dropped in autofix mode rather than auto-generated.
+- **Collections**: Intersection collections now cross-validate that referenced set IDs exist; dangling references are warned and dropped in autofix mode.
+- **Collections**: GFQL parsing consolidated to use `_wrap_gfql_expr` from `collections.py` as the canonical implementation with precise exception handling.
+
 ### Tests
 - **GFQL / Cypher binder**: Added PR-4 white-box binder semantic conformance coverage for name resolution success/failure (including unresolved alias errors), WITH scope-reset visibility, OPTIONAL MATCH `null_extended_from` lineage as `frozenset` clause ids, label narrowing from MATCH labels + conjunctive `WHERE alias:Label` checks, and SchemaConfidence rules (min-rule propagation, operand inheritance, and strong literal/`COUNT` behavior). Parser/lowering regression lanes remain green (#1114).
 - **Plugins / cuDF**: 14 GPU tests in `TestCpuOnlyPluginsCudfRoundTrip` (`test_call_operations_gpu.py`) verifying real cuDF→pandas→cuDF round-trip for `compute_igraph` (pagerank, spanning_tree Graph-returning path, articulation_points list-return path, edge-attribute merge path), `layout_igraph`, `layout_graphviz`, `render_graphviz`, `execute_call`, `ensure_pandas` nullable dtype preservation, and `restore_engine` conversion. Requires `TEST_CUDF=1` and RAPIDS.
@@ -413,6 +420,17 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ### Infra
 - **CI**: Added pandas 2.2.3/3.0.0 compatibility jobs and minimal suite coverage.
 
+### Added
+- **Collections**: New `g.collections(...)` API for defining subsets via GFQL expressions with priority-based visual encodings. Includes helper constructors `graphistry.collection_set(...)` and `graphistry.collection_intersection(...)`, support for `showCollections`, `collectionsGlobalNodeColor`, and `collectionsGlobalEdgeColor` URL params, and automatic JSON encoding. Accepts GFQL AST, Chain objects, or wire-protocol dicts (#874).
+- **Docs / Collections**: Added collections usage guide in visualization/layout/settings, tutorial notebook (`demos/more_examples/graphistry_features/collections.ipynb`), and cross-references in 10-minute guides, cheatsheet, and GFQL docs (#875).
+
+### Changed
+- **Collections**: Autofix validation now drops invalid collections (e.g., invalid GFQL ops) and non-string collection color fields instead of string-coercing them; warnings still emit when `warn=True`.
+- **Collections**: `collections(...)` now always canonicalizes to URL-encoded JSON (string inputs are parsed + re-encoded); the `encode` parameter was removed to avoid ambiguous behavior.
+
+### Tests
+- **Collections**: Added `test_collections.py` covering encoding, GFQL Chain/AST normalization, wire-protocol acceptance, validation modes, and helper constructors.
+
 ## [0.50.4 - 2026-01-15]
 
 ### Fixed

ai/prompts/PLAN.md

@@ -125,7 +125,7 @@ git log --oneline -n 10
 - Source: `graphistry/`
 - Tests: `graphistry/tests/` (mirrors source structure: `graphistry/foo/bar.py` → `graphistry/tests/foo/test_bar.py`)
 - Docs: `docs/`
-- Plans: `plans/` (gitignored - safe for auxiliary files, temp secrets, working data)
+- Plans: `plans/` (gitignored - safe for auxiliary files, temp secrets, working data; Codex: avoid `~/.codex/plans`; if used, copy here then delete)
 - AI prompts: `ai/prompts/`
 - AI docs: `ai/docs/`
 

graphistry/Plottable.py

@@ -17,6 +17,7 @@
 from graphistry.Engine import EngineAbstractType
 from graphistry.utils.json import JSONVal
 from graphistry.client_session import ClientSession, AuthManagerProtocol
+from graphistry.models.collections import CollectionsInput
 from graphistry.models.types import ValidationParam
 
 if TYPE_CHECKING:
@@ -783,6 +784,17 @@ def settings(self,
     ) -> 'Plottable':
         ...
 
+    def collections(
+        self,
+        collections: Optional[CollectionsInput] = None,
+        show_collections: Optional[bool] = None,
+        collections_global_node_color: Optional[str] = None,
+        collections_global_edge_color: Optional[str] = None,
+        validate: ValidationParam = 'autofix',
+        warn: bool = True
+    ) -> 'Plottable':
+        ...
+
     def privacy(self, mode: Optional[PrivacyMode] = None, notify: Optional[bool] = None, invited_users: Optional[List[str]] = None, message: Optional[str] = None, mode_action: Optional[ModeAction] = None) -> 'Plottable':
         ...
 

graphistry/PlotterBase.py

@@ -2,6 +2,7 @@
 from typing import Any, Callable, Dict, List, Optional, Union, Tuple, cast, overload, TYPE_CHECKING
 from typing_extensions import Literal
 from graphistry.io.types import ComplexEncodingsDict
+from graphistry.models.collections import CollectionsInput
 from graphistry.models.types import ValidationMode, ValidationParam
 from graphistry.plugins_types.hypergraph import HypergraphResult
 from graphistry.render.resolve_render_mode import resolve_render_mode
@@ -1872,7 +1873,8 @@ def graph(self, ig: Any) -> Plottable:
     def settings(self, height=None, url_params={}, render=None):
         """Specify iframe height and add URL parameter dictionary.
 
-        The library takes care of URI component encoding for the dictionary.
+        Collections URL params are normalized and URL-encoded at plot time; other
+        params should already be URL-safe.
 
         :param height: Height in pixels.
         :type height: int
@@ -1892,6 +1894,51 @@ def settings(self, height=None, url_params={}, render=None):
         return res
 
 
+    def collections(
+        self,
+        collections: Optional[CollectionsInput] = None,
+        show_collections: Optional[bool] = None,
+        collections_global_node_color: Optional[str] = None,
+        collections_global_edge_color: Optional[str] = None,
+        validate: ValidationParam = 'autofix',
+        warn: bool = True
+    ) -> 'Plottable':
+        """Set collections URL parameters. Additive over previous settings.
+
+        :param collections: List/dict of collections or JSON/URL-encoded JSON string (stored as URL-encoded JSON).
+        :param show_collections: Toggle collections panel display.
+        :param collections_global_node_color: Hex color for non-collection nodes (leading # stripped).
+        :param collections_global_edge_color: Hex color for non-collection edges (leading # stripped).
+        :param validate: Validation mode. 'autofix' (default) drops invalid collections and color fields with warnings, 'strict' raises on issues.
+        :param warn: Whether to emit warnings when validate='autofix'. validate=False forces warn=False.
+        """
+        from graphistry.validate.validate_collections import (
+            encode_collections,
+            normalize_collections,
+            normalize_collections_url_params,
+        )
+
+        settings: Dict[str, Any] = {}
+        if collections is not None:
+            normalized = normalize_collections(collections, validate=validate, warn=warn)
+            settings['collections'] = encode_collections(normalized)
+        extras: Dict[str, Any] = {}
+        if show_collections is not None:
+            extras['showCollections'] = show_collections
+        if collections_global_node_color is not None:
+            extras['collectionsGlobalNodeColor'] = collections_global_node_color
+        if collections_global_edge_color is not None:
+            extras['collectionsGlobalEdgeColor'] = collections_global_edge_color
+        if extras:
+            extras = normalize_collections_url_params(extras, validate=validate, warn=warn)
+            settings.update(extras)
+
+        if len(settings.keys()) > 0:
+            return self.settings(url_params={**self._url_params, **settings})
+        else:
+            return self
+
+
     def privacy(
         self, 
         mode: Optional[Mode] = None, 
@@ -2239,7 +2286,11 @@ def plot(
             'viztoken': str(uuid.uuid4())
         }
 
-        viz_url = self._pygraphistry._viz_url(info, self._url_params)
+        # Validate collections in url_params (catches bypass of .collections() method)
+        from graphistry.validate.validate_collections import normalize_collections_url_params
+        url_params = normalize_collections_url_params(self._url_params, validate=validate_mode, warn=warn)
+
+        viz_url = self._pygraphistry._viz_url(info, url_params)
         cfg_client_protocol_hostname = self.session.client_protocol_hostname
         full_url = ('%s:%s' % (self.session.protocol, viz_url)) if cfg_client_protocol_hostname is None else viz_url
 

graphistry/__init__.py

@@ -24,6 +24,7 @@
     nodes,
     graph,
     settings,
+    collections,
     encode_point_color,
     encode_point_size,
     encode_point_icon,
@@ -65,6 +66,13 @@
     from_cugraph
 )
 
+from graphistry.collections import (
+    collection_set,
+    collection_intersection,
+    CollectionSet,
+    CollectionIntersection,
+)
+
 from graphistry.compute import (
     n, e, e_forward, e_reverse, e_undirected,
     let, ref,

graphistry/collections.py

@@ -0,0 +1,77 @@
+from typing import Optional, Sequence, TypeVar
+
+from graphistry.models.collections import (
+    CollectionIntersection,
+    CollectionExprInput,
+    CollectionSet,
+)
+
+CollectionDict = TypeVar("CollectionDict", CollectionSet, CollectionIntersection)
+
+
+def _apply_collection_metadata(collection: CollectionDict, **metadata: Optional[str]) -> CollectionDict:
+    value = metadata.get("id")
+    if value is not None:
+        collection["id"] = value
+    value = metadata.get("name")
+    if value is not None:
+        collection["name"] = value
+    value = metadata.get("description")
+    if value is not None:
+        collection["description"] = value
+    value = metadata.get("node_color")
+    if value is not None:
+        collection["node_color"] = value
+    value = metadata.get("edge_color")
+    if value is not None:
+        collection["edge_color"] = value
+    return collection
+
+
+def collection_set(
+    *,
+    expr: CollectionExprInput,
+    id: Optional[str] = None,
+    name: Optional[str] = None,
+    description: Optional[str] = None,
+    node_color: Optional[str] = None,
+    edge_color: Optional[str] = None,
+) -> CollectionSet:
+    """Build a collection dict for a GFQL-defined set."""
+    from graphistry.compute.ast import normalize_gfql_to_wire
+    collection: CollectionSet = {"type": "set", "expr": {"type": "gfql_chain", "gfql": normalize_gfql_to_wire(expr)}}
+    return _apply_collection_metadata(
+        collection,
+        id=id,
+        name=name,
+        description=description,
+        node_color=node_color,
+        edge_color=edge_color,
+    )
+
+
+def collection_intersection(
+    *,
+    sets: Sequence[str],
+    id: Optional[str] = None,
+    name: Optional[str] = None,
+    description: Optional[str] = None,
+    node_color: Optional[str] = None,
+    edge_color: Optional[str] = None,
+) -> CollectionIntersection:
+    """Build a collection dict for an intersection of set IDs."""
+    collection: CollectionIntersection = {
+        "type": "intersection",
+        "expr": {
+            "type": "intersection",
+            "sets": list(sets),
+        },
+    }
+    return _apply_collection_metadata(
+        collection,
+        id=id,
+        name=name,
+        description=description,
+        node_color=node_color,
+        edge_color=edge_color,
+    )

graphistry/compute/ast.py

@@ -1588,6 +1588,60 @@ def from_json(o: JSONVal, validate: bool = True) -> Union[ASTNode, ASTEdge, ASTL
     return out
 
 
+def normalize_gfql_to_wire(expr: Any) -> List[Dict[str, JSONVal]]:
+    """
+    Normalize GFQL expression to wire format (list of JSON-serializable dicts).
+
+    Accepts:
+    - Chain object
+    - Single ASTObject
+    - List of ASTObjects
+    - Dict with 'type': 'Chain' and 'chain' key
+    - Dict with 'type': 'gfql_chain' and 'gfql' key
+    - Dict with just 'chain' or 'gfql' key
+    - Single dict (parsed as AST op)
+
+    Returns:
+    - List of JSON-serializable dicts ready for wire protocol
+
+    Raises:
+    - TypeError: if expr type is not supported
+    - ValueError: if expr is empty
+    - GFQLSyntaxError: if dict cannot be parsed as valid AST
+    """
+    from graphistry.compute.chain import Chain
+
+    def _normalize_op(op: object) -> Dict[str, JSONVal]:
+        if isinstance(op, ASTObject):
+            return op.to_json()
+        if isinstance(op, dict):
+            return from_json(op, validate=True).to_json()
+        raise TypeError("GFQL operations must be AST objects or dictionaries")
+
+    def _normalize_ops(raw: object) -> List[Dict[str, JSONVal]]:
+        if isinstance(raw, Chain):
+            return _normalize_ops(raw.to_json().get("chain", []))
+        if isinstance(raw, ASTObject):
+            return [raw.to_json()]
+        if isinstance(raw, list):
+            if len(raw) == 0:
+                raise ValueError("GFQL operations list cannot be empty")
+            return [_normalize_op(op) for op in raw]
+        if isinstance(raw, dict):
+            if raw.get("type") == "Chain" and "chain" in raw:
+                return _normalize_ops(raw.get("chain"))
+            if raw.get("type") == "gfql_chain" and "gfql" in raw:
+                return _normalize_ops(raw.get("gfql"))
+            if "chain" in raw:
+                return _normalize_ops(raw.get("chain"))
+            if "gfql" in raw:
+                return _normalize_ops(raw.get("gfql"))
+            return [_normalize_op(raw)]
+        raise TypeError("GFQL expr must be Chain, ASTObject, list, or dict")
+
+    return _normalize_ops(expr)
+
+
 ###############################################################################
 # User-friendly aliases for public API
 

graphistry/models/collections.py

@@ -0,0 +1,46 @@
+from __future__ import annotations
+
+from typing import Dict, List, TYPE_CHECKING, Union
+from typing_extensions import Literal, NotRequired, Required, TypedDict
+
+from graphistry.utils.json import JSONVal
+
+if TYPE_CHECKING:
+    from graphistry.compute.ast import ASTObject
+    from graphistry.compute.chain import Chain
+
+
+CollectionExprInput = Union[
+    "Chain",
+    "ASTObject",
+    List["ASTObject"],
+    Dict[str, JSONVal],
+    List[Dict[str, JSONVal]],
+]
+
+
+class IntersectionExpr(TypedDict):
+    type: Literal["intersection"]
+    sets: List[str]
+
+
+class CollectionBase(TypedDict, total=False):
+    id: str
+    name: str
+    description: str
+    node_color: str
+    edge_color: str
+
+
+class CollectionSet(CollectionBase):
+    type: NotRequired[Literal["set"]]
+    expr: Required[CollectionExprInput]
+
+
+class CollectionIntersection(CollectionBase):
+    type: NotRequired[Literal["intersection"]]
+    expr: Required[IntersectionExpr]
+
+
+Collection = Union[CollectionSet, CollectionIntersection]
+CollectionsInput = Union[str, Collection, List[Collection]]

graphistry/pygraphistry.py

@@ -6,6 +6,8 @@
 from graphistry.plugins_types.gexf_types import GexfEdgeViz, GexfNodeViz, GexfParseEngine
 from graphistry.client_session import ClientSession, ApiVersion, ENV_GRAPHISTRY_API_KEY, DatasetInfo, AuthManagerProtocol, strtobool
 from graphistry.Engine import EngineAbstractType
+from graphistry.models.collections import CollectionsInput
+from graphistry.models.types import ValidationParam
 from graphistry.otel import inject_trace_headers, otel as otel_config
 
 """Top-level import of class PyGraphistry as "Graphistry". Used to connect to the Graphistry server and then create a base plotter."""
@@ -2376,6 +2378,24 @@ def settings(self, height=None, url_params={}, render=None):
 
         return self._plotter().settings(height, url_params, render)
 
+    def collections(
+        self,
+        collections: Optional[CollectionsInput] = None,
+        show_collections: Optional[bool] = None,
+        collections_global_node_color: Optional[str] = None,
+        collections_global_edge_color: Optional[str] = None,
+        validate: ValidationParam = 'autofix',
+        warn: bool = True
+    ):
+        return self._plotter().collections(
+            collections=collections,
+            show_collections=show_collections,
+            collections_global_node_color=collections_global_node_color,
+            collections_global_edge_color=collections_global_edge_color,
+            validate=validate,
+            warn=warn
+        )
+
     def _viz_url(self, info: DatasetInfo, url_params: Dict[str, Any]) -> str:
         splash_time = int(calendar.timegm(time.gmtime())) + 15
         extra = "&".join([k + "=" + str(v) for k, v in list(url_params.items())])
@@ -2604,6 +2624,7 @@ def _handle_api_response(self, response):
 pipe = PyGraphistry.pipe
 graph = PyGraphistry.graph
 settings = PyGraphistry.settings
+collections = PyGraphistry.collections
 hypergraph = PyGraphistry.hypergraph
 bolt = PyGraphistry.bolt
 cypher = PyGraphistry.cypher