Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
- **Seeded typed-hop property projection keeps its fast path through DISTINCT / ORDER BY / SKIP / LIMIT**: those trailing row ops are plain frame operations, so the fast path now delegates them to the canonical chain instead of declining the whole query.

### Changed
- **`rows(table=...)` and rewrite-param regression suites now run on all four engines** (tests only; zero runtime delta — no production line changed). The #1788 and #1790 suites were parametrized over pandas + polars only, so the `table` guards they exist to pin were never exercised on cuDF or polars-gpu at all. Both files were already in `bin/test-polars.sh` (#1805), so their polars params did run; what ran nowhere was `test_rows_table_named_middle.py` outside that one lane — a module-level `pytest.importorskip("polars")` skipped the whole file in `test-gfql-core`, which installs no polars. Both files are now parametrized over pandas / cuDF / polars / polars-gpu from a **fixed** engine list plus a classified availability check; dropping the module-level `importorskip` also makes 5 pandas cases live in `test-gfql-core`. The availability check is deliberately not `available_nonpandas_engines()`, which builds its list by importability so a missing engine vanishes from the report instead of showing as SKIPPED, and it is deliberately not a blanket `except Exception` either: a missing module skips, a recognisable GPU-stack error skips **with its text quoted**, and any other failure propagates — because a skipped GPU parameter reads as evidence of passing. Both traps were hit by trying rather than reasoned about: a probe that ran the shape under test turned a reverted production guard into 20 SKIPS instead of 20 failures, and a blanket probe silently dropped cuDF from an otherwise-green GPU run on a transient `cudaErrorMemoryAllocation`. GPU receipts on dgx GB10 / cuDF 26.02.01 / polars 1.35.2 / `docker run --gpus all`: **45 passed, 3 xfailed, 0 skipped**; mutation-checked by reverting the #1788 and #1790 `table` guards — **24 failures spread evenly over all four engines (6 each)**, so the added parameters are not decorative. Two gaps the new parameters surfaced are pinned as strict xfails rather than papered over: #1803 (the indexed bindings bypass excludes `Engine.POLARS_GPU`, so polars-gpu silently takes the scan path) and #1804 (the native polars bindings builder never receives `alias_prefilters` — previously an unnumbered xfail).
- **Row execution context: `clear_row_exec_context` NULLs rather than restores, and that is now pinned rather than assumed** (tests + comments only; zero runtime delta — no production line changed). Review of #1793 asked whether the exit should RESTORE what the graph carried on entry instead of nulling, since `attach_row_exec_context` INHERITS on the way in (a `None` argument keeps what `g` already carries). Settled as NULL, for three reasons now each carried by a test: (1) `clear` is pure — it returns `g.bind()` and never writes through to the object it was handed, so an outer scope that set the field still has it afterwards and there is no caller state to save; that is what separates this from #1786, which WAS an in-place write onto the caller's own graph, and restore is the fix for a mutation. (2) The only channel restore would change is the RETURN value, and putting the seed back there IS the second half of #1786 — measured, a seed hand-restored onto a result changes the answer of the next query run on that result (7 → 2 → 1 rows). (3) No execution frame inherits a context it did not set: instrumenting `attach` over `graphistry/tests/compute` recorded 3907 calls and zero inheriting ones (53 entered on a graph already carrying a seed, but each was handed the identical `start_nodes` parameter, so the frame still owns the value), because the cross-segment `WITH` seed travels as the explicit `start_nodes` PARAMETER and never through the graph field. The new `test_exec_context_scoping.py` re-runs that ownership measurement as an assertion, so a future path that starts relying on inheritance reopens the decision loudly instead of silently losing an outer value. Mutation-checked in the deciding direction: implementing save/restore at all three attach sites fails 4 of the new tests on every runnable engine, while the existing #1793 suite passes unchanged — i.e. those tests could not distinguish the two designs and this file can. Engine-parametrized over pandas/cuDF/polars/polars-gpu from a fixed list plus a classified availability check (a missing module skips, a recognisable GPU-stack error skips with its text quoted, and any other failure propagates — a silently skipped GPU parameter reads as evidence of passing); GPU receipts on dgx GB10 / cuDF 26.02.01 / `docker run --gpus all`: 43 passed, 0 skipped.
- **`is_lazy` is a type predicate, so the polars engine's eager/lazy split is checked instead of asserted**: `dtypes.is_lazy` decides WHICH member of the two-member `PolarsFrame` union a frame is, but it was declared `-> bool`, so that answer was discarded at the call boundary — an eager-only attribute (`.height`, `.columns`, `.schema`) reached after a lazy guard was unprovable, and the native polars chain closed the gap with `cast("pl.DataFrame", frame)`. A cast is not a type: it re-asserts, unchecked, the exact fact the predicate had just established, once per call site, and a wrong one surfaces as a production `AttributeError` rather than a CI error. `is_lazy` is now declared `-> TypeIs["pl.LazyFrame"]` (PEP 742), which narrows the *negative* branch as well as the positive — the eager side is the one that needed it, so `TypeGuard` would not have done — and the cast is gone. Four previously untyped chain-combine helpers (`_semi`, `_combine_edges`, `_apply_node_names`, and the `_known_empty` guard) now carry real signatures: `_semi`'s two frames share the `PolarsT` TypeVar because polars joins do not mix eagerness, and the two combine helpers take the `_LazyShim` collect-once duck-type they are actually called with. Typing `_apply_node_names` also retired a `getattr(next_step, "edges_empty", None)` — the shim declares that tri-state in `__slots__`, so a typo is now a checker error instead of a silent `None` that would have re-armed the very cardinality gate the guard disarms. Internal only: no runtime behaviour, no public API, and the `TypeIs` import is `TYPE_CHECKING`-only, so no new runtime dependency floor.
- **GFQL execution context is declared rather than attached at runtime**: the private `_gfql_*` per-execution fields (index policy and registry, row-pipeline base graph, carried seed nodes, edge aliases, shortest-path backend, and the indexed-bindings handoff) are now declared on `Plottable` with defaults on `PlotterBase`, instead of being set with `setattr` and read back with `getattr(..., default)`. No public API or behaviour change; internal call sites are typed, and hand-rolled `Plottable` stand-ins must now construct the full context.
Expand Down
65 changes: 65 additions & 0 deletions graphistry/tests/compute/gfql/polars_test_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
signature tier; files with weaker stringified sigs keep them deliberately (documented there).
NOT a pytest module (no ``test_`` prefix): importers do their own pytest.importorskip("polars")
— nothing here imports polars at module scope."""
from typing import Callable, Mapping, Optional, Tuple

import pandas as pd
import pytest

Expand Down Expand Up @@ -181,3 +183,66 @@ def assert_surfaces_agree(res_a, res_b, label):
assert res_a[0] == res_b[0], f"{label}: surface divergence {res_a[0]} != {res_b[0]} (silent-bridge class)"
if res_a[0] == "ok":
assert res_a[1] == res_b[1], f"{label}: surfaces both ok but signatures differ"


# --- fixed-engine-list plumbing (#1795 / CB3). ONE definition, because the alternative is the
# trap this replaces: `available_nonpandas_engines()` above BUILDS its list by importability, so
# an engine that is missing simply vanishes from the report instead of showing as SKIPPED. ---

#: Substrings that identify a broken/absent GPU STACK rather than a product defect. Matched
#: against `type(ex).__name__: str(ex)`, lowercased. Deliberately narrow: everything not listed
#: here is treated as a REAL failure, because the cost of guessing wrong in that direction is a
#: silently green GPU lane.
_GPU_ENVIRONMENT_MARKERS = (
"libnvrtc", "libcuda", "libcudart", "nvrtc", "cuinit",
"no cuda-capable device", "cuda driver", "cuda runtime",
"cuda_error", "cudaerror", "driver version",
"gpu engine requested", # polars' own message when cudf_polars is absent
)


def gpu_environment_reason(ex: BaseException) -> Optional[str]:
"""Return a skip reason iff ``ex`` is an ENVIRONMENT fact, else None (= a real failure).

A probe that swallows every exception is worse than no probe: it converts a genuine
regression into a SKIP, and a skipped GPU parameter reads as evidence of passing. Measured
on the dgx box — a probe that ran the shape under test turned a reverted production guard
into 20 skips instead of 20 failures, and an intermittent cold-start error in a fresh
container silently dropped cuDF from a run that otherwise passed.
"""
if isinstance(ex, ImportError):
return f"{type(ex).__name__}: {ex}"
text = f"{type(ex).__name__}: {ex}".lower()
if any(marker in text for marker in _GPU_ENVIRONMENT_MARKERS):
return f"{type(ex).__name__}: {ex}"
return None


#: Engine name -> the modules whose ABSENCE is a stated skip. Fixed on purpose: an engine that
#: is not listed here has no excuse to skip, which is what keeps a missing engine visible.
_ENGINE_REQUIRED_MODULES: Mapping[str, Tuple[str, ...]] = {
"polars": ("polars",),
"polars-gpu": ("polars", "cudf_polars"),
"cudf": ("cudf",),
}


def engine_skip_reason(engine: str, smoke: Callable[[], object]) -> Optional[str]:
"""``None`` => this engine MUST run here. A string => a stated, checkable skip reason.

``smoke`` is a zero-argument callable running a query that is NOT the shape under test —
running the asserted shape would let a regression disarm its own test.
"""
import importlib.util

for module in _ENGINE_REQUIRED_MODULES.get(engine, ()):
if importlib.util.find_spec(module) is None:
return f"{module} is not installed"
try:
smoke()
except BaseException as ex: # noqa: BLE001 — classified below, never blanket-swallowed
reason = gpu_environment_reason(ex)
if reason is not None:
return reason
raise
return None
Loading
Loading