GFQL: pandas vs polars silently disagree on toUpper/toLower under pandas>=3 (simple vs full Unicode case mapping)

[lmeyerov]

Summary

Under pandas >= 3 the GFQL pandas engine and the GFQL polars engine return
different rows for the same toUpper / toLower query, silently — no decline, no
NotImplementedError. The polars answer is the Cypher-correct one.

This violates the parity-or-NIE invariant the conformance suites are built on
(graphistry/tests/compute/gfql/test_viz_pipeline_conformance.py,
test_engine_polars_conformance_matrix.py): a non-pandas engine may only match the pandas
oracle or decline honestly — never answer differently.

Repro (measured, pandas 3.0.5 / polars 1.43.1 / pyarrow 25.0.0, py3.11)

import pandas as pd, graphistry

nodes = pd.DataFrame({"id": [8, 9, 7], "name": ["straße", "STRASSE", "istanbul İ"]})
edges = pd.DataFrame({"s": [8], "d": [9]})
g = graphistry.nodes(nodes, "id").edges(edges, "s", "d")

q = "MATCH (n) WHERE toUpper(n.name) = 'STRASSE' RETURN n.id AS id"
g.gfql(q, engine="pandas")._nodes   # -> [9]
g.gfql(q, engine="polars")._nodes   # -> [8, 9]

q2 = "MATCH (n) WHERE toLower(n.name) = 'istanbul i̇' RETURN n.id AS id"
g.gfql(q2, engine="pandas")._nodes  # -> []
g.gfql(q2, engine="polars")._nodes  # -> [7]

Root cause

graphistry/compute/gfql/row/pipeline.py:1521 implements toLower/toUpper/lower/upper
as inner.str.lower() / inner.str.upper().

• pandas < 3 stores text as object and delegates to Python str.upper/str.lower, which
  apply full Unicode case mapping — 'straße'.upper() == 'STRASSE',
  'İ'.lower() == 'i̇'.
• pandas >= 3 makes the Arrow-backed str dtype the default; its utf8_upper/utf8_lower
  are simple per-codepoint mappings — 'straße'.upper() == 'STRAẞE',
  'İ'.lower() == 'i'.

>>> pd.__version__
'3.0.5'
>>> pd.Series(["straße", "istanbul İ"]).str.upper().tolist()
['STRAẞE', 'ISTANBUL İ']
>>> pd.Series(["straße", "istanbul İ"]).str.lower().tolist()
['straße', 'istanbul i']
>>> pd.Series(["straße"]).astype(object).str.upper().tolist()   # pandas<3 behaviour
['STRASSE']

The polars lowering (graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py:202) uses
Rust to_uppercase/to_lowercase, which are full mappings — matching Cypher: neo4j's
toUpper/toLower are java.lang.String.toUpperCase()/toLowerCase(), i.e. full mapping.
So polars is right and pandas is wrong, and pandas is the engine the suites use as oracle.

Why CI never saw it

CI's own test-py3.12.lock pins pandas==3.0.3, so this is not an exotic environment — it is
the environment CI runs. The test that catches it,
test_viz_pipeline_conformance.py::test_case_regex_unicode_trick_matrix, has a module-level
pytest.importorskip("polars") and is not in bin/test-polars.sh's file list, so it is
collection-skipped in every pandas lane and never selected in the only lane with polars. It
executed in zero jobs of run 30311675717. See the companion PR for the lane fix.

Suggested fix and its cost

Make toLower/toUpper apply full case mapping independently of the pandas string backend.
The naive form (series.astype(object).map(str.upper)) is correct but elementwise, so it
should be scoped — e.g. vectorized accessor first, then repair only the rows containing a
codepoint whose full mapping differs from its simple one (the unconditional
SpecialCasing.txt set, plus the context-sensitive final-sigma case Python's str.lower()
implements and Arrow's does not: 'ΟΔΟΣ'.lower() is 'οδος' in Python, 'οδοσ' in Arrow).

That sizing — plus the CB5 rule that any code change measures its pyg-bench delta — is why
this is filed rather than patched inside the CI-visibility PR. The two affected conformance
rows are marked xfail(strict=True) there, keyed on the pandas accessor's behaviour, so the
fix will XPASS and force the marks to be retired.

Not audited here (likely the same class, worth checking with the fix): =~ '(?i)...'
case-insensitive regex, searchAny's lowercasing, ordering.py's .str.lower() type probes,
and whether cuDF's .str.upper() is simple or full.

[lmeyerov]

CI-visibility half is now open as #1805: it adds the module to bin/test-polars.sh, adds a completeness guard so a polars-gated module cannot go missing from the lane again, and marks the two affected conformance rows xfail(strict=True) keyed on the pandas accessor's own behaviour. Confirmed version-discriminating in CI on that PR: the rows PASS on the py3.9/py3.10 polars lanes (pandas 2.3.3, full case mapping) and XFAIL on py3.12/3.13/3.14 (pandas 3.0.3). Fixing this issue will turn those into XPASS and force the marks to be retired.