GFQL toUpper/toLower: pandas 3 (PyArrow strings) does SIMPLE case mapping, so pandas and polars disagree

[lmeyerov]

What

pandas 3 defaults to a PyArrow-backed str dtype whose utf8_upper / utf8_lower kernels do simple Unicode case mapping. Python's str.upper()/str.lower() — and polars, and pandas 2's object dtype — do full case mapping. GFQL's toUpper / toLower inherit whichever the engine uses, so the same query gets different answers on different engines under pandas 3.

Measured, not inferred:

input	Python	polars 1.42	pandas 2.3.3	pandas 3.0.3
'ß'.upper()	'SS'	'SS'	'SS'	'ẞ' (U+1E9E)
'İ'.lower() (U+0130)	'i̇' (i + U+0307)	'i̇'	'i̇'	'i'

import pandas as pd, polars as pl
pd.Series(["ß"]).str.upper().iloc[0]        # pandas 3.0.3 -> 'ẞ'   ; pandas 2.3.3 -> 'SS'
pl.Series(["ß"]).str.to_uppercase()[0]      # -> 'SS'
pd.Series(["İ"]).str.lower().iloc[0]   # pandas 3.0.3 -> 'i'   ; pandas 2.3.3 -> 'i̇'
pl.Series(["İ"]).str.to_lowercase()[0] # -> 'i̇'

Impact

MATCH (n) WHERE toUpper(n.name) = 'STRASSE' RETURN n.id returns a different row set on pandas 3 than on polars — no error, and pandas is this repo's oracle, so the parity suites would report it as a polars divergence if they ran there at all.

Scope is narrow: only codepoints whose full and simple case mappings differ (sharp s, the Turkish dotted capital I, ligatures, a handful of Greek forms). It is not a general casing break.

How it was found, and why it was invisible

graphistry/tests/compute/gfql/test_viz_pipeline_conformance.py already pins both cases (toupper-eq-ss-fold, tolower-turkish-dotted-i) — but that file ran in no CI lane at all: it carries a module-level pytest.importorskip("polars") and was absent from bin/test-polars.sh, so test-gfql-core (no polars) skipped it and the polars lane never listed it. Adding it to the lane (#1811) turned it red on the first run, because the polars lane resolves pandas 3.0.3 while local dev and test-gfql-core resolve 2.3.3.

The failure lands on the test's own self-check — hand pin disagrees with the pandas mirror — i.e. pandas' own behaviour no longer matches what was pinned, before any engine comparison happens.

Pinned, not papered over

Both cases now carry pytest.mark.xfail(_ARROW_SIMPLE_CASING, strict=True), where _ARROW_SIMPLE_CASING probes the runtime (pd.Series(["ß"]).str.upper().iloc[0] != "SS"). On pandas 2 the condition is False and the marker is inert — the cases must pass. On pandas 3 they must fail. Whoever reconciles the backends is therefore told to delete the marker rather than finding a stale skip.

Deciding what "correct" is

This needs an owner call, which is why it is filed rather than fixed:

1. Full case mapping everywhere (match Python/polars/pandas-2): GFQL would have to stop delegating toUpper/toLower to the pandas kernel on pandas 3 — e.g. route through .map(str.upper) for object-dtype fallback, at a real cost on the hot path.
2. Simple case mapping everywhere: cheaper and Arrow-aligned, but it changes existing pandas-2 answers and diverges from Cypher implementations that follow Unicode full mapping.
3. Declare it engine-defined and document it — the honest low-cost option, but it means toUpper is not portable, which is a real hole in the parity contract.

openCypher does not pin the case-mapping algorithm, so there is no external oracle to appeal to; Neo4j uses Java's toUpperCase(), which is full mapping (option 1).

[lmeyerov]

Duplicate of #1802, which was filed ~1h earlier and describes the same divergence (pandas ≥3's Arrow-backed str dtype does simple case mapping; GFQL's pandas toUpper/toLower inherit it and diverge from polars and from Cypher).

Filed independently while adding test_viz_pipeline_conformance.py to the polars lane — I did not see #1802 before opening this. Closing in favour of #1802; the extra detail here (the measured 4-way table across Python / polars / pandas 2.3.3 / pandas 3.0.3, and the three resolution options with the Neo4j/openCypher note) is reproduced below so nothing is lost.

input	Python	polars 1.42	pandas 2.3.3	pandas 3.0.3
'ß'.upper()	'SS'	'SS'	'SS'	'ẞ' (U+1E9E)
'İ'.lower() (U+0130)	'i̇'	'i̇'	'i̇'	'i'

Resolution options, all needing an owner call:

1. Full mapping everywhere — GFQL stops delegating to the pandas kernel on pandas 3 (e.g. .map(str.upper)), at a real hot-path cost. Matches Neo4j, whose toUpper is Java's String.toUpperCase().
2. Simple mapping everywhere — cheaper and Arrow-aligned, but changes existing pandas-2 answers.
3. Declare it engine-defined and document it — honest and free, but leaves toUpper non-portable, which is a real hole in the parity contract.

openCypher does not pin the case-mapping algorithm, so there is no external oracle.