Skip to content

Commit c0ab5c4

Browse files
NickCrewsclaude
andcommitted
fix: unbreak CI, pyarrow 25 added binary_view compare kernels
PR CI installs the latest pyarrow, and pyarrow 25.0.0 added binary_view compare kernels. The strict-xfail canary TestCanaries::test_pyarrow_gains_binary_view_filter_support only exercised scanner construction, so it now XPASSes and fails CI for every PR. pyarrow's support is still only partial: the scanner can be built, but executing it raises ArrowNotImplementedError because 'array_filter' has no binary_view kernel. So DuckDB's post-scan fallback for view types is still required. The canary is converted to a version-gated assertion: * pyarrow < 25: scanner construction raises ArrowNotImplementedError * pyarrow >= 25: scanner construction succeeds, execution raises Either branch failing means pyarrow moved again and we should revisit the fallback in TestUnsupportedTypes. The sibling string_view canary (unchanged behavior through pyarrow 25) now executes the scanner as well, so it will only fire on full end-to-end support instead of breaking CI when compare kernels land without filter kernels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 5829acc commit c0ab5c4

1 file changed

Lines changed: 22 additions & 10 deletions

File tree

tests/fast/arrow/test_filter_pushdown.py

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -754,28 +754,40 @@ class TestCanaries:
754754

755755
@pytest.mark.xfail(
756756
raises=pa_lib.ArrowNotImplementedError,
757-
reason="pyarrow does not yet implement string_view filter compare kernels",
757+
reason="pyarrow does not yet implement string_view filter kernels",
758758
strict=True,
759759
)
760760
def test_pyarrow_gains_string_view_filter_support(self):
761-
"""When pyarrow adds string_view comparison kernels this will xpass.
761+
"""When pyarrow can execute a string_view filter end-to-end this will xpass.
762762
763763
At that point we should remove the post-scan fallback in TestUnsupportedTypes.
764+
Today the scanner cannot even be constructed ('equal' has no string_view
765+
kernel); executing it is included so this canary only fires on full support
766+
(see the binary_view canary below for how compare kernels can land first).
764767
"""
765768
filter_expr = pa_ds.field("col") == pa_ds.scalar("val1")
766769
table = pa.table({"col": pa.array(["val1", "val2"], type=pa.string_view())})
767-
pa_ds.dataset(table).scanner(columns=["col"], filter=filter_expr)
770+
pa_ds.dataset(table).scanner(columns=["col"], filter=filter_expr).to_table()
768771

769-
@pytest.mark.xfail(
770-
raises=pa_lib.ArrowNotImplementedError,
771-
reason="pyarrow does not yet implement binary_view filter compare kernels",
772-
strict=True,
773-
)
774772
def test_pyarrow_gains_binary_view_filter_support(self):
775-
"""When pyarrow adds binary_view comparison kernels this will xpass."""
773+
"""Canary tracking pyarrow's binary_view filter support.
774+
775+
pyarrow 25.0.0 added binary_view compare kernels, so constructing a
776+
scanner with a binary_view filter now succeeds. Executing the scan still
777+
raises: 'array_filter' has no binary_view kernel. Once execution works
778+
too, this test fails and we should remove the post-scan fallback in
779+
TestUnsupportedTypes.
780+
"""
776781
filter_expr = pa_ds.field("col") == pa_ds.scalar(pa.scalar(b"bin1", pa.binary_view()))
777782
table = pa.table({"col": pa.array([b"bin1", b"bin2"], type=pa.binary_view())})
778-
pa_ds.dataset(table).scanner(columns=["col"], filter=filter_expr)
783+
if Version(pa.__version__) < Version("25.0.0"):
784+
# No binary_view compare kernels: the scanner cannot even be constructed.
785+
with pytest.raises(pa_lib.ArrowNotImplementedError, match="binary_view"):
786+
pa_ds.dataset(table).scanner(columns=["col"], filter=filter_expr)
787+
else:
788+
scanner = pa_ds.dataset(table).scanner(columns=["col"], filter=filter_expr)
789+
with pytest.raises(pa_lib.ArrowNotImplementedError, match="binary_view"):
790+
scanner.to_table()
779791

780792
# ----- DuckDB optimizer decisions we expect to change --------------
781793

0 commit comments

Comments
 (0)