Skip to content

Type errors from new Pyarrow release #3560

@MarcoGorelli

Description

@MarcoGorelli

PyArrow 24 has just come out, and:

narwhals/_arrow/typing.py:59: error: Class cannot subclass "DataType" (has type "Any")  [misc]
        class _BasicDataType(pa.DataType, Generic[_AsPyType]): ...
                             ^
narwhals/_utils.py:2108: error: Overloaded function signature 14 will never be matched: signature 6's parameter type(s) are the same or broader  [overload-cannot-match]
        def __get__(
        ^
narwhals/_utils.py:2108: error: Overloaded function signature 14 will never be matched: signature 7's parameter type(s) are the same or broader  [overload-cannot-match]
        def __get__(
        ^
narwhals/_utils.py:2112: error: Overloaded function signature 15 will never be matched: signature 6's parameter type(s) are the same or broader  [overload-cannot-match]
        def __get__(self, instance: LazyFrame[Any], owner: Any) -> _LazyAllowedImpl: ...
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_utils.py:2112: error: Overloaded function signature 15 will never be matched: signature 7's parameter type(s) are the same or broader  [overload-cannot-match]
        def __get__(self, instance: LazyFrame[Any], owner: Any) -> _LazyAllowedImpl: ...
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_duckdb/dataframe.py:267: error: Unused "type: ignore" comment  [unused-ignore]
            return self.lazy().collect(Implementation.PYARROW).native  # type: ignore[no-any-return]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/utils.py:706: error: Unused "type: ignore" comment  [unused-ignore]
            pa_array = pd.array(repeat(value, len(index)), dtype=native.dtype)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/utils.py:499: error: Unused "type: ignore" comment  [unused-ignore]
        return pa.arange(start, end, step)  # type: ignore[attr-defined]
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/utils.py:532: error: Unused "type: ignore" comment  [unused-ignore]
                    non_empty_mask.fill_null(False),  # type: ignore[arg-type]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/utils.py:561: error: Unused "type: ignore" comment  [unused-ignore]
        offsets = not_sorted_part.column(v).combine_chunks().offsets  # type: ignore[attr-defined]
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/series_dt.py:263: error: Unused "type: ignore" comment  [unused-ignore]
                ca = pa.chunked_array([compliant.to_arrow()])  # type: ignore[arg-type]
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/series_dt.py:265: error: "ArrowDataFrame" has no attribute "dt"  [attr-defined]
                    compliant._version.namespace.from_backend("pyarrow")
                    ^
narwhals/_arrow/series_str.py:72: error: Unused "type: ignore" comment  [unused-ignore]
            split_series = pc.split_pattern(self.native, by)  # type: ignore[call-overload]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/series_dt.py:226: error: Unused "type: ignore" comment  [unused-ignore]
                offset = lit(interval.multiple, pa.duration("ns"))  # type: ignore[assignment]
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/series.py:1090: error: Unused "type: ignore" comment  [unused-ignore]
                    return pc.logb(ca, base)  # type: ignore[return-value]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/series.py:1124: error: Unused "type: ignore" comment  [unused-ignore]
                    pc.sin,  # type: ignore[arg-type]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/series.py:1139: error: Unused "type: ignore" comment  [unused-ignore]
                    pc.cos,  # type: ignore[arg-type]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/series.py:93: error: A function returning TypeVar should receive at least one argument containing the same TypeVar  [type-var]
    ) -> _AsPyType: ...
         ^
narwhals/_arrow/series.py:97: error: Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader  [overload-cannot-match]
    def maybe_extract_py_scalar(
    ^
narwhals/_arrow/series.py:104: error: Overloaded function signature 3 will never be matched: signature 1's parameter type(s) are the same or broader  [overload-cannot-match]
    def maybe_extract_py_scalar(
    ^
narwhals/_arrow/series.py:104: error: Overloaded function signature 3 will never be matched: signature 2's parameter type(s) are the same or broader  [overload-cannot-match]
    def maybe_extract_py_scalar(
    ^
narwhals/_arrow/series.py:111: error: Overloaded function signature 4 will never be matched: signature 1's parameter type(s) are the same or broader  [overload-cannot-match]
    def maybe_extract_py_scalar(
    ^
narwhals/_arrow/series.py:111: error: Overloaded function signature 4 will never be matched: signature 2's parameter type(s) are the same or broader  [overload-cannot-match]
    def maybe_extract_py_scalar(
    ^
narwhals/_arrow/series.py:111: error: Overloaded function signature 4 will never be matched: signature 3's parameter type(s) are the same or broader  [overload-cannot-match]
    def maybe_extract_py_scalar(
    ^
narwhals/_arrow/dataframe.py:128: error: Unused "type: ignore" comment  [unused-ignore]
                return cls.from_native(pa.table(data), context=context)  # type: ignore[arg-type]
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/dataframe.py:143: error: Unused "type: ignore" comment  [unused-ignore]
                    name: pa.chunked_array(  # type: ignore[misc]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/dataframe.py:454: error: Unused "type: ignore" comment  [unused-ignore]
                    keys=left_on or [],  # type: ignore[arg-type]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/dataframe.py:455: error: Unused "type: ignore" comment  [unused-ignore]
                    right_keys=right_on,  # type: ignore[arg-type]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/dataframe.py:502: error: Unused "type: ignore" comment  [unused-ignore]
                self.native.take(pc.select_k_unstable(self.native, k, sorting)),  # type: ignore[call-overload]
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_arrow/expr.py:213: error: Unused "type: ignore" comment  [unused-ignore]
                                .indices  # type: ignore[attr-defined]
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/dataframe.py:789: error: Unused "type: ignore" comment  [unused-ignore]
            return pa_table.__arrow_c_stream__(requested_schema=requested_schema)  # type: ignore[no-untyped-call]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
narwhals/_pandas_like/namespace.py:105: error: Unused "type: ignore[arg-type, list-item]" comment  [unused-ignore]
                    pa_array = pa.array([array_value], type=pa_dtype)  # type: ignore[arg-type, list-item]
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/dtypes/pyarrow_extension_dtypes_test.py:12: error: Class cannot subclass "ExtensionType" (has type "Any")  [misc]
    class CustomInt16(pa.ExtensionType):
                      ^
tests/dtypes/pyarrow_extension_dtypes_test.py:20: error: Class cannot subclass "ExtensionType" (has type "Any")  [misc]
    class CustomInt32(pa.ExtensionType):
                      ^
tests/dtypes/pyarrow_extension_dtypes_test.py:31: error: Unused "type: ignore" comment  [unused-ignore]
    pa.register_extension_type(CustomInt16())  # type: ignore[arg-type]
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/dtypes/pyarrow_extension_dtypes_test.py:32: error: Unused "type: ignore" comment  [unused-ignore]
    pa.register_extension_type(CustomInt32())  # type: ignore[arg-type]
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/implementation_test.py:186: error: Unused "type: ignore" comment  [unused-ignore]
            any_df.lazy(df_impl)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/implementation_test.py:207: error: Argument 1 to "lazy" of "DataFrame" has incompatible type "Literal[Implementation.PYARROW]"; expected
"Literal['pyspark', 'sqlframe', 'pyspark[connect]', Implementation.PYSPARK, Implementation.SQLFRAME, Implementation.PYSPARK_CONNECT, 'dask', Implementation.DASK, 'duckdb', Implementation.DUCKDB, 'ibis', Implementation.IBIS, 'polars', Implementation.POLARS] | Module | None"
 [arg-type]
            any_df.lazy(ldf_impl)
                        ^~~~~~~~
tests/implementation_test.py:209: error: Unused "type: ignore" comment  [unused-ignore]
            any_ldf.collect(ldf_impl)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/implementation_test.py:295: error: Argument 1 to "lazy" of "DataFrame" has incompatible type "Literal[Implementation.PYARROW]"; expected
"Literal['pyspark', 'sqlframe', 'pyspark[connect]', Implementation.PYSPARK, Implementation.SQLFRAME, Implementation.PYSPARK_CONNECT, 'dask', Implementation.DASK, 'duckdb', Implementation.DUCKDB, 'ibis', Implementation.IBIS, 'polars', Implementation.POLARS] | Module | None"
 [arg-type]
            any_df.lazy(ldf_impl)
                        ^~~~~~~~
tests/implementation_test.py:298: error: Unused "type: ignore" comment  [unused-ignore]
            any_ldf.collect(df_impl)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/implementation_test.py:299: error: Unused "type: ignore" comment  [unused-ignore]
            any_ldf.collect(ldf_impl)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/implementation_test.py:300: error: Unused "type: ignore" comment  [unused-ignore]
            any_ldf.collect(ser_impl)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/implementation_test.py:315: error: Unused "type: ignore" comment  [unused-ignore]
            any_df.lazy(mix_impl)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/namespace_test.py:84: error: Unused "type: ignore" comment  [unused-ignore]
            Namespace.from_native_object(data)  # type: ignore[call-overload]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/from_pycapsule_test.py:68: error: Unused "type: ignore" comment  [unused-ignore]
            nw.from_arrow({"a": [1]}, backend=pa)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/series_only/from_iterable_test.py:80: error: Returning Any from function declared to return "Iterable[Any]"  [no-any-return]
            return pa.chunked_array([iterable])
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/frame/schema_test.py:658: error: Unused "type: ignore" comment  [unused-ignore]
            nw.Schema.from_native([("a", nw.String())])  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/frame/schema_test.py:662: error: Unused "type: ignore" comment  [unused-ignore]
            nw.Schema.from_native({"a": str})  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/frame/schema_test.py:667: error: Unused "type: ignore" comment  [unused-ignore]
            nw.Schema.from_native(nw.Schema({"a": nw.Int64()}))  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/frame/from_arrow_test.py:90: error: Unused "type: ignore" comment  [unused-ignore]
            nw.DataFrame.from_arrow(data, backend=pa)  # type: ignore[arg-type]
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/expr_and_series/over_test.py:410: error: Unused "type: ignore" comment  [unused-ignore]
        tbl = pa.table(data)  # type: ignore[arg-type]
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/translate/from_native_test.py:721: error: Unused "type: ignore" comment  [unused-ignore]
        pa_ser = cast("pa.ChunkedArray[Any]", pa.chunked_array([pa.array([1])]))  # type: ignore[redundant-cast]
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/testing/assert_series_equal_test.py:83: error: Unused "type: ignore" comment  [unused-ignore]
                series_from_native(pa.chunked_array([[2]])),  # type: ignore[misc] # pyright: ignore[reportArgumentType]
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/testing/assert_frame_equal_test.py:46: error: Unused "type: ignore" comment  [unused-ignore]
                nw.from_native(pa.table({"a": [1]})),  # type: ignore[type-var] # pyright: ignore[reportArgumentType]
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 55 errors in 25 files (checked 471 source files)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions