Fix Arrow string_view ingestion and correct pandas interop claims

CTable.from_arrow() raised TypeError on Arrow's string_view/binary_view
layout (Polars' default string export via the PyCapsule protocol), so
ingesting a bare Polars DataFrame with any string column failed. Added
_is_arrow_string_type()/_is_arrow_binary_type() helpers and used them
everywhere an Arrow type is inspected during schema inference.

Also corrected the "pandas >= 2.2 consumes a CTable directly" claim in
4.9.0 release material: the plain pd.DataFrame(t) constructor does not
use the Arrow PyCapsule protocol at all and silently produces a broken
DataFrame. The working API is pandas.DataFrame.from_arrow(t), added in
pandas 3.0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: FrancescAlted

ANNOUNCE.rst

@@ -13,17 +13,21 @@ a fast, compact layer underneath the tools you already use.
 The main highlights are:
 
 - **Arrow PyCapsule interchange**: ``CTable.__arrow_c_stream__`` lets
-  pyarrow, DuckDB, Polars, and pandas >= 2.2 consume a ``CTable`` directly as
-  a stream of record batches, with bounded memory — no ``to_arrow()`` copy
-  step needed::
+  pyarrow, DuckDB, and Polars consume a ``CTable`` directly as a stream of
+  record batches, with bounded memory — no ``to_arrow()`` copy step
+  needed::
 
       import duckdb, blosc2
       t = blosc2.CTable.open("trips.b2z")
       duckdb.sql("SELECT company, avg(fare) FROM t GROUP BY company").show()
 
-  ``CTable.from_arrow()`` now accepts any object implementing the same
-  protocol on ingest too — a pyarrow Table, a Polars DataFrame, a Parquet
-  reader — in addition to the existing ``(schema, batches)`` form.
+  pandas >= 3.0 gets the same treatment via the new
+  ``pandas.DataFrame.from_arrow(t)`` classmethod (the plain ``pd.DataFrame(t)``
+  constructor doesn't use this protocol). ``CTable.from_arrow()`` now
+  accepts any object implementing the same protocol on ingest too — a
+  pyarrow Table, a Polars DataFrame, a Parquet reader — in addition to the
+  existing ``(schema, batches)`` form, including Arrow's ``string_view``
+  layout (Polars' default string export type).
 
 - **``blosc2.utf8()``: string columns that speak Arrow's layout**. Instead of
   a blosc2-specific encoding, ``utf8()`` stores text exactly as Arrow does —

RELEASE_NOTES.md

@@ -4,25 +4,29 @@
 
 This release is about cooperation: `CTable` now speaks the tabular
 ecosystem's own protocols instead of asking it to speak blosc2's. Arrow
-tools (pyarrow, DuckDB, Polars, pandas >= 2.2) can consume or produce a
-`CTable` directly through the Arrow PyCapsule interface; a new `utf8()`
-string column stores text in Arrow's own offsets+bytes layout and reads
-back as NumPy `StringDType`; and `engine=blosc2.jit` now runs correctly
-inside pandas 3 itself. Alongside that, `CTable` gained a proper
-missing-data story (`fillna`/`dropna`, null-safe arithmetic and
-comparisons) and a pandas-3-style chaining API (`assign()`/`col()`,
-UDF aggregations, `CTable.apply()`).
+tools (pyarrow, DuckDB, Polars, and pandas >= 3.0 via `DataFrame.from_arrow()`)
+can consume or produce a `CTable` directly through the Arrow PyCapsule
+interface; a new `utf8()` string column stores text in Arrow's own
+offsets+bytes layout and reads back as NumPy `StringDType`; and
+`engine=blosc2.jit` now runs correctly inside pandas 3 itself. Alongside
+that, `CTable` gained a proper missing-data story (`fillna`/`dropna`,
+null-safe arithmetic and comparisons) and a pandas-3-style chaining API
+(`assign()`/`col()`, UDF aggregations, `CTable.apply()`).
 
 ### New features
 
 #### Ecosystem interop
 
 - **Arrow PyCapsule interchange**: `CTable.__arrow_c_stream__` lets
-  pyarrow, DuckDB, Polars, and pandas >= 2.2 consume a `CTable` directly
-  as a stream of record batches, with bounded memory — no
-  `to_arrow()`/copy step required. `CTable.from_arrow()` now accepts any
-  object implementing the same protocol on ingest (single-argument
-  form), in addition to the existing `(schema, batches)` form.
+  pyarrow, DuckDB, and Polars consume a `CTable` directly as a stream of
+  record batches, with bounded memory — no `to_arrow()`/copy step
+  required. pandas >= 3.0 can do the same via the new
+  `pandas.DataFrame.from_arrow()` classmethod (the plain `pd.DataFrame(t)`
+  constructor does not use this protocol). `CTable.from_arrow()` now
+  accepts any object implementing the same protocol on ingest
+  (single-argument form), in addition to the existing `(schema, batches)`
+  form — including Arrow's `string_view` layout (Polars' default string
+  export type), which raised `TypeError` before this release.
 - **`blosc2.utf8()`**: a new column type for high-cardinality/free-text
   strings, storing each column as two companion NDArrays — int64 row
   offsets plus a UTF-8 byte blob — the same layout Arrow uses for
@@ -118,6 +122,13 @@ UDF aggregations, `CTable.apply()`).
 - Fixed a `NameError` when `nan`/`inf` scalars appeared in lazy
   expressions; `ShapeInferencer` no longer ignores user-provided shapes
   for `nan`/`inf`.
+- Fixed `CTable.from_arrow()` raising `TypeError: No blosc2 spec for Arrow
+  type DataType(string_view)` on any Arrow `string_view`/`binary_view`
+  column — the layout Polars exports by default through the PyCapsule
+  protocol. `string_view`/`binary_view` now import exactly like
+  `string`/`large_string`/`binary`/`large_binary` everywhere a column's
+  Arrow type is inspected (schema inference, null-sentinel selection,
+  list/struct/dictionary value types).
 
 ## Changes from 4.8.0 to 4.8.1
 

doc/getting_started/tutorials/13.ctable-basics.ipynb

@@ -1890,8 +1890,9 @@
     "### 8.3 Zero-copy Arrow interchange (new in 4.9.0)\n",
     "\n",
     "`to_arrow()`/`from_arrow()` above go through a full `pyarrow.Table` copy. The Arrow **PyCapsule interchange protocol**\n",
-    "skips that step: `CTable.__arrow_c_stream__` lets any tool that speaks the protocol — pyarrow, DuckDB, Polars, pandas >= 2.2 —\n",
-    "read a `CTable` directly as a stream of record batches, with bounded memory."
+    "skips that step: `CTable.__arrow_c_stream__` lets any tool that speaks the protocol — pyarrow, DuckDB, Polars —\n",
+    "read a `CTable` directly as a stream of record batches, with bounded memory. pandas >= 3.0 gets the same treatment\n",
+    "via the new `pandas.DataFrame.from_arrow(t)` classmethod (the plain `pd.DataFrame(t)` constructor does not use this protocol)."
    ]
   },
   {

doc/reference/ctable.rst

@@ -621,14 +621,18 @@ well as import/export paths for CSV, Arrow, and Parquet data.
 
 CTable also implements the `Arrow PyCapsule interchange protocol
 <https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html>`_
-via :meth:`~blosc2.CTable.__arrow_c_stream__`, so pyarrow, DuckDB, Polars, and
-pandas >= 2.2 can consume a CTable directly as a stream of record batches
-(``pa.table(ct)``, ``duckdb.sql("SELECT ... FROM ct")``, ``pl.DataFrame(ct)``),
-and :meth:`~blosc2.CTable.from_arrow` accepts any object implementing that
-protocol on ingest. Strict zero-copy is not possible — the underlying data is
-compressed, so decompression is unavoidably a copy — but there is no
-intermediate materialization: batches are decompressed and handed to the
-consumer one at a time, so memory use stays bounded regardless of table size.
+via :meth:`~blosc2.CTable.__arrow_c_stream__`, so pyarrow, DuckDB, and Polars
+can consume a CTable directly as a stream of record batches
+(``pa.table(ct)``, ``duckdb.sql("SELECT ... FROM ct")``, ``pl.DataFrame(ct)``).
+pandas >= 3.0 can do the same via the new ``pandas.DataFrame.from_arrow(ct)``
+classmethod (the plain ``pd.DataFrame(ct)`` constructor does not use this
+protocol). :meth:`~blosc2.CTable.from_arrow` accepts any object implementing
+the protocol on ingest, including Arrow's ``string_view`` layout (Polars'
+default string export type). Strict zero-copy is not possible — the
+underlying data is compressed, so decompression is unavoidably a copy — but
+there is no intermediate materialization: batches are decompressed and
+handed to the consumer one at a time, so memory use stays bounded regardless
+of table size.
 
 .. automethod:: CTable.load
 .. automethod:: CTable.open

src/blosc2/ctable.py

@@ -91,6 +91,28 @@
 )
 
 
+def _is_arrow_string_type(pa, pa_type) -> bool:
+    """True for any Arrow string-like type, including the view-based layout.
+
+    ``string_view`` (Arrow's variable-length view layout, e.g. Polars'
+    default export type for string columns via the PyCapsule interface) is
+    not one of ``string``/``large_string``/``utf8``/``large_utf8`` but is
+    handled identically everywhere those are.
+    """
+    if pa_type in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8()):
+        return True
+    is_string_view = getattr(pa.types, "is_string_view", None)
+    return bool(is_string_view is not None and is_string_view(pa_type))
+
+
+def _is_arrow_binary_type(pa, pa_type) -> bool:
+    """True for any Arrow binary-like type, including the view-based layout."""
+    if pa.types.is_binary(pa_type) or pa.types.is_large_binary(pa_type):
+        return True
+    is_binary_view = getattr(pa.types, "is_binary_view", None)
+    return bool(is_binary_view is not None and is_binary_view(pa_type))
+
+
 @dataclass(frozen=True)
 class NullPolicy:
     """Default sentinels for inferred CTable scalar nulls.
@@ -165,9 +187,9 @@ def sentinel_for_arrow_type(self, pa, pa_type):
             return self.float_value
         if pa_type == pa.bool_():
             return self.bool_value
-        if pa_type in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8()):
+        if _is_arrow_string_type(pa, pa_type):
             return self.string_value
-        if pa.types.is_binary(pa_type) or pa.types.is_large_binary(pa_type):
+        if _is_arrow_binary_type(pa, pa_type):
             return self.bytes_value
         if pa.types.is_timestamp(pa_type):
             return self.timestamp_value
@@ -6844,7 +6866,9 @@ def _arrow_type_needs_object_fallback(pa, pa_type) -> bool:
         """True when *pa_type* has no typed CTable mapping."""
         if pa.types.is_dictionary(pa_type):
             vt = pa_type.value_type
-            return vt not in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8())
+            return not _is_arrow_string_type(pa, vt)
+        if _is_arrow_string_type(pa, pa_type):
+            return False
         if pa_type in (
             pa.int8(),
             pa.int16(),
@@ -6857,13 +6881,9 @@ def _arrow_type_needs_object_fallback(pa, pa_type) -> bool:
             pa.float32(),
             pa.float64(),
             pa.bool_(),
-            pa.string(),
-            pa.large_string(),
-            pa.utf8(),
-            pa.large_utf8(),
         ):
             return False
-        if pa.types.is_binary(pa_type) or pa.types.is_large_binary(pa_type):
+        if _is_arrow_binary_type(pa, pa_type):
             return False
         if pa.types.is_timestamp(pa_type):
             return False
@@ -6915,7 +6935,7 @@ def _arrow_type_to_spec(  # noqa: C901
 
         if pa.types.is_dictionary(pa_type):
             vt = pa_type.value_type
-            if vt in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8()):
+            if _is_arrow_string_type(pa, vt):
                 index_type = pa_type.index_type
                 # Accept signed and unsigned integer index types; validate fit in int32.
                 if not (pa.types.is_integer(index_type) or pa.types.is_unsigned_integer(index_type)):
@@ -6988,7 +7008,7 @@ def _arrow_type_to_spec(  # noqa: C901
                 item_arrow_col = None
                 nullable = True
             item_string_max_length = string_max_length
-            if pa_type.value_type in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8()):
+            if _is_arrow_string_type(pa, pa_type.value_type):
                 item_string_max_length = max(string_max_length or 1, 1_000_000)
             item_spec = CTable._arrow_type_to_spec(
                 pa,
@@ -7009,7 +7029,7 @@ def _arrow_type_to_spec(  # noqa: C901
                     )
                     child_col = combined.field(field.name)
                 child_string_max_length = string_max_length
-                if field.type in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8()):
+                if _is_arrow_string_type(pa, field.type):
                     child_string_max_length = max(string_max_length or 1, 1_000_000)
                 fields[field.name] = CTable._arrow_type_to_spec(
                     pa,
@@ -7021,7 +7041,7 @@ def _arrow_type_to_spec(  # noqa: C901
                 )
             return b2s.struct(fields, nullable=nullable)
 
-        if pa_type in (pa.string(), pa.large_string(), pa.utf8(), pa.large_utf8()):
+        if _is_arrow_string_type(pa, pa_type):
             if string_max_length is None:
                 from blosc2.utf8_array import have_string_dtype
 
@@ -7036,7 +7056,7 @@ def _arrow_type_to_spec(  # noqa: C901
             max_length = max(string_max_length, len(null_value) if null_value is not None else 1, 1)
             return b2s.string(max_length=max_length, null_value=null_value)
 
-        if pa.types.is_binary(pa_type) or pa.types.is_large_binary(pa_type):
+        if _is_arrow_binary_type(pa, pa_type):
             if string_max_length is None:
                 # No fixed-width threshold given: store as variable-length scalar bytes.
                 return b2s.vlbytes(nullable=nullable)
@@ -7102,12 +7122,8 @@ def _compiled_columns_from_arrow(
                 and not field_is_dictionary
                 and column_string_max_length is None
                 and (
-                    pa.types.is_binary(field.type)
-                    or pa.types.is_large_binary(field.type)
-                    or (
-                        not have_string_dtype()
-                        and (pa.types.is_string(field.type) or pa.types.is_large_string(field.type))
-                    )
+                    _is_arrow_binary_type(pa, field.type)
+                    or (not have_string_dtype() and _is_arrow_string_type(pa, field.type))
                 )
             )
             field_needs_object_fallback = cls._arrow_type_needs_object_fallback(pa, field.type)

tests/ctable/test_arrow_interop.py

@@ -323,6 +323,24 @@ def test_from_arrow_string_default_is_utf8():
     assert list(t["name"][:]) == ["hi", "hello world", "!"]
 
 
+def test_from_arrow_string_view_imports_as_utf8():
+    """Arrow's view-based string layout (Polars' default PyCapsule export type)
+
+    is not one of string/large_string/utf8/large_utf8, but must still import
+    like them instead of raising "No blosc2 spec for Arrow type".
+    """
+    if not hasattr(pa.types, "is_string_view"):
+        pytest.skip("this pyarrow version has no string_view type")
+    at = pa.table({"name": pa.array(["hi", None, "!"], type=pa.string())}).cast(
+        pa.schema([pa.field("name", pa.string_view(), nullable=True)])
+    )
+    assert pa.types.is_string_view(at.schema.field("name").type)
+    t = CTable.from_arrow(at.schema, at.to_batches())
+    if hasattr(np.dtypes, "StringDType"):
+        assert t["name"].is_utf8
+    assert list(t["name"].fillna("<null>")) == ["hi", "<null>", "!"]
+
+
 def test_from_arrow_string_fixed_width_with_max_length():
     """Passing string_max_length gives a fixed-width NDArray string column."""
     at = pa.table({"name": pa.array(["hi", "hello world", "!"], type=pa.string())})