Changes from 4.9.0 to 4.9.1
A small hot-fix release for the Arrow interop work in 4.9.0: a real
performance regression in dictionary-column export, and a clearer error
message when opening a nonexistent CTable in append mode.
Improvements
CTable.iter_arrow_batches()(and thereforeto_arrow()and the Arrow
PyCapsule interchange,__arrow_c_stream__) no longer recomputes the
full live-row-position array from scratch on every batch, for every
dictionary column — anO(n_rows)scan that was repeated
O(n_rows / batch_size)times. The position array is now computed once
per export call instead. Measured 6-14x faster export for
dictionary-encoded string columns (e.g.company) on a 1M-row table.- Reminder for anyone consuming a
CTablethrough the Arrow PyCapsule
protocol (DuckDB, pyarrow, Polars, pandas): the raw Arrow C Stream
interface has no column-projection pushdown, so a consumer that only
needs a few columns still triggers export of every column in the table.
UseCTable.select([...])to project down to the columns you actually
need before handing the table to the consumer, particularly if any
column is an expensive nested/list type.
Bug fixes
- Opening a
CTablewithmode="a"at a path that doesn't exist yet now
raises a clearFileNotFoundError("mode='a' opens an existing table;
use mode='w' to create a new one") instead of silently falling through
and creating a new, empty table.