Getting ready for release 4.4.3

Author: FrancescAlted

ANNOUNCE.rst

@@ -1,57 +1,61 @@
-Announcing Python-Blosc2 4.4.2
+Announcing Python-Blosc2 4.4.3
 ==============================
 
-We are happy to announce this feature and maintenance release that promotes
-DSL kernels to first-class CTable computed columns, adds a convenient new
-column-assignment API, speeds up bulk NDArray writes, and fixes several
-correctness issues.
+We are happy to announce this maintenance release that makes CTable
+cold-start, printing, querying and groupby noticeably faster, trims the
+memory footprint of ``import blosc2``, adds raw-storage access for columns,
+and exposes the new JPEG 2000 codec plugins.
 
 The main highlights are:
 
-- **DSL kernels as first-class CTable columns**: ``@blosc2.jit``-decorated
-  functions can now back virtual computed columns, stored generated columns,
-  and ``where()`` filter predicates directly — in addition to the existing
-  string-expression form.  Columns survive save/open round-trips via
-  persisted source.
+- **Faster CTable cold-start**: views created by ``select()`` now open
+  columns lazily — only the columns actually read are loaded from storage —
+  and queries only open the SUMMARY indexes referenced by the predicate,
+  instead of every indexed column on a wide persistent table.
 
-- **New ``t["col"] = arr`` assignment**: a clean shorthand for overwriting all
-  live rows of a column.  Accepts any array-like, including ``blosc2.NDArray``.
+- **Faster printing and groupby**: table rendering now performs a single
+  combined sparse read per column (instead of ~6), and groupby takes the
+  dense fast path for float key columns whose values are integral and fit a
+  compact non-negative range.
 
-- **Chunked NDArray writes**: passing a ``blosc2.NDArray`` to ``extend()`` or
-  ``col[:] = arr`` now decompresses one chunk at a time instead of loading
-  the entire array into memory, keeping peak RSS bounded for large columns.
+- **Lighter imports**: the on-disk chunk prefetcher no longer uses asyncio,
+  so ``import blosc2`` skips ~30 asyncio modules and saves ~3 MB of memory.
+  A latent prefetcher deadlock on early iterator close was fixed as well.
 
-- **``BLOSC_ME_JIT`` full override**: the environment variable now takes
-  unconditional priority over both ``jit=`` and ``jit_backend=`` keyword
-  arguments, making backend switching from the command line effortless.
+- **New ``Column.raw`` accessor**: the underlying storage container
+  (``NDArray``, ``ListArray``, …) as a blosc2-native compressed object —
+  unlike ``col[...]`` reads, which always materialize NumPy arrays.  Useful
+  as a lazy-expression operand and for storage introspection.
 
-- **Correctness fixes**: a ``None == None`` guard bug that could corrupt rows
-  when writing to a view-backed column via the NDArray fast path; a missing
-  view guard in the new ``__setitem__``; and the fast path being silently
-  disabled for disk-opened tables have all been fixed.
+- **J2K and HTJ2K codecs**: ``blosc2.Codec.J2K`` and ``blosc2.Codec.HTJ2K``
+  expose the IDs for the new JPEG 2000 plugins (``pip install blosc2-j2k``
+  / ``pip install blosc2-htj2k``).  Also, C-Blosc2 has been updated to 3.1.3.
 
-A quick example of the new DSL computed column API::
+- **Fixes**: ``--float-trunc-prec`` in the ``parquet_to_blosc2`` CLI now
+  propagates to nested columns; unsupported computed-column expressions are
+  rejected early with an actionable error; and opening a ``.b2z``/``.b2d``
+  store in read mode no longer creates a temporary directory.
+
+A quick example of the new ``Column.raw`` accessor::
 
     import blosc2
     import dataclasses
-    import numpy as np
 
     @dataclasses.dataclass
     class Row:
         price: float = blosc2.field(blosc2.float64())
         qty:   int   = blosc2.field(blosc2.int64())
 
-    @blosc2.dsl_kernel
-    def revenue(price, qty):
-        return price * qty
-
     t = blosc2.CTable(Row, new_data=[(1.5, 10), (2.0, 5), (3.0, 3)])
-    t.add_computed_column("revenue", revenue, inputs=["price", "qty"])
-    print(t["revenue"][:])   # array([15., 10.,  9.])
 
-    # Column assignment with a blosc2.NDArray — written chunk-by-chunk
-    new_prices = blosc2.array([1.0, 2.5, 4.0])
-    t["price"] = new_prices
+    # The raw storage container, as a blosc2-native compressed object.
+    # It is over-allocated to chunk capacity, so slice to the live row count.
+    raw = t["price"].raw      # a blosc2.NDArray
+    print(raw[:len(t)])         # [1.5 2.  3. ]
+
+    # Usable directly as a lazy-expression operand, without decompressing
+    expr = raw * 2.0
+    print(expr[:len(t)])        # [3. 4. 6.]
 
 Install it with::
 

RELEASE_NOTES.md

@@ -2,7 +2,74 @@
 
 ## Changes from 4.4.2 to 4.4.3
 
-XXX version-specific blurb XXX
+This is a maintenance release focused on faster CTable cold-start, printing
+and groupby performance, a lighter `import blosc2`, new raw-storage access
+for columns, and support for the new J2K/HTJ2K codec plugins.
+
+### CTable performance
+
+- **Lazy column opening in views**: `select()` (and other view-producing
+  operations) no longer open every projected column up front.  A column is
+  only opened from storage when the view actually reads it, so selecting and
+  then touching a subset of columns — or aggregating a single one — skips
+  the cold-start cost of the rest.
+- **Lazy index opening in queries**: query planning no longer opens every
+  SUMMARY-indexed column on a wide persistent table; only indexes for
+  columns actually referenced by the predicate are loaded.
+- **Faster table printing**: `repr()`/`to_string()` now memoise per-column
+  sparse gathers for the duration of a render and combine the head and tail
+  rows into a single sparse read per column.  Each column is read from
+  storage once instead of ~6 times (precision detection, width sizing and
+  row rendering all hit the cache).
+- **Groupby with integral float keys**: float key columns whose values are
+  integral and fit a compact non-negative range (e.g. float32 id/second
+  columns) now take the dense single-key fast path instead of the markedly
+  slower generic float-hash path.  Fractional or non-finite keys fall back
+  automatically.
+- **No tempdir in read mode**: opening a `.b2z`/`.b2d` store in `'r'` mode
+  no longer creates a temporary working directory, since nothing is ever
+  written.
+
+### Lighter imports and prefetcher rework
+
+- **asyncio dependency dropped**: the on-disk chunk prefetcher used by the
+  UDF and numexpr fallback engines now uses plain `concurrent.futures`
+  instead of an asyncio event loop.  `import blosc2` no longer pulls in
+  ~30 asyncio modules, saving ~3 MB of memory footprint at import time.
+- **Prefetcher deadlock fixed**: an exception during evaluation could leave
+  the generator finalizer blocked forever in `thread.join()` while the
+  reader thread was stuck on a full prefetch queue.  A stop event now makes
+  the producer bail out when its consumer goes away.
+
+### New features
+
+- **`Column.raw` accessor**: returns the underlying storage container of a
+  column (`NDArray`, `ListArray`, `DictionaryColumn`, …) directly.  Unlike
+  `Column.__getitem__`, which always materializes NumPy arrays, this is the
+  column as a blosc2-native compressed object — usable as a lazy-expression
+  operand without decompressing, and exposing storage details like `schunk`,
+  `chunks` or `cparams`.  Note that this is a *physical* view: fixed-width
+  containers are over-allocated to chunk capacity, so slice to `len(table)`
+  to get just the live rows, and no validity-mask or null-sentinel
+  processing is applied.  Raises `AttributeError` for computed columns,
+  which have no backing storage.
+- **J2K and HTJ2K codec IDs**: `blosc2.Codec.J2K` and `blosc2.Codec.HTJ2K`
+  expose the IDs for the new JPEG 2000 codec plugins (installable with
+  `pip install blosc2-j2k` and `pip install blosc2-htj2k`).
+
+### Fixes
+
+- **`--float-trunc-prec` and nested columns**: the precision-truncation
+  filter of the `parquet_to_blosc2` CLI now propagates to float fields
+  inside nested (struct/list) columns too.
+- **Guard for unsupported computed-column expressions**: expressions that
+  would serialize to an empty or non-round-trippable string are now rejected
+  with an early, actionable `ValueError` at `add_computed_column()` time,
+  instead of silently breaking on reload.
+
+### Build
+
+- **C-Blosc2 updated to 3.1.3.**
 
 ## Changes from 4.4.1 to 4.4.2
 

doc/reference/ctable.rst

@@ -565,11 +565,13 @@ Data access
 .. autosummary::
 
     Column.view
+    Column.raw
     Column.take
     Column.iter_chunks
     Column.assign
 
 .. autoproperty:: Column.view
+.. autoproperty:: Column.raw
 .. automethod:: Column.take
 .. automethod:: Column.iter_chunks
 .. automethod:: Column.assign

pyproject.toml

@@ -40,7 +40,7 @@ dependencies = [
     "requests",
     "threadpoolctl; platform_machine != 'wasm32'",
 ]
-version = "4.4.3.dev0"
+version = "4.4.3"
 [project.entry-points."array_api"]
 blosc2 = "blosc2"
 

src/blosc2/ctable.py

@@ -819,13 +819,23 @@ def is_ndarray(self) -> bool:
         return col is not None and isinstance(col.spec, NDArraySpec)
 
     @property
-    def array(self):
+    def raw(self):
         """The underlying storage container for this column, without null-value processing.
 
         Returns the raw :class:`blosc2.NDArray`, :class:`~blosc2.ListArray`,
-        :class:`~blosc2.DictionaryColumn`, or scalar varlen array directly,
-        allowing bulk reads that bypass the null-sentinel scan in
-        :meth:`__getitem__`.
+        :class:`~blosc2.DictionaryColumn`, or scalar varlen array directly.
+        Unlike :meth:`__getitem__`, which always materializes NumPy arrays,
+        this is the column as a blosc2-native compressed object: usable as a
+        lazy-expression operand without decompressing, and exposing storage
+        details such as ``schunk``, ``chunks``, ``cparams`` or
+        ``iterchunks_info()``.
+
+        This is a physical view of the column: fixed-width containers are
+        over-allocated to chunk capacity for appends, so their first axis is
+        longer than ``len(column)`` and positions of rows deleted from the
+        table still hold their old values.  No validity-mask or null-sentinel
+        processing is applied; use the :class:`Column` interface for logical
+        reads.
 
         Raises :exc:`AttributeError` for computed (virtual) columns, which have
         no backing storage.

src/blosc2/version.py

@@ -1,2 +1,2 @@
-__version__ = "4.4.3.dev0"
+__version__ = "4.4.3"
 __array_api_version__ = "2024.12"

tests/ctable/test_column.py

@@ -1042,7 +1042,7 @@ class R:
 
 
 # -------------------------------------------------------------------
-# Column.array property
+# Column.raw property
 # -------------------------------------------------------------------
 
 
@@ -1056,40 +1056,43 @@ class ArrayRow:
     )
 
 
-def test_column_array_ndarray():
-    """array on a numeric column returns the underlying NDArray directly."""
+def test_column_raw_ndarray():
+    """raw on a numeric column returns the underlying NDArray directly."""
     t = CTable(Row, new_data=DATA20)
-    raw = t.id.array
+    raw = t.id.raw
     assert isinstance(raw, blosc2.NDArray)
     assert raw is t._cols["id"]
+    # Physical view: over-allocated to chunk capacity, so slice to live rows.
+    assert raw.shape[0] >= len(t)
+    np.testing.assert_array_equal(raw[: len(t)], np.arange(20, dtype=np.int64))
 
 
-def test_column_array_dictionary():
-    """array on a dictionary column returns the underlying DictionaryColumn."""
+def test_column_raw_dictionary():
+    """raw on a dictionary column returns the underlying DictionaryColumn."""
     from blosc2.dictionary_column import DictionaryColumn
 
     t = CTable(DictRow, new_data=[("acme", 1.0), ("globex", 2.0), ("acme", 3.0)])
-    raw = t.vendor.array
+    raw = t.vendor.raw
     assert isinstance(raw, DictionaryColumn)
     assert raw is t._cols["vendor"]
 
 
-def test_column_array_varlen_scalar():
-    """array on a varlen-string column returns the underlying _ScalarVarLenArray."""
+def test_column_raw_varlen_scalar():
+    """raw on a varlen-string column returns the underlying _ScalarVarLenArray."""
     from blosc2.scalar_array import _ScalarVarLenArray
 
     @dataclass
     class VLRow:
         note: str = blosc2.field(blosc2.vlstring())
 
     t = CTable(VLRow, new_data=[("hello",), ("world",)])
-    raw = t.note.array
+    raw = t.note.raw
     assert isinstance(raw, _ScalarVarLenArray)
     assert raw is t._cols["note"]
 
 
-def test_column_array_list():
-    """array on a list column returns the underlying ListArray."""
+def test_column_raw_list():
+    """raw on a list column returns the underlying ListArray."""
     from blosc2.list_array import ListArray
 
     t = CTable(
@@ -1099,28 +1102,28 @@ def test_column_array_list():
             (2.0, "b", "bar", ["z"]),
         ],
     )
-    raw = t.tags.array
+    raw = t.tags.raw
     assert isinstance(raw, ListArray)
     assert raw is t._cols["tags"]
 
 
-def test_column_array_computed_raises():
-    """array raises AttributeError for computed (virtual) columns."""
+def test_column_raw_computed_raises():
+    """raw raises AttributeError for computed (virtual) columns."""
     t = CTable(Row, new_data=DATA20)
     t.add_computed_column("double_score", lambda cols: cols["score"] * 2)
     with pytest.raises(AttributeError, match="computed"):
-        _ = t.double_score.array
+        _ = t.double_score.raw
 
 
-def test_column_array_bypasses_null_scan():
-    """array returns values that include the raw sentinel, not NaN-substituted values."""
+def test_column_raw_bypasses_null_scan():
+    """raw returns values that include the raw sentinel, not NaN-substituted values."""
 
     @dataclass
     class NullRow:
         x: float = blosc2.field(blosc2.float64(null_value=-1.0))
 
     t = CTable(NullRow, new_data=[(1.0,), (-1.0,), (3.0,)])
-    raw = t.x.array[:]
+    raw = t.x.raw[:]
     # The raw array contains the sentinel as-is, not converted to NaN.
     assert raw[1] == -1.0