Getting ready for release 4.4.2

Author: FrancescAlted

ANNOUNCE.rst

@@ -1,69 +1,57 @@
-Announcing Python-Blosc2 4.4.1
+Announcing Python-Blosc2 4.4.2
 ==============================
 
-We are happy to announce this feature release that brings an interactive
-TUI data viewer, automatic SUMMARY indexes for fast WHERE queries,
-chunk-aligned Arrow/Parquet imports, expanded ``where()`` acceleration, and a
-range of CTable ergonomics and performance improvements.
+We are happy to announce this feature and maintenance release that promotes
+DSL kernels to first-class CTable computed columns, adds a convenient new
+column-assignment API, speeds up bulk NDArray writes, and fixes several
+correctness issues.
 
 The main highlights are:
 
-- **New ``b2view`` interactive viewer**: a terminal-based viewer for all blosc2
-  containers (``NDArray``, ``CTable``, ``SChunk``, ``BatchArray``, …), launched
-  with ``b2view <file>`` or ``blosc2.b2view()``.  Supports full 1-D/2-D/N-D
-  browsing, ``CTable`` row navigation, a vlmeta pane, and keyboard shortcuts.
+- **DSL kernels as first-class CTable columns**: ``@blosc2.jit``-decorated
+  functions can now back virtual computed columns, stored generated columns,
+  and ``where()`` filter predicates directly — in addition to the existing
+  string-expression form.  Columns survive save/open round-trips via
+  persisted source.
 
-- **Automatic SUMMARY indexes**: when a ``CTable`` is closed after writing,
-  SUMMARY indexes (per-block min/max) are built by default for all eligible
-  scalar columns.  They are accumulated *incrementally* during writes so the
-  close step adds almost no extra cost.  At query time, a block-skip prefilter
-  uses these bitmaps to skip blocks that cannot satisfy the WHERE predicate,
-  reducing decompression work for selective queries.
+- **New ``t["col"] = arr`` assignment**: a clean shorthand for overwriting all
+  live rows of a column.  Accepts any array-like, including ``blosc2.NDArray``.
 
-- **Chunk-aligned Arrow/Parquet imports**: fixed-size columns are now written on
-  a shared chunk/block grid and incoming batches are buffered to exact chunk
-  boundaries, so every chunk is compressed exactly once.  Dictionary columns are
-  imported in bulk.  A new ``--reduce-mem`` CLI flag caps Arrow read-batch size
-  for memory-constrained nested imports.
+- **Chunked NDArray writes**: passing a ``blosc2.NDArray`` to ``extend()`` or
+  ``col[:] = arr`` now decompresses one chunk at a time instead of loading
+  the entire array into memory, keeping peak RSS bounded for large columns.
 
-- **``where()`` and miniexpr acceleration**: single- and two-argument ``where``
-  calls are now dispatched directly to miniexpr, avoiding numexpr overhead.
-  Sparse boolean masks trigger a fast gather path, and a new pre-check skips
-  per-chunk numexpr setup when the condition is trivially true or false.
+- **``BLOSC_ME_JIT`` full override**: the environment variable now takes
+  unconditional priority over both ``jit=`` and ``jit_backend=`` keyword
+  arguments, making backend switching from the command line effortless.
 
-- **``CTable.copy()`` enhancements**: a new C-level bulk copy path
-  (``chunk_copy()``) transfers pre-compressed chunks without
-  serialization/recompression.  ``copy()`` now accepts ``chunks``, ``blocks``,
-  and ``cparams`` overrides; the ``parquet-to-blosc2`` CLI gains ``--chunks``
-  and ``--blocks`` flags.
+- **Correctness fixes**: a ``None == None`` guard bug that could corrupt rows
+  when writing to a view-backed column via the NDArray fast path; a missing
+  view guard in the new ``__setitem__``; and the fast path being silently
+  disabled for disk-opened tables have all been fixed.
 
-- **``sort_by()`` on views is now lazy**: sorting a filtered view returns a
-  position-reordered view whose columns are read in sorted order without a full
-  materialization pass.
+A quick example of the new DSL computed column API::
 
-- **``context manager`` support for ``blosc2.open()``**: all objects returned by
-  ``blosc2.open()`` now support the ``with`` statement for clean flush-and-close
-  semantics.
-
-- **``NestedColumn`` public class**: the dotted-column accessor is now a proper
-  public class with aggregate metadata (``nbytes``, ``cbytes``, ``cratio``) and
-  a structured ``.info`` report.
-
-- **Python 3.10 dropped**: Python 3.11 is now the minimum supported version.
+    import blosc2
+    import dataclasses
+    import numpy as np
 
-A small example showing the new SUMMARY index benefit::
+    @dataclasses.dataclass
+    class Row:
+        price: float = blosc2.field(blosc2.float64())
+        qty:   int   = blosc2.field(blosc2.int64())
 
-    import blosc2
+    @blosc2.dsl_kernel
+    def revenue(price, qty):
+        return price * qty
 
-    # Create a table and let SUMMARY indexes be built automatically on close
-    t = blosc2.CTable(Row, urlpath="my_table.b2d", mode="w")
-    t.extend(data)
-    t.close()   # SUMMARY indexes built here
+    t = blosc2.CTable(Row, new_data=[(1.5, 10), (2.0, 5), (3.0, 3)])
+    t.add_computed_column("revenue", revenue, inputs=["price", "qty"])
+    print(t["revenue"][:])   # array([15., 10.,  9.])
 
-    # Re-open and run a selective WHERE query — block skipping kicks in
-    t = blosc2.open("my_table.b2d")
-    result = t.where(t.value > 0.99)
-    print(result[:])
+    # Column assignment with a blosc2.NDArray — written chunk-by-chunk
+    new_prices = blosc2.array([1.0, 2.5, 4.0])
+    t["price"] = new_prices
 
 Install it with::
 

RELEASE_NOTES.md

@@ -2,7 +2,89 @@
 
 ## Changes from 4.4.1 to 4.4.2
 
-XXX version-specific blurb XXX
+This is a feature and maintenance release that promotes DSL kernels to
+first-class CTable computed columns, adds a new `CTable.__setitem__`
+assignment idiom, optimises bulk NDArray writes, and fixes several
+correctness issues.
+
+### DSL kernels as first-class CTable columns
+
+- **`add_computed_column()` accepts DSL kernels**: `@blosc2.dsl_kernel`-decorated
+  functions can now back virtual computed columns directly, in addition to
+  the existing string-expression form.  The column survives save/open
+  round-trips via persisted `dsl_source`.
+- **`add_generated_column()` accepts DSL kernels**: stored generated columns
+  (written during `append`/`extend` and on `refresh_generated_column()`)
+  now support DSL kernels as their transformer.
+- **`CTable.where()` accepts UDF/DSL kernels**: filter predicates are no
+  longer limited to expression strings — any DSL kernel can be passed directly.
+- **`dtype` inference for DSL kernels**: when `dtype` is omitted,
+  `lazyudf()` infers the output dtype via NumPy type promotion of the input
+  column dtypes.  Pass `dtype` explicitly for type-changing kernels
+  (comparisons, casts).
+- **`kernel_from_source()` utility**: new `dsl_kernel.kernel_from_source()`
+  reconstructs a `DSLKernel` from its stored source text, shared by the
+  CTable DSL-column loaders and the persisted `LazyUDF` decoder.
+- **Security note**: `.b2d` files from untrusted sources that contain DSL
+  computed columns execute stored Python source on open.  A warning is now
+  included in the documentation.
+
+### New `CTable.__setitem__` column-assignment API
+
+- **`t["col"] = arr`**: new shorthand equivalent to `t["col"][:] = arr`.
+  Accepts any array-like including `blosc2.NDArray`.  Raises `KeyError` for
+  unknown columns and `ValueError` for views or read-only tables.
+
+### Chunked NDArray writes in `extend()` and `Column.__setitem__`
+
+- **`extend({"col": ndarray})` decompresses chunk-by-chunk**: when a
+  `blosc2.NDArray` is passed as a column value to `extend()`, it is now
+  written in chunks instead of being fully decompressed upfront.  Pass
+  `validate=False` to avoid a transient full decompression during constraint
+  checking.
+- **`col[:] = blosc2_ndarray` fast path**: a new no-holes fast path in
+  `Column.__setitem__` skips the O(n) validity-mask gather and writes the
+  NDArray one chunk at a time using contiguous slice writes.  Works for both
+  scalar and fixed-shape ndarray columns.  Falls back to a chunked fancy-index
+  path when deleted rows are present.
+
+### `BLOSC_ME_JIT` environment variable override
+
+- **Full CLI override**: `BLOSC_ME_JIT` now takes unconditional priority over
+  both the `jit=` and `jit_backend=` keyword arguments, making it easy to
+  switch JIT backends from the command line without modifying code.
+
+### Correctness fixes
+
+- **View corruption in `Column.__setitem__`**: a `None == None` guard
+  evaluation on view-backed columns could fire the NDArray fast path,
+  bypassing physical-position remapping and silently corrupting rows.  Fixed
+  by explicitly checking `base is None` before activating the fast path.
+- **`CTable.__setitem__` view guard**: the new `t["col"] = arr` API now
+  raises `ValueError` on views, matching the contract of all other mutating
+  CTable methods.
+- **Fast path enabled for disk-opened tables**: the fast path previously
+  remained dormant for tables opened from disk because `_last_pos` starts as
+  `None`.  The guard now calls `_resolve_last_pos()` to lazily initialise it.
+- **DSL column `jit_backend` preserved in `_empty_copy`**: the `jit_backend`
+  setting was silently dropped during internal table copies; it is now
+  retained.
+- **`lazyexpr` Column unwrapping**: `convert_inputs()` now automatically
+  unwraps `CTable.Column` objects to their backing NDArray so that shape and
+  identity checks work correctly.
+
+### Documentation and examples
+
+- **Parquet-to-blosc2 walkthrough**: new step-by-step tutorial added to the
+  getting-started section. Thanks to @SyedIshmumAhnaf.
+- **CTable performance tips**: new section in the overview covering when to
+  prefer computed vs. generated columns, chunk sizing, and query optimisation.
+- **Simplified docstring examples**: examples throughout `ndarray.py` and
+  `ctable.py` now use `blosc2.array()`, `blosc2.arange()`, and
+  `blosc2.linspace()` directly instead of two-step numpy-then-`asarray`
+  patterns.
+- **`udf-computed-col.py` example**: new end-to-end example demonstrating DSL
+  kernel computed and generated columns.
 
 ## Changes from 4.3.3 to 4.4.1
 

pyproject.toml

@@ -40,7 +40,7 @@ dependencies = [
     "requests",
     "threadpoolctl; platform_machine != 'wasm32'",
 ]
-version = "4.4.2.dev0"
+version = "4.4.2"
 [project.entry-points."array_api"]
 blosc2 = "blosc2"
 

src/blosc2/version.py

@@ -1,2 +1,2 @@
-__version__ = "4.4.2.dev0"
+__version__ = "4.4.2"
 __array_api_version__ = "2024.12"