Getting ready for release 4.8.1

Author: FrancescAlted

ANNOUNCE.rst

@@ -1,55 +1,43 @@
-Announcing Python-Blosc2 4.8.0
+Announcing Python-Blosc2 4.8.1
 ==============================
 
-We are happy to announce this release, which brings support for
-**sharing containers safely across processes** — opt-in file locking for
-``SChunk``/``NDArray``/``EmbedStore``/``DictStore``, atomic archive
-replacement, and readers that follow another process's writes — plus a
-couple of data-loss and data-correctness bug fixes worth upgrading for even
-if you don't touch any of the new locking API.
+This is a maintenance release with a solid batch of bug fixes — including
+use-after-free hazards around zero-copy cframes, wrong-chunk deletion with
+negative-step slices, and inconsistent ``DictStore`` overwrite semantics —
+plus read-only memory mapping for ``CTable`` stores and a documentation
+restructuring with a new Optimization Tips section.
 
 The main highlights are:
 
-- **Cross-process locking**: a new ``locking`` storage parameter (and the
-  ``BLOSC_LOCKING`` env var to enable it fleet-wide) serializes accesses to an
-  on-disk ``SChunk``/``NDArray``/``EmbedStore``/``DictStore`` against other
-  handles and processes via a small sidecar lock file. ``holding_lock()``
-  holds the exclusive lock across several operations and now auto-refreshes
-  the handle right after acquiring it, so a decision made inside the block
-  never acts on a stale in-memory read.
-
-- **Cross-process writes for ``EmbedStore``/``DictStore``** (``.b2d``): under
-  locking, one process can add or remove keys while another has the store
-  open, and the other side's next lookup sees the change — no need to
-  reopen the store.
-
-- **Atomic ``.b2z`` archives**: writing a ``DictStore.to_b2z()`` file (which
-  ``TreeStore`` also uses) now swaps the new file in atomically. A process
-  reading the archive concurrently always gets either the complete old
-  version or the complete new one — never a partially-written file from a
-  save that's still in progress. This needs no locking on the reader's side.
-
-- **Growth-SWMR**: a reader ``NDArray`` handle opened before a ``resize()``
-  made through another handle follows the new shape on its next data access,
-  or via the new explicit ``NDArray.refresh()`` / ``SChunk.refresh()``.
-
-- **Two bug fixes worth knowing about**: ``NDArray.append()`` could silently
-  delete another writer's just-appended data under concurrent growth (fixed
-  by refreshing the cached shape before computing the resize target); and
-  ``detect_aligned_chunks()`` could silently return the wrong chunk's data
-  for an aligned slice on arrays whose shape isn't a multiple of the chunk
-  shape (a floor-division bug that undercounted the chunk grid).
-
-- New user guide page,
-  `Sharing containers across processes
-  <https://www.blosc.org/python-blosc2/guides/sharing_across_processes.html>`_,
-  covering all of the above plus the caveats (NFS, ``mmap_mode``, Windows
-  in-use-file rename).
-
-A quick taste — hold the lock across a read-modify-write from two processes::
-
-    with ndarr.holding_lock():
-        ndarr[:] = ndarr[:] + 1   # atomic w.r.t. other locked handles
+- **Read-only mmap for ``CTable`` stores**: ``CTable.open()`` gains an
+  ``mmap_mode="r"`` parameter, mirroring ``blosc2.open()``. All members of a
+  read-only store — scalar, list, varlen and dictionary columns alike — are
+  read from mapped pages; for ``.b2z`` archives, in place inside the single
+  mapped container file. With several concurrent readers on one file this
+  pays off quickly: 2.5x/4.4x/4.5x faster wall time for 1/4/8 readers in
+  our benchmark.
+
+- **Zero-copy cframe fix**: ``schunk_from_cframe()`` /
+  ``ndarray_from_cframe()`` with ``copy=False`` (the default) returned
+  objects pointing into the caller's bytes buffer without keeping it alive,
+  so a temporary cframe could be reclaimed under the live object, corrupting
+  reads. The buffer is now pinned on the returned object.
+
+- **More correctness fixes**: negative-step slice deletion in
+  ``BatchArray``/``ObjectArray`` removed the wrong chunks; ``DictStore``
+  overwrite semantics depended on value size (now uniformly dict-like);
+  ``stack()``/``vecdot()`` shape inference was off for negative axes;
+  chunked ``matmul()`` mishandled broadcast batch dims; and
+  ``ListArray.extend_arrow()`` could reorder unflushed rows.
+
+- **Faster ``.b2z``/``.b2d`` opens**: a builtin-shadowing bug made store
+  detection in ``blosc2.open()`` silently recurse ~250 times on every open.
+
+- **Docs restructuring**, with a new `Optimization tips
+  <https://www.blosc.org/python-blosc2/guides/optimization_tips.html>`_
+  section, including tips on grouping related data into a single
+  memory-mapped ``.b2z`` file and on using ``mmap_mode="r"`` with many
+  concurrent readers.
 
 Install it with::
 

RELEASE_NOTES.md

@@ -2,7 +2,62 @@
 
 ## Changes from 4.8.0 to 4.8.1
 
-XXX version-specific blurb XXX
+### Improvements
+
+- Read-only memory mapping for `CTable` stores: `CTable.open()` (and
+  `FileTableStorage`) gain an `mmap_mode="r"` parameter, mirroring
+  `blosc2.open()`. All members of a read-only store — scalar, list, varlen
+  and dictionary columns alike — are then read from mapped pages; for `.b2z`
+  archives, in place at their offsets inside the single mapped container
+  file. With several concurrent readers on one file this pays off quickly:
+  2.5x/4.4x/4.5x faster wall time for 1/4/8 readers in our benchmark
+  (`bench/optim_tips/tip_10_mmap_many_readers.py`).
+- Reduced memory consumption in `CTable.extend()` when passed an `NDArray`.
+
+### Bug fixes
+
+- Fixed lifetime/use-after-free hazards around zero-copy cframes and
+  `vlmeta`: `schunk_from_cframe()`/`ndarray_from_cframe()` with `copy=False`
+  (the default) returned objects pointing into the caller's bytes buffer
+  without keeping it alive, so a temporary cframe (e.g.
+  `ndarray_from_cframe(response.content)`) could be reclaimed under the live
+  object, corrupting reads. The buffer is now pinned on the returned object.
+  Also, `vlmeta` read paths (`__getitem__`/`__len__`/`__iter__`) now raise
+  `ReferenceError` on an orphaned owner instead of segfaulting, matching the
+  write paths.
+- `BatchArray.delete()` / `ObjectArray.delete()`: negative-step slices
+  (e.g. `del arr[3:0:-1]`) deleted chunks in ascending order, shifting the
+  indices of chunks still to be deleted and removing the wrong ones (or
+  raising `RuntimeError`).
+- `ListArray.extend_arrow()`: Arrow chunks were appended to the backend
+  without flushing pending cells first, reordering unflushed rows after the
+  new ones.
+- Shape inference: `stack()` with a negative axis inserted the new dimension
+  one position too early, so a lazyexpr's reported `.shape` disagreed with
+  its computed result; `vecdot()` also normalized positive axes as if they
+  were negative.
+- Chunked `matmul()`: broadcast (size-1) operand batch dims were sliced with
+  the result-chunk coordinates, producing empty slices when the broadcast
+  dim spans several result chunks.
+- `DictStore.__setitem__()`: overwrite semantics depended on value size —
+  embedded keys refused overwrite ("already exists"), while an
+  embedded-to-external overwrite double-stored the key and resurrected the
+  stale embedded value after a delete. Assignments now behave uniformly
+  dict-like, dropping any previous value.
+- Explicitly passing `cparams=None` or `dparams=None` to NDArray
+  constructors crashed; both now mean "defaults".
+- Fixed a builtin-shadowing bug that made store detection in
+  `blosc2.open()` recurse ~250 times (silently swallowed) on every
+  `.b2z`/`.b2d` open; opening a `.b2z` is now ~10x faster under allocation
+  tracing.
+
+### Documentation
+
+- Restructured docs (#674), with a new
+  [Optimization tips](https://www.blosc.org/python-blosc2/guides/optimization_tips.html)
+  section, including new tips on grouping related data into a single
+  memory-mapped `.b2z` file and on using `mmap_mode="r"` with many
+  concurrent readers.
 
 ## Changes from 4.7.0 to 4.8.0
 

pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "rich",
     "threadpoolctl; platform_machine != 'wasm32'",
 ]
-version = "4.8.1.dev0"
+version = "4.8.1"
 [project.entry-points."array_api"]
 blosc2 = "blosc2"
 

src/blosc2/version.py

@@ -1,2 +1,2 @@
-__version__ = "4.8.1.dev0"
+__version__ = "4.8.1"
 __array_api_version__ = "2024.12"