docs: 📝 document the in-place merge, empty-chunk skipping and block sizing

Covers the second review round: the occupancy map now lives beside
image.zarr and needs a tile-derived block (a block as coarse as the tile
makes every tile test occupied); chunks holding no labels are skipped in both
the boundary scan and the relabel, so background reaches neither disk nor
memory; and the workflow stages into the label group and relabels it in
place, taking three passes over the label volume down to two.

The restart trade-off is stated rather than glossed: in place, a job killed
mid-relabel leaves a half-relabelled array, where a separate output store
left the input intact to redo from. The label group is unregistered until the
merge finishes, so NGFF readers do not list a partial result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: lguerard

docs/guide/merging.md

@@ -28,16 +28,23 @@ This is the same approach used by
 
 ### Step 1: stage
 
-Each tile's labels are written to a temporary zarr once. This is critical:
-without staging, any downstream operation that reads the label array re-runs
-your segmentation function. The merge internally reads labels multiple times.
+Each tile's labels are written to zarr once. This is critical: without
+staging, any downstream operation that reads the label array re-runs your
+segmentation function. The merge internally reads labels multiple times.
 
 ```text
 tile_process calls fn once per tile → staged zarr
                                          │
                          merge reads from staged zarr (no fn calls)
 ```
 
+The Snakemake workflow goes further and stages **directly into**
+`image.zarr/labels/<name>/0`, then has the merge rewrite that array in place
+— saving a whole extra write of the volume plus the scratch store's disk. It
+falls back to a separate store when the tile is larger than the label chunk
+cap, since in place the chunking cannot be changed and level 0 has to stay
+pageable for a viewer.
+
 ### Step 2: make the ids globally unique
 
 Tiles write local `1..n`, which collide, so the boundary scan could not
@@ -56,7 +63,10 @@ Only the two voxels on either side of each tile boundary are read. For any
 pair of touching non-zero labels `(a, b)`, they must be the same object. The
 per-tile offsets are applied here, on the fly.
 
-I/O cost: `O(n_boundaries × face_area)`, not `O(full_volume)`.
+I/O cost: `O(n_boundaries × face_area)`, not `O(full_volume)`. The columns
+are read in parallel, and a boundary next to a chunk that holds no labels is
+skipped outright — a pair needs a non-zero label on *both* sides, so it could
+never produce one.
 
 ### Step 4: connected components
 
@@ -77,6 +87,17 @@ The LUT is applied to every tile in parallel via `multiprocessing.Pool`. The
 LUT is shared via process initializer to avoid re-pickling it for every chunk
 (LUTs can be hundreds of MB for dense label volumes).
 
+Chunks whose tile wrote no labels are skipped entirely — not read, and not
+written. Zarr never materialises an unwritten chunk and reads it back as the
+fill value, so background regions cost neither I/O nor disk. On a sparse
+image that is most of the volume.
+
+When the merge's output *is* its input, this pass rewrites the array in
+place. That is safe because the boundary scan (step 3) has already finished,
+so nothing still needs the original ids. The trade-off is restartability: a
+job killed part-way through leaves a half-relabelled array, where a separate
+output store would have left the input intact to redo from.
+
 ## Using the merge step standalone
 
 You can call the merge step directly on any existing label array or zarr:

docs/guide/skip_empty.md

@@ -63,14 +63,23 @@ that beside `image.zarr` (about 1/16384 of the image at the default 128 px
 block). `tile_occupancy` then reduces the map over each tile's full footprint.
 
 ```python
-from patchworks import auto_empty_threshold, build_occupancy_map, tile_occupancy
+from patchworks import (
+    auto_empty_threshold, block_for_tile, build_occupancy_map, tile_occupancy
+)
 
-build_occupancy_map("image.zarr")             # once per image, all channels
+# Size the block from the tile, or every tile over-covers the same block and
+# tests occupied — correct, but useless as a skip list.
+build_occupancy_map("image.zarr", block=block_for_tile(TILE))
 info = tile_occupancy(
     "image.zarr", TILE, channel=0, threshold=auto_empty_threshold(img, 0, 0)
 )
 ```
 
+The map is stored **beside** the image (`image.occupancy.zarr`), not inside
+it: it is not an NGFF array, and a zarr hierarchy containing one cannot be
+walked. It is rebuilt automatically if an existing map was built at a
+different block.
+
 This is **exact, not approximate**: `block_max > threshold` is true exactly
 when some voxel in that block exceeds the threshold, so comparing pooled
 maxima against a threshold derived from raw voxels answers the same question
@@ -81,6 +90,10 @@ a three-config run pays for one pooling pass instead of three sampling passes.
 This is what the Snakemake workflow uses, and it builds the map on first use
 for stores converted before the map existed.
 
+Skipping a tile pays off twice over: it is never segmented, **and** the merge
+skips its chunk too — neither reading it nor writing zeros over it, so the
+background never reaches disk at all.
+
 ## Threshold selection
 
 ```python

docs/guide/snakemake.md

@@ -242,8 +242,8 @@ result. Keep `mtime` and reruns happen only when an output is missing or stale.
 
 ## Running two segmentations (e.g. nuclei + cytoplasm)
 
-Every path the workflow writes — `tiles.json`, `stage.zarr`, per-tile `seg/`,
-the cached model, `labels.done` — lives under `work_dir/<label_name>/`, so
+Every path the workflow writes — `tiles.json`, per-batch `seg/` markers, the
+cached model, `labels.done` — lives under `work_dir/<label_name>/`, so
 running the workflow **twice with two configs against the same `work_dir`**
 never collides: each run gets its own private subdirectory, and both reuse
 the *same* already-converted `image.zarr` (conversion never re-runs).
@@ -489,7 +489,14 @@ merged = merge_tile_labels("stage.zarr", input_component="staged",
 write_labels("image.zarr", merged, name="cells")
 ```
 
-The workflow goes one step further and points the merge straight at
-`image.zarr/labels/cells/0` (via `write_to=` + `output_component="0"`), then
-calls `register_labels` to add the pyramid — skipping the scratch store and
-the full-volume copy that `write_labels` would otherwise do.
+The workflow goes two steps further. It stages **into**
+`image.zarr/labels/cells/0` in the first place, then has the merge rewrite
+that array **in place** (`write_to=` and `input_component=`/
+`output_component=` all pointing at it) and calls `register_labels` to add
+the pyramid. That removes both the scratch store and the full-volume copy
+`write_labels` would otherwise do — three passes over the label volume become
+two.
+
+It falls back to the separate store above when the tile is larger than the
+label chunk cap, because in place the chunking cannot be changed and level 0
+has to stay pageable for a viewer.

workflow/README.md

@@ -29,9 +29,15 @@ calibrated, multi-scale `labels/` group.
    image), create the empty `stage.zarr`, and group the tiles into batches.
 3. **segment {batch}** — one GPU job per `tiles_per_job` tiles, processed
    sequentially so they share a single CUDA init and model load: read tile +
-   halo, run Cellpose, trim, write to the stage. Scattered across the cluster.
-4. **merge** — zarr-native boundary stitch + renumber, written straight into
-   `image.zarr/labels/<name>/0`, then pyramided.
+   halo, run Cellpose, trim, write into `image.zarr/labels/<name>/0` (or a
+   scratch store for oversized tiles). Scattered across the cluster.
+4. **merge** — zarr-native boundary stitch + renumber, applied **in place**
+   to that array, then pyramided. Chunks holding no labels are skipped
+   entirely, so background costs neither I/O nor disk.
+
+The label group is only registered in `labels/.zattrs` once the merge
+finishes, so a run that dies midway leaves a group NGFF readers won't list;
+re-running recreates it.
 
 ## Install