Skip to content

Commit ecfa441

Browse files
Implement primitives for blockwise skeletonization (#67)
* Implement primitives for blockwise skeletonization * Fix skeleton test * Address claude code review
1 parent 6b1ccd6 commit ecfa441

18 files changed

Lines changed: 3881 additions & 147 deletions

File tree

CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ nanobind_add_module(_core
2323
src/bindings/mesh.cxx
2424
src/bindings/segmentation.cxx
2525
src/bindings/skeleton.cxx
26+
src/bindings/skeleton_distributed.cxx
2627
src/bindings/transformation.cxx
2728
src/bindings/util.cxx
2829
src/bindings/utils.cxx

MIGRATION_GUIDE.md

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2589,12 +2589,14 @@ Important differences and current scope:
25892589
distance-to-boundary field, a deterministic two-sweep root, a physical
25902590
Dijkstra distance-from-root field, penalized repeated Dijkstra paths, and
25912591
rolling invalidation with radius `scale * radius + constant`.
2592-
- This first correctness-oriented version uses a physical axis-aligned
2593-
invalidation cube. It does not implement kimimaro's soma handling, hole
2594-
filling, border stitching, manual targets, component dust filtering,
2595-
cross-section metadata, or postprocessing heuristics. These differences can
2596-
change branch positions and vertex counts, so output is not expected to be
2597-
vertex-for-vertex identical to kimimaro.
2592+
- This correctness-oriented implementation uses a physical axis-aligned
2593+
invalidation cube. The ordinary in-core entry points do not automatically
2594+
perform block stitching or accept manual targets; the dedicated block APIs
2595+
below provide mandatory interface targets and exact consolidation. Soma
2596+
handling, hole filling, component dust filtering, cross-section metadata,
2597+
and kimimaro's other postprocessing heuristics are not implemented. These
2598+
differences can change branch positions and vertex counts, so output is not
2599+
expected to be vertex-for-vertex identical to kimimaro.
25982600
- The C++ core remains dependency-free. Component discovery uses x-runs and a
25992601
union-find rather than dense component-label images. `number_of_threads=1`
26002602
is the default and `0` uses hardware concurrency; one shared budget covers
@@ -2604,14 +2606,76 @@ Important differences and current scope:
26042606
heap. Compact IDs avoid full-volume shortest-path fields; the public Dijkstra
26052607
functions remain dense and exact FP64.
26062608

2607-
Correctness tests are under `tests/skeleton/test_teasar.py` and
2608-
`tests/skeleton/test_teasar_labels.py`. The independent
2609+
### Blockwise skeletonization and exact stitching
2610+
2611+
`bioimage_cpp.skeleton.distributed` provides the dependency-light primitives
2612+
needed by an external block scheduler. It does not perform I/O, enumerate
2613+
blocks, launch tasks, or serialize artifacts. The implemented layout uses one
2614+
shared voxel plane: a core block with interval `[a, b)` is processed through
2615+
`b + 1` when it has a high-side neighbor, while the neighbor begins at `b`.
2616+
2617+
```python
2618+
dist = bic.skeleton.distributed
2619+
2620+
# `left` includes its high-side overlap plane and starts at `left_origin`.
2621+
targets = dist.block_border_targets(
2622+
left,
2623+
faces=[(2, "high")],
2624+
origin=left_origin,
2625+
spacing=spacing,
2626+
)
2627+
left_fragment = dist.block_teasar(
2628+
left,
2629+
open_faces=[(2, "high")],
2630+
origin=left_origin,
2631+
required_targets=targets,
2632+
spacing=spacing,
2633+
)
2634+
2635+
# Repeat independently for every processing block. Neighboring calls select
2636+
# identical global targets on their shared plane.
2637+
merged = dist.merge_block_skeletons(block_fragments)
2638+
forest = dist.minimum_spanning_forest(merged, spacing=spacing) # optional
2639+
vertices, edges, radii = dist.lattice_to_physical(forest, spacing=spacing)
2640+
```
2641+
2642+
Block fragments use global `int64` lattice vertices, `uint64` edges, and
2643+
`float32` physical radii. Integer coordinates are the exact merge identity;
2644+
physical proximity is never used to invent a connection. Duplicate radii are
2645+
reduced with `maximum`, and canonical sort-and-unique output makes
2646+
`merge_block_skeletons` associative, commutative, idempotent, and suitable for
2647+
hierarchical reduction. `merge_block_skeleton_maps` applies the same operation
2648+
per exact semantic label, including negative and large `uint64` labels.
2649+
2650+
`open_faces` is mandatory even when empty. It identifies artificial cuts where
2651+
the distance transform may look through the processing-block boundary; paths
2652+
remain restricted to real input foreground. This prevents interface radii from
2653+
collapsing to one voxel. When present, the lexicographically greatest required
2654+
target on an open face becomes the deterministic component root.
2655+
2656+
The labeled counterparts are `block_border_targets_labels` and
2657+
`block_teasar_labels`. Border targets are selected per 8-connected same-label
2658+
face patch by maximum anisotropic 2D distance transform with deterministic,
2659+
orientation-stable plateau resolution. Several valid targets can create cycles
2660+
after graph union, so exact consolidation keeps cycles and
2661+
`minimum_spanning_forest` removes them only when explicitly called. Blocked
2662+
skeletons are connected through their anchors but are not expected to be
2663+
vertex-for-vertex identical to whole-volume TEASAR because each block still has
2664+
less path-selection context.
2665+
2666+
Correctness tests are under `tests/skeleton/test_teasar.py`,
2667+
`tests/skeleton/test_teasar_labels.py`, and
2668+
`tests/skeleton/test_distributed.py`. The independent
26092669
Dijkstra benchmark is `development/distance/benchmark_dijkstra.py`; the
26102670
end-to-end benchmark is `development/skeleton/benchmark_teasar.py`. Use
26112671
`--suite all --kimimaro` to compare paired binary and multi-label scenarios,
26122672
`--memory` for fresh-process peak-RSS probes, or `--sequential-backends` for
26132673
the dense/compact FP64 binary design matrix. Extended density, spacing, and
26142674
PDRF regimes are in `development/skeleton/benchmark_teasar_sequential.py`.
2675+
The development-only serial block harness is
2676+
`development/skeleton/blockwise_stitching.py`; the rigorous one-thread layered
2677+
comparison against whole and blocked Kimimaro is
2678+
`development/skeleton/compare_blockwise_kimimaro.py`.
26152679

26162680
## I/O and Build Dependencies
26172681

development/skeleton/PERFORMANCE_NOTES.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -736,3 +736,58 @@ leave the initial linear mode. Skeleton output remained array-exact across
736736
backends and worker counts. Final verification after the optimization and
737737
benchmark addition: `1182 passed` with third-party pytest plugin autoload
738738
disabled.
739+
740+
## Blockwise open-boundary correctness and Kimimaro comparison (2026-07-16)
741+
742+
The first block prototype reused the ordinary per-component zero crop halo for
743+
its DBF. On an artificial cut this made every interface radius exactly one
744+
voxel. Thick branching tubes remained connected, but accumulated 2.1--3.1x the
745+
whole-volume Kimimaro cable length. A controlled Kimimaro run reproduced the
746+
failure when `black_border=True`, isolating the problem from target selection,
747+
exact consolidation, and cycle removal.
748+
749+
The corrected block path keeps its real foreground path mask unchanged and
750+
uses a separate EDT-only mask with one ghost layer across explicitly declared
751+
`open_faces`. One deterministic face target roots each affected component.
752+
Ghost voxels can never become graph vertices. The complete reference workflow
753+
is:
754+
755+
```bash
756+
python development/skeleton/compare_blockwise_kimimaro.py \
757+
--warmup 1 --repeats 7 --check \
758+
--json /tmp/blockwise-kimimaro.json
759+
```
760+
761+
The run used bioimage-cpp 0.7.0, Kimimaro 5.8.1, EDT 3.1.1, NumPy 2.4.6,
762+
SciPy 1.17.1, isotropic spacing, and one thread for every backend. Kimimaro's
763+
blocked fragments used `fix_borders=True`; both block backends then used the
764+
same exact lattice merge and minimum-spanning forest.
765+
766+
| case | bio block | Kimimaro whole | Kimimaro block | bio / Kimimaro-block cable | direct p95 / Hausdorff |
767+
| --- | ---: | ---: | ---: | ---: | ---: |
768+
| thin line, 16 blocks | 3.52 ms | 1.66 ms | 6.95 ms | 1.000 | 0.00 / 0.00 |
769+
| oblique tube `64^3`, anisotropic, 8 blocks | 4.41 ms | 11.44 ms | 17.78 ms | 1.000 | 0.00 / 0.00 |
770+
| tube `64^3`, 8 blocks | 6.73 ms | 18.34 ms | 46.97 ms | 0.988 | 0.20 / 1.41 |
771+
| tube `96^3`, 8 blocks | 13.04 ms | 48.95 ms | 72.01 ms | 0.916 | 1.41 / 3.74 |
772+
| tube `128^3`, seam-aligned, 8 blocks | 21.70 ms | 102.29 ms | 123.76 ms | 0.988 | 0.00 / 4.24 |
773+
| tube `128^3`, 27 blocks | 24.85 ms | 102.02 ms | 118.78 ms | 0.903 | 1.27 / 5.00 |
774+
775+
The thin line remains graph-exact. The anisotropic oblique case has identical
776+
blocked vertex geometry and cable length. All 112 expected local interface
777+
anchors were present in both controlled implementations, and their radii
778+
matched Kimimaro exactly. Across all thick cases, final graphs were connected
779+
and acyclic, cable length stayed within 10% of Kimimaro's blocked result, and
780+
every quality gate passed. On the nontrivial timing cases, bioimage-cpp was
781+
2.6--4.7x faster than whole-volume one-thread Kimimaro and 4.0--7.0x faster than
782+
its serial block workflow.
783+
784+
On the final `128^3` calls, the last measured phase breakdowns were:
785+
786+
| layout | targets | local TEASAR | merge | forest |
787+
| --- | ---: | ---: | ---: | ---: |
788+
| 8 blocks | 2.39 ms | 17.91 ms | 0.25 ms | 0.09 ms |
789+
| 27 blocks | 4.68 ms | 15.89 ms | 0.35 ms | 0.10 ms |
790+
791+
Merge and cycle removal remain negligible. More blocks mainly increase face
792+
analysis and Python orchestration. The comparison intentionally does not apply
793+
Kimimaro's heuristic nearest-component joining, dust removal, or tick pruning.
Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
"""Minimal serial harness for blockwise TEASAR and exact stitching.
2+
3+
This is deliberately development-only orchestration. It performs no I/O,
4+
parallel scheduling, retries, or artifact serialization; those responsibilities
5+
belong to downstream block-processing systems.
6+
"""
7+
8+
from __future__ import annotations
9+
10+
from collections.abc import Sequence
11+
12+
import numpy as np
13+
14+
import bioimage_cpp as bic
15+
16+
17+
dist = bic.skeleton.distributed
18+
19+
20+
def _processing_blocks(volume: np.ndarray, block_shape: Sequence[int]):
21+
blocking = bic.utils.Blocking([0, 0, 0], list(volume.shape), list(block_shape))
22+
for block_id in range(blocking.number_of_blocks):
23+
core = blocking.get_block(block_id)
24+
begin = [int(value) for value in core.begin]
25+
end = [int(value) for value in core.end]
26+
faces = []
27+
for axis in range(3):
28+
if blocking.get_neighbor_id(block_id, axis, lower=True) >= 0:
29+
faces.append((axis, "low"))
30+
if blocking.get_neighbor_id(block_id, axis, lower=False) >= 0:
31+
faces.append((axis, "high"))
32+
end[axis] += 1
33+
slices = tuple(slice(begin[axis], end[axis]) for axis in range(3))
34+
yield blocking, block_id, np.ascontiguousarray(volume[slices]), begin, faces
35+
36+
37+
def _unique_coordinates(parts: list[np.ndarray]) -> np.ndarray:
38+
if not parts:
39+
return np.empty((0, 3), dtype=np.int64)
40+
return np.unique(np.concatenate(parts, axis=0), axis=0)
41+
42+
43+
def _unique_target_map(parts: list[dict[int, np.ndarray]]) -> dict[int, np.ndarray]:
44+
labels = sorted({label for part in parts for label in part})
45+
return {
46+
label: _unique_coordinates([part[label] for part in parts if label in part])
47+
for label in labels
48+
}
49+
50+
51+
def _assert_matching_interfaces(blocking, per_face) -> None:
52+
for block_id in range(blocking.number_of_blocks):
53+
for axis in range(3):
54+
neighbor = blocking.get_neighbor_id(block_id, axis, lower=False)
55+
if neighbor < 0:
56+
continue
57+
left = per_face[(block_id, axis, "high")]
58+
right = per_face[(neighbor, axis, "low")]
59+
if isinstance(left, dict):
60+
if left.keys() != right.keys():
61+
raise AssertionError(
62+
f"target labels disagree across blocks {block_id}/{neighbor}"
63+
)
64+
for label in left:
65+
np.testing.assert_array_equal(left[label], right[label])
66+
else:
67+
np.testing.assert_array_equal(left, right)
68+
69+
70+
def run_blockwise_binary(
71+
mask: np.ndarray,
72+
block_shape: Sequence[int],
73+
*,
74+
spacing=(1.0, 1.0, 1.0),
75+
remove_cycles: bool = False,
76+
):
77+
"""Run the complete binary block pipeline serially and return a lattice graph."""
78+
fragments = []
79+
per_face = {}
80+
blocking = None
81+
for blocking, block_id, block, origin, faces in _processing_blocks(
82+
np.asarray(mask), block_shape
83+
):
84+
face_targets = []
85+
for axis, side in faces:
86+
targets = dist.block_border_targets(
87+
block,
88+
[(axis, side)],
89+
origin=origin,
90+
spacing=spacing,
91+
number_of_threads=1,
92+
)
93+
per_face[(block_id, axis, side)] = targets
94+
face_targets.append(targets)
95+
targets = _unique_coordinates(face_targets)
96+
fragments.append(
97+
dist.block_teasar(
98+
block,
99+
open_faces=faces,
100+
origin=origin,
101+
required_targets=targets,
102+
spacing=spacing,
103+
number_of_threads=1,
104+
)
105+
)
106+
if blocking is None:
107+
raise ValueError("mask must be a three-dimensional array")
108+
_assert_matching_interfaces(blocking, per_face)
109+
merged = dist.merge_block_skeletons(fragments)
110+
if remove_cycles:
111+
merged = dist.minimum_spanning_forest(merged, spacing=spacing)
112+
return merged
113+
114+
115+
def run_blockwise_labels(
116+
labels: np.ndarray,
117+
block_shape: Sequence[int],
118+
*,
119+
background: int = 0,
120+
spacing=(1.0, 1.0, 1.0),
121+
remove_cycles: bool = False,
122+
):
123+
"""Run the labeled block pipeline serially and return lattice graphs by label."""
124+
fragment_maps = []
125+
per_face = {}
126+
blocking = None
127+
for blocking, block_id, block, origin, faces in _processing_blocks(
128+
np.asarray(labels), block_shape
129+
):
130+
face_targets = []
131+
for axis, side in faces:
132+
targets = dist.block_border_targets_labels(
133+
block,
134+
[(axis, side)],
135+
origin=origin,
136+
background=background,
137+
spacing=spacing,
138+
number_of_threads=1,
139+
)
140+
per_face[(block_id, axis, side)] = targets
141+
face_targets.append(targets)
142+
targets = _unique_target_map(face_targets)
143+
fragment_maps.append(
144+
dist.block_teasar_labels(
145+
block,
146+
open_faces=faces,
147+
origin=origin,
148+
required_targets=targets,
149+
background=background,
150+
spacing=spacing,
151+
number_of_threads=1,
152+
)
153+
)
154+
if blocking is None:
155+
raise ValueError("labels must be a three-dimensional array")
156+
_assert_matching_interfaces(blocking, per_face)
157+
merged = dist.merge_block_skeleton_maps(fragment_maps)
158+
if remove_cycles:
159+
merged = {
160+
label: dist.minimum_spanning_forest(fragment, spacing=spacing)
161+
for label, fragment in merged.items()
162+
}
163+
return merged
164+
165+
166+
if __name__ == "__main__":
167+
example = np.zeros((17, 17, 25), dtype=np.uint8)
168+
example[8, 8, 2:23] = 1
169+
result = run_blockwise_binary(example, (9, 9, 8), remove_cycles=True)
170+
print(
171+
f"stitched {len(result[0])} lattice vertices and {len(result[1])} edges "
172+
"from serial processing blocks"
173+
)

0 commit comments

Comments
 (0)