Add PDZ signal compression as a user-selectable alternative to VBZ

[GuilleDufortFing]

Summary

Adds PDZ (Piecewise-Differential-Zstd), a lossless signal codec that can be
selected as an alternative to VBZ.

How it works: delta-code the int16 signal, split the Rice-mapped differences into
three fixed-width bit-planes, then Zstd-compress the two larger planes. Scalar, SSE,
and NEON kernels all produce byte-identical output.

The new on-disk extension type is minknow.pdz (large binary), sitting alongside
minknow.vbz. As with VBZ, the codec applies to the whole file.

Using it

• C API: PDZ_SIGNAL_COMPRESSION
• Python: pod5.Writer(..., signal_compression_type=pod5.SignalType.PdzSignal)
• pybind: SignalType.PdzSignal

Readers detect and decode PDZ automatically, and pod5 inspect reports it.

import pod5
with pod5.Writer("out.pod5", signal_compression_type=pod5.SignalType.PdzSignal) as w:
    ...

Performance

PDZ targets a better compression ratio than VBZ while being faster, particularly with
SIMD: it ships scalar, SSE, and NEON kernels (VBZ has no NEON path). Full benchmarks
are reported in the publication referenced below.

Testing

• Round-trip and cross-architecture byte-identity tests (scalar vs SSE vs NEON).
• Compression-adapter, schema, and C-API round-trip tests, plus a fuzz target for the
  decode path.
• Built and tested on x86 (gcc) and Arm64 (clang).

Reference

The PDZ algorithm is described in Bioinformatics Advances (2026):
https://doi.org/10.1093/bioadv/vbag157

[Copilot]

Pull request overview

Adds a new lossless signal compression option (PDZ / minknow.pdz) alongside VBZ, wiring it through the C++ core, C API, Python bindings, CLI inspection output, documentation, tests, and fuzzing.

Changes:

• Introduces the PDZ codec implementation (scalar/SSE/NEON) and C++ adapter APIs, plus schema/extension-type support (minknow.pdz).
• Exposes PDZ as a user-selectable compression option via C API, pybind enum, and Python pod5.Writer, and teaches readers/pod5 inspect to detect/report it.
• Adds broad test coverage (codec/adapter/schema/cross-arch/C-API/Python) and a decode-path fuzz target.

Reviewed changes

Copilot reviewed 52 out of 52 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
python/pod5/src/tests/test_api.py	Adds PDZ round-trip and pod5 inspect reporting tests; relaxes compression-ratio assertion for PDZ.
python/pod5/src/pod5/writer.py	Updates Writer docs/type hints to include PDZ as a supported signal compression type.
python/pod5/src/pod5/tools/pod5_inspect.py	Extends summary output to report PDZ compression.
python/pod5/src/pod5/signal_tools.py	Adds Python-level PDZ decompression helpers wired to pybind.
python/pod5/src/pod5/reader.py	Adds PDZ signal decoding path and PDZ/VBZ detection via Arrow extension metadata.
python/lib_pod5/src/lib_pod5/pod5_format_pybind.pyi	Exposes PDZ compress/decompress/max-size functions in typing stubs.
python/lib_pod5/src/lib_pod5/init.py	Re-exports PDZ pybind functions for Python consumers.
fuzz/fuzz_pdz.cpp	Adds a PDZ adapter fuzz target (round-trip + corrupt-input safety).
fuzz/CMakeLists.txt	Registers the new pdz fuzzer build target.
docs/tables/signal.toml	Documents minknow.pdz as an allowed logical type for the signal column.
docs/SPECIFICATION.md	Specifies the minknow.pdz on-disk extension type and its storage.
CHANGELOG.md	Adds an Unreleased entry documenting PDZ availability and APIs.
c++/test/signal_table_tests.cpp	Extends signal table tests to cover PDZ and adds PDZ dispatch validation.
c++/test/pdz_schema_tests.cpp	Adds PDZ extension-type registration and schema round-trip tests.
c++/test/pdz_cross_arch_tests.cpp	Adds cross-backend byte-identity and decode-compatibility tests (scalar/SSE/NEON).
c++/test/pdz_compression_tests.cpp	Adds adapter-level PDZ round-trip and corrupt-input rejection tests.
c++/test/pdz_codec_tests.cpp	Adds codec-level PDZ encode/decode and encode_bound tests.
c++/test/CMakeLists.txt	Adds PDZ test sources and links zstd for codec tests.
c++/test/c_api_tests.cpp	Adds C-API selectable compression scenario including PDZ.
c++/pod5_format/types.h	Introduces PdzSignalType/PdzSignalArray and pdz_signal() accessor.
c++/pod5_format/types.cpp	Implements PDZ extension type/array and registers/unregisters it.
c++/pod5_format/signal_table_utils.h	Extends SignalType enum with PdzSignal (appended for ABI stability).
c++/pod5_format/signal_table_schema.cpp	Adds schema creation/dispatch support for SignalType::PdzSignal.
c++/pod5_format/signal_table_reader.h	Adds typed accessor for the PDZ signal column.
c++/pod5_format/signal_table_reader.cpp	Adds PDZ extraction, byte-count, and decode dispatch in the reader.
c++/pod5_format/signal_builder.h	Adds PDZ signal builder variant and PDZ in-place compression path.
c++/pod5_format/pdz/utils.hpp	Adds small PDZ utility helpers (round-up division, compile-time assertion macro).
c++/pod5_format/pdz/simd_backend.hpp	Defines PDZ backend selection enum (Native/Scalar/Sse/Neon).
c++/pod5_format/pdz/rice_map_sse.hpp	Implements Rice/zig-zag mapping helpers for SSE + scalar fallback.
c++/pod5_format/pdz/rice_map_neon.hpp	Implements Rice/zig-zag mapping helpers for NEON.
c++/pod5_format/pdz/pdz.hpp	Umbrella include for PDZ codec headers.
c++/pod5_format/pdz/memory_utils.hpp	Adds aligned scratch allocation helpers used by the codec.
c++/pod5_format/pdz/codecs.hpp	Adds fixed-width integer (de)serialization helpers for PDZ headers.
c++/pod5_format/pdz/codec/piecewise_split_sse.hpp	SSE kernel for piecewise split/merge with byte-identical layout.
c++/pod5_format/pdz/codec/piecewise_split_serial.hpp	Shared scalar tail for non-multiple-of-block sizes.
c++/pod5_format/pdz/codec/piecewise_split_scalar.hpp	Portable reference kernel used as the cross-arch oracle.
c++/pod5_format/pdz/codec/piecewise_split_neon.hpp	NEON kernel for piecewise split/merge with byte-identical layout.
c++/pod5_format/pdz/codec/pdz_codec.hpp	Header-only PDZ codec implementation (block header + zstd integration).
c++/pod5_format/pdz/codec/buffer_u_writer.hpp	Writes the B_u bitplane in the layout matching SIMD kernels.
c++/pod5_format/pdz/codec/buffer_sizes.hpp	Defines buffer sizing formulas for PDZ split buffers.
c++/pod5_format/pdz/codec/buffer_readers.hpp	Readers for B_u/B_l/B_h layouts used during decode.
c++/pod5_format/pdz/codec/buffer_reader_types.hpp	Helper types for reading packed B_u lanes.
c++/pod5_format/pdz/codec/buffer_lh_writer.hpp	Writer for B_l/B_h byte-plane buffers in SIMD-matching layout.
c++/pod5_format/pdz/buffered_skip_bit_output_stream.hpp	Bitstream helper for B_u interleaved packing.
c++/pod5_format/pdz/attributes.hpp	Defines a portable pdz_restrict qualifier.
c++/pod5_format/pdz_compression.h	Declares public PDZ adapter APIs (max size, compress, decompress).
c++/pod5_format/pdz_compression.cpp	Implements PDZ adapter APIs with structural validation for safe decode.
c++/pod5_format/c_api.h	Adds PDZ_SIGNAL_COMPRESSION option to public C API.
c++/pod5_format/c_api.cpp	Maps the new C API option to internal SignalType::PdzSignal.
c++/pod5_format_pybind/bindings.cpp	Exposes PDZ enum value and PDZ compress/decompress/max-size to Python.
c++/pod5_format_pybind/api.h	Implements pybind wrappers for PDZ adapter functions.
c++/CMakeLists.txt	Adds PDZ sources/headers to the build and installs PDZ headers.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[0x55555555]

Thank you @GuilleDufortFing !