Skip to content

Support zarr v3 - #4260

Open
alejoe91 wants to merge 40 commits into
SpikeInterface:mainfrom
alejoe91:zarrv3
Open

Support zarr v3#4260
alejoe91 wants to merge 40 commits into
SpikeInterface:mainfrom
alejoe91:zarrv3

Conversation

@alejoe91

@alejoe91 alejoe91 commented Dec 11, 2025

Copy link
Copy Markdown
Member

Fixes #4014

Implement bulk changes from zarr v2 to zarr v3

  • create_dataset -> create_array
  • Serialization of non JSON serializable objects (recording/sorting/extensions) not supported anymore
  • Compressors use zarr.codecs, which wrap numcodecs compressor (e.g., Blosc -> BloscCodec)
  • Codec Pipeline: improved generation of codec pipeline to automatically chain filters (ArrayArrayCodec), serializers (standard unless an ArrayBytesCodec is used - e.g. Wavpack), and compressors (BytesBytesCodec)
  • Clean up of tests

TODO:

  • Add sharding parameter to save function
  • Backcompatibility
  • Test v2 reading in v3: make an action that generates v2 zarr for Recording/Sorting/Template/SortingAnalyzer using an old SI version and reads it in the new version

Depends on:

@alejoe91 alejoe91 added dependencies Issue/PR that is related to dependencies core Changes to core module labels Dec 11, 2025
@alejoe91 alejoe91 closed this Mar 20, 2026
@alejoe91
alejoe91 deleted the zarrv3 branch March 20, 2026 09:41
@alejoe91
alejoe91 restored the zarrv3 branch March 20, 2026 09:41
@alejoe91 alejoe91 reopened this Mar 23, 2026
@alejoe91 alejoe91 added this to the 0.105.0 milestone Mar 24, 2026
Comment thread src/spikeinterface/core/zarrextractors.py
@alejoe91 alejoe91 mentioned this pull request Mar 26, 2026
@alejoe91
alejoe91 marked this pull request as ready for review June 24, 2026 11:21
Comment thread src/spikeinterface/core/zarrextractors.py Outdated
@chrishalcrow

Copy link
Copy Markdown
Member

I'm having issues loading principal_components, quality_metrics and template_metrics from a v2 analyzer. Looking into it...

Comment thread src/spikeinterface/core/sortinganalyzer.py
@alejoe91

Copy link
Copy Markdown
Member Author

I'm having issues loading principal_components, quality_metrics and template_metrics from a v2 analyzer. Looking into it...

We should add full analyzers in the cross-serialization checks!

@@ -3199,8 +3209,9 @@ def load_data(self, lazy=False):
extension_group = self._get_zarr_extension_group(mode="r")
for ext_data_name in extension_group.keys():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type of hack works for v2 backwards compat. Annoyingly, we don't necessarily know the names of the pc models apiori: they're called pca_model_by_channel_local_323 in my dataset - I guess 323 are the channel_indices...

Suggested change
for ext_data_name in extension_group.keys():
if self.extension_name =='template_metrics':
extension_group_keys = ['metrics', 'main_channel_templates', 'peaks_data']
elif self.extension_name =='quality_metrics':
extension_group_keys = ['metrics']
elif self.extension_name =='principal_components':
extension_group_keys = ['pca_projection']
else:
extension_group_keys = extension_group.keys()
for ext_data_name in extension_group_keys:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Changes to core module dependencies Issue/PR that is related to dependencies

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrating to zarr v3

2 participants