Merge branch 'main' into add-BaseRasterWidget

Author: chrishalcrow

.github/workflows/all-tests.yml

@@ -25,7 +25,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.12"]  # Lower and higher versions we support
+        python-version: ["3.9", "3.13"]  # Lower and higher versions we support
         os: [macos-latest, windows-latest, ubuntu-latest]
     steps:
       - uses: actions/checkout@v4

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ repos:
     -   id: end-of-file-fixer
     -   id: trailing-whitespace
 -   repo: https://github.com/psf/black
-    rev: 24.10.0
+    rev: 25.1.0
     hooks:
     -   id: black
         files: ^src/

doc/modules/core.rst

@@ -310,12 +310,13 @@ Since these core extensions are important for all other extensions it is importa
   and the time after (:code:`ms_after`) to ensure that you have a full waveform. Because waveforms occur on multiple channels with multiple
   samples this can be a big data structure.
 * :code:`templates` are calculated from the raw waveform data and are used for downstream analyses
-  (e.g. :code:`spike_amplitudes` are calculated based on the templates). This raises the question: if the :code:`templates` are what are used,
-  then why save the :code:`waveforms`? Well, there are two ways to obtain :code:`templates` data: 1) directly from the raw data (based on the
-  :code:`random_spikes`) or 2) from the :code:`waveforms`. When getting :code:`templates` from the raw data we are limited to obtaining averages
-  and standard deviations. If we calculate the templates from the waveforms, however, we can also calculate the templates as medians or percentiles
+  (e.g. :code:`spike_amplitudes` are calculated based on the templates). This raises the question: if the :code:`templates` are all we need,
+  then why compute and save the :code:`waveforms`? Well, there are two ways to obtain :code:`templates` data: 1) directly from the raw data (based on the
+  :code:`random_spikes`) or 2) from the :code:`waveforms` extension. When getting :code:`templates` from the raw data we are limited to obtaining averages
+  and standard deviations. If we calculate the templates from the waveforms, however, we can also calculate the template medians or percentiles
   in addition to the average or standard deviations of the :code:`waveforms`. So it is important to think about the type of downstream analyses that
-  you may want to do in deciding whether to calculate :code:`templates` with :code:`random_spikes` or using :code:`waveforms`.
+  you may want to do in deciding whether to calculate :code:`templates` with :code:`random_spikes` or using :code:`waveforms`. Note that if the :code:`waveforms`
+  extension is available, :code:`templates` will be calculated from it (the :code:`ms_before` and :code:`ms_after` parameters are also inherited from the :code:`waveforms`).
 * :code:`noise_levels` compute noise-levels in a channel-wise fashion. This provides important information about the specific recording session
   and is important for some downstream quality analyses.
 

doc/releases/0.102.0.rst

@@ -0,0 +1,155 @@
+.. _release0.102.0:
+
+SpikeInterface 0.102.0 release notes
+------------------------------------
+
+7th February 2025
+
+Main changes:
+
+* Added auto-label functions in curation module (#2918)
+* Refactored and improved auto-merge functions in curation module (#3435, #3601)
+* Added `spikeinterface.load()` function to load any SpikeInterface object (#3613, #3651)
+* Improved handling of time in base recording (#3509, #3623)
+* Multi-segment handling of motion interpolation (#3659)
+* Support for Numpy 2.0 and Zarr<3.0 (#3481,#3598)
+
+
+core:
+
+* Extend si.load capabilities (#3651)
+* Use process as default in global job kwargs (#3648)
+* Fix node pipeline when multiple retrievers (#3642)
+* Add `get_start_time`/`get_end_time` functions and use them in `get_duration` (#3623)
+* Fix dataframe index dtype due to csv in  quality metrics  (#3622)
+* Replace `load_extractor` with `load_function` from `loading.py` (#3613)
+* Adding SharedMemoryTemplates (#3580)
+* Implement engine="thread" in ChunkRecordingExecutor (#3526)
+* Add `shift start time` function. (#3509)
+* recording_slices in run_node_pipeline() (#3506)
+* Template loading average or median (#3495)
+* Refactor matching with nodepiepeline (#3447)
+* Improve noise level machinery (#3359)
+* nodepipeline : skip chunks when no peaks inside and skip_after_n_peaks (#3356)
+
+extractors:
+
+* Pin `pyedflib` version after breaking change in annotations (#3644)
+* Add experiment_name annotation to open ephys (#3624)
+* Add `NwbTimeSeriesExtractor` to load non-electrical series data from NWB (#3587)
+* Add `stream_mode` as extra_requirements for NWB when streaming (#3548)
+* Fix cbin_file_path (#3524)
+* Expose reading attempts in Plexon2 (#3401)
+* Add neuronexus allego recording Extractor (#3235)
+
+preprocessing:
+
+* Check for channel_indices is None (#3636)
+* Fix silence periods (#3557)
+* Extend whitening tests (#3531)
+* Don't let decimate mess with times and skim tests (#3519)
+* Whitening fix - compute covariance matrix in float (#3505)
+
+sorters:
+
+* Change to default value `use_binary_file=True` for KS4 (#3614)
+* Remove Non-Serializable writing of json (#3482)
+* Sc2 fixes (#3250, #3518)
+
+postprocessing:
+
+* Add a fast correlogram merge (#3607)
+* Unify template and quality metrics (#3537)
+* Only load `template_metrics` extension on compute if keeping some metrics (#3478)
+* Unit localization (#3476)
+* Optimizations for template_similarity (numba and dependencies) (#3405)
+
+qualitymetrics:
+
+* Hard code `synchony_sizes` (#3559)
+* Unify template and quality metrics (#3537)
+* Fix dtype of quality metrics before and after merging (#3497)
+
+curation:
+
+* Small fixes in curation format and `apply_curation()` (#3601)
+* Refactoring auto_merge (#3435)
+* Automatic curation with metrics  (#2918)
+
+widgets:
+
+* Make `time_range` conversion to array dtype `float64`  (#3638)
+* Fix `plot_sorting_summary` widget  (#3627)
+* Changes for spikeinterface-gui (#3616)
+* Patch to allow ipywidget plot_unit_templates to work with sparse templates (instead of analyzer) (#3575)
+* Allow `export_report` to run without waveforms (#3493)
+* Add vspacing_factor as a param for TracesWidget (#3461)
+
+generators:
+
+* Use strings as ids in generators (#3588)
+
+
+sortingcomponents:
+
+* Split PCA (#3610)
+* Adding Hanning filtering (#3604)
+* Fix use of keep during run_sorter benchmark (#3556)
+* Clustering components api (#3533)
+* Kilosort-matching in SpikeInterface (#3488)
+* Matched filtering with both peak signs simultaneously (#2914)
+
+motion correction:
+
+* Multi-segment handling in motion interpolation (#3659)
+* MotionEstimationStudy : plot drift with the scatter plot (#3553)
+* Fix a cross-band interpolation bug, and allow time_vector in interpolate_motion (#3517)
+
+documentation:
+
+* Fix plot_1_automated_curation.py (#3674)
+* Extend templates extension docs (#3663)
+* Use SpikeInterface-ported pre-trained models in doc (#3652)
+* Dev Doc touchup (#3641)
+* Fix precision/recall in hybrid example (#3635)
+* Improve docstring of `get_neuropixels_sample_shifts`  (#3620)
+* Switch RTD build to fail on gallery example failure (#3619)
+* Fix sphinx doc build (#3612)
+* Convert generated unit_ids in docs to strings (#3600)
+* Correct method default in docstring (#3570)
+* Add custom tutorials index page. (#3504)
+* Add error messaging around use of get data in templates (#3501)
+* Fix a few typos in the curation docs (#3496)
+* Improve error message for `set_probegroup`/`set_probe` (#3479)
+
+continuous integration:
+
+* Fix codecov tests (#3569)
+* Add python 313 to CI test (#3481)
+
+packaging:
+
+* Support numpy 2.0 (#3640)
+* Force latest hdmf-zarr version (#3639)
+* Update zarr version (#3598)
+
+testing:
+
+* Add python Numpy2.0 to CI test (#3481)
+
+Contributors:
+
+* @522848942
+* @JoeZiminski
+* @OlivierPeron
+* @alejoe91
+* @anoushkajain
+* @chrishalcrow
+* @cwindolf
+* @guptadivyansh
+* @h-mayorquin
+* @jakeswann1
+* @RobertoDF
+* @samuelgarcia
+* @yger
+* @zm711

doc/whatisnew.rst

@@ -8,6 +8,7 @@ Release notes
 .. toctree::
   :maxdepth: 1
 
+  releases/0.102.0.rst
   releases/0.101.2.rst
   releases/0.101.1.rst
   releases/0.101.0.rst

examples/tutorials/curation/plot_1_automated_curation.py

@@ -79,7 +79,7 @@
 # We can check that this is true by accessing the extension data.
 
 all_metric_names = list(sorting_analyzer.get_extension('quality_metrics').get_data().keys()) + list(sorting_analyzer.get_extension('template_metrics').get_data().keys())
-print(set(all_metric_names) == set(model.feature_names_in_))
+print(set(model.feature_names_in_).issubset(set(all_metric_names)))
 
 ##############################################################################
 # Great! We can now use the model to predict labels. Here, we pass the HF repo id directly

pyproject.toml

@@ -1,13 +1,13 @@
 [project]
 name = "spikeinterface"
-version = "0.102.0"
+version = "0.102.1"
 authors = [
   { name="Alessio Buccino", email="alessiop.buccino@gmail.com" },
   { name="Samuel Garcia", email="sam.garcia.die@gmail.com" },
 ]
 description = "Python toolkit for analysis, visualization, and comparison of spike sorting output"
 readme = "README.md"
-requires-python = ">=3.9,<3.13"
+requires-python = ">=3.9,<3.14"
 classifiers = [
     "Programming Language :: Python :: 3 :: Only",
     "License :: OSI Approved :: MIT License",
@@ -20,7 +20,8 @@ classifiers = [
 
 
 dependencies = [
-    "numpy>=1.20",
+    "numpy>=1.20;python_version<'3.13'",
+    "numpy>=2.0.0;python_version>='3.13'",
     "threadpoolctl>=3.0.0",
     "tqdm",
     "zarr>=2.18,<3",
@@ -164,15 +165,6 @@ test = [
     # for sortingview backend
     "sortingview",
 
-
-    ## install tridesclous for testing ##
-    "tridesclous>=1.6.8",
-
-    ## sliding_nn
-    "pymde",
-    "torch",
-    "pynndescent",
-
     # curation
     "skops",
     "huggingface_hub",

src/spikeinterface/__init__.py

@@ -1,7 +1,4 @@
-"""
-
-
-"""
+""" """
 
 import importlib.metadata
 

src/spikeinterface/benchmark/benchmark_base.py

@@ -430,7 +430,7 @@ def load_folder(cls, folder):
 
                 result[k] = load(folder / k)
             elif format == "Motion":
-                from spikeinterface.sortingcomponents.motion import Motion
+                from spikeinterface.core.motion import Motion
 
                 result[k] = Motion.load(folder / k)
             elif format == "zarr_templates":

src/spikeinterface/benchmark/benchmark_motion_estimation.py

@@ -8,15 +8,15 @@
 import numpy as np
 
 from spikeinterface.core import get_noise_levels
-from .benchmark_base import Benchmark, BenchmarkStudy
-from .benchmark_plot_tools import _simpleaxis
+from spikeinterface.core.motion import Motion
 from spikeinterface.sortingcomponents.motion import estimate_motion
 from spikeinterface.sortingcomponents.peak_detection import detect_peaks
 from spikeinterface.sortingcomponents.peak_selection import select_peaks
 from spikeinterface.sortingcomponents.peak_localization import localize_peaks
 from spikeinterface.widgets import plot_probe_map
 
-from spikeinterface.sortingcomponents.motion import Motion
+from .benchmark_base import Benchmark, BenchmarkStudy
+from .benchmark_plot_tools import _simpleaxis
 
 # import MEArec as mr