Merge branch 'main' into token

Author: zm711

doc/api.rst

@@ -76,7 +76,7 @@ Low-level
     .. autoclass:: ChunkRecordingExecutor
 
 
-Back-compatibility with ``WaveformExtractor`` (version < 0.101.0)
+Back-compatibility with ``WaveformExtractor`` (version > 0.100.0)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. automodule:: spikeinterface.core
@@ -179,6 +179,8 @@ spikeinterface.preprocessing
     .. autofunction:: correct_motion
     .. autofunction:: get_motion_presets
     .. autofunction:: get_motion_parameters_preset
+    .. autofunction:: load_motion_info
+    .. autofunction:: save_motion_info
     .. autofunction:: depth_order
     .. autofunction:: detect_bad_channels
     .. autofunction:: directional_derivative

doc/get_started/import.rst

@@ -73,7 +73,7 @@ For example:
 .. code-block:: python
 
     from spikeinterface.preprocessing import bandpass_filter, common_reference
-    from spikeinterface.core import extract_waveforms
+    from spikeinterface.core import create_sorting_analyzer
     from spikeinterface.extractors import read_binary
 
 As mentioned this approach only imports exactly what you plan on using so it is the most minimalist. It does require

doc/get_started/install_sorters.rst

@@ -12,7 +12,7 @@ and in many cases the easiest way to run them is to do so via Docker or Singular
 **This is the approach we recommend for all users.**
 To run containerized sorters see our documentation here: :ref:`containerizedsorters`.
 
-There are some cases where users will need to install the spike sorting algorithms in their own environment. If you
+There are some cases where you will need to install the spike sorting algorithms on your own computer. If you
 are on a system where it is infeasible to run Docker or Singularity containers, or if you are actively developing the
 spike sorting software, you will likely need to install each spike sorter yourself.
 
@@ -24,7 +24,7 @@ opencl (Tridesclous) to use hardware acceleration (GPU).
 Here is a list of the implemented wrappers and some instructions to install them on your local machine.
 Installation instructions are given for an **Ubuntu** platform. Please check the documentation of the different spike
 sorters to retrieve installation instructions for other operating systems.
-We use **pip** to install packages, but **conda** should also work in many cases.
+We use **pip** to install packages, but **conda** or **uv** should also work in many cases.
 
 Some novel spike sorting algorithms are implemented directly in SpikeInterface using the
 :py:mod:`spikeinterface.sortingcomponents` module. Checkout the :ref:`get_started/install_sorters:SpikeInterface-based spike sorters` section of this page
@@ -140,10 +140,12 @@ Kilosort4
 
 * Python, requires CUDA for GPU acceleration (highly recommended)
 * Url: https://github.com/MouseLand/Kilosort
-* Authors: Marius Pachitariu, Shashwat Sridhar, Carsen Stringer
+* Authors: Marius Pachitariu, Shashwat Sridhar, Carsen Stringer, Jacob Pennington
 * Installation::
 
-      pip install kilosort==4.0 torch
+      pip install kilosort
+      pip uninstall torch
+      pip install torch --index-url https://download.pytorch.org/whl/cu118
 
 * For more installation instruction refer to https://github.com/MouseLand/Kilosort
 
@@ -240,7 +242,7 @@ Waveclus
 * Also supports Snippets (waveform cutouts) objects (:py:class:`~spikeinterface.core.BaseSnippets`)
 * Url: https://github.com/csn-le/wave_clus/wiki
 * Authors: Fernando Chaure, Hernan Rey and Rodrigo Quian Quiroga
-* Installation needs Matlab::
+* Installation requires Matlab::
 
       git clone https://github.com/csn-le/wave_clus/
       # provide installation path by setting the WAVECLUS_PATH environment variable
@@ -270,7 +272,7 @@ with SpikeInterface.
 SpykingCircus2
 ^^^^^^^^^^^^^^
 
-This is a upgraded version of SpykingCircus, natively written in SpikeInterface.
+This is an upgraded version of SpykingCircus, natively written in SpikeInterface.
 The main differences are located in the clustering (now using on-the-fly features and less prone to finding
 noise clusters), and in the template-matching procedure, which is now a fully orthogonal matching pursuit,
 working not only at peak times but at all times, recovering more spikes close to noise thresholds.
@@ -289,7 +291,7 @@ Tridesclous2
 ^^^^^^^^^^^^
 
 This is an upgraded version of Tridesclous, natively written in SpikeInterface.
-#Same add his notes.
+
 
 * Python
 * Requires: HDBSCAN and Numba
@@ -314,7 +316,7 @@ Klusta (LEGACY)
 * Authors: Cyrille Rossant, Shabnam Kadir, Dan Goodman, Max Hunter, Kenneth Harris
 * Installation::
 
-       pip install Cython h5py tqdm
+       pip install cython h5py tqdm
        pip install click klusta klustakwik2
 
 * See also: https://github.com/kwikteam/phy

doc/get_started/installation.rst

@@ -81,14 +81,16 @@ Requirements
 
   * numpy
   * probeinterface
-  * neo>=0.9.0
-  * joblib
+  * neo
   * threadpoolctl
   * tqdm
+  * zarr
+  * pydantic
+  * numcodecs
+  * packaging
 
 Sub-modules have more dependencies, so you should also install:
 
-  * zarr
   * h5py
   * scipy
   * pandas
@@ -98,8 +100,13 @@ Sub-modules have more dependencies, so you should also install:
   * matplotlib
   * numba
   * distinctipy
+  * skops
+  * huggingface_hub
   * cuda-python (for non-macOS users)
 
 
+For developers we offer a :code:`[dev]` option which installs testing, documentation, and linting packages necessary
+for testing and building the docs.
+
 All external spike sorters can be either run inside containers (Docker or Singularity - see :ref:`containerizedsorters`)
 or must be installed independently (see :ref:`get_started/install_sorters:Installing Spike Sorters`).

doc/how_to/handle_drift.rst

@@ -1322,8 +1322,6 @@ A preset is a nested dict that contains theses methods/parameters.
 Run motion correction with one function!
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Correcting for drift is easy! You just need to run a single function. We
-will try this function with some presets.
 
 Here we also save the motion correction results into a folder to be able
 to load them later.

doc/how_to/read_various_formats.rst

@@ -0,0 +1,173 @@
+.. code:: ipython3
+
+    %matplotlib inline
+
+Read various format into SpikeInterface
+=======================================
+
+SpikeInterface can read various formats of “recording” (traces) and
+“sorting” (spike train) data.
+
+Internally, to read different formats, SpikeInterface either uses: \* a
+wrapper to ``neo <https://github.com/NeuralEnsemble/python-neo>``\ \_
+rawio classes \* or a direct implementation
+
+Note that:
+
+- file formats contain a “recording”, a “sorting”, or “both”
+- file formats can be file-based (NWB, …) or folder based (SpikeGLX,
+  OpenEphys, …)
+
+In this example we demonstrate how to read different file formats into
+SI
+
+.. code:: ipython3
+
+    import matplotlib.pyplot as plt
+
+    import spikeinterface.core as si
+    import spikeinterface.extractors as se
+
+Let’s download some datasets in different formats from the
+``ephy_testing_data <https://gin.g-node.org/NeuralEnsemble/ephy_testing_data>``\ \_
+repo:
+
+- MEArec: a simulator format which is hdf5-based. It contains both a
+  “recording” and a “sorting” in the same file.
+- Spike2: file from spike2 devices. It contains “recording” information
+  only.
+
+.. code:: ipython3
+
+
+    spike2_file_path = si.download_dataset(remote_path="spike2/130322-1LY.smr")
+    print(spike2_file_path)
+
+    mearec_folder_path = si.download_dataset(remote_path="mearec/mearec_test_10s.h5")
+    print(mearec_folder_path)
+
+
+.. parsed-literal::
+
+    Downloading data from 'https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/raw/master/mearec/mearec_test_10s.h5' to file '/Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/mearec/mearec_test_10s.h5'.
+
+
+.. parsed-literal::
+
+     modified: spike2/130322-1LY.smr (file)
+    1 annex'd file (15.8 MB recorded total size)
+    /Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/spike2/130322-1LY.smr
+    1 annex'd file (59.4 MB recorded total size)
+    nothing to save, working tree clean
+
+
+.. parsed-literal::
+
+    100%|█████████████████████████████████████| 62.3M/62.3M [00:00<00:00, 76.7GB/s]
+
+
+.. parsed-literal::
+
+    /Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/mearec/mearec_test_10s.h5
+
+
+Now that we have downloaded the files, let’s load them into SI.
+
+The :py:func:``~spikeinterface.extractors.read_spike2`` function returns
+one object, a :py:class:``~spikeinterface.core.BaseRecording``.
+
+Note that internally this file contains 2 data streams (‘0’ and ‘1’), so
+we need to specify which one we want to retrieve (‘0’ in our case). the
+stream information can be retrieved by using the
+:py:func:``~spikeinterface.extractors.get_neo_streams`` function.
+
+.. code:: ipython3
+
+    stream_names, stream_ids = se.get_neo_streams("spike2", spike2_file_path)
+    print(stream_names)
+    print(stream_ids)
+    stream_id = stream_ids[0]
+    print("stream_id", stream_id)
+
+    recording = se.read_spike2(spike2_file_path, stream_id="0")
+    print(recording)
+    print(type(recording))
+    print(isinstance(recording, si.BaseRecording))
+
+
+.. parsed-literal::
+
+    ['Signal stream 0', 'Signal stream 1']
+    ['0', '1']
+    stream_id 0
+    Spike2RecordingExtractor: 1 channels - 20833.333333 Hz - 1 segments - 4,126,365 samples
+                              198.07s (3.30 minutes) - int16 dtype - 7.87 MiB
+      file_path: /Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/spike2/130322-1LY.smr
+    <class 'spikeinterface.extractors.neoextractors.spike2.Spike2RecordingExtractor'>
+    True
+
+
+The
+:py:func::literal:`~spikeinterface.extractors.read_spike2`\` function is equivalent to instantiating a :py:class:`\ ~spikeinterface.extractors.Spike2RecordingExtractor\`
+object:
+
+.. code:: ipython3
+
+    recording = se.read_spike2(spike2_file_path, stream_id="0")
+    print(recording)
+
+
+.. parsed-literal::
+
+    Spike2RecordingExtractor: 1 channels - 20833.333333 Hz - 1 segments - 4,126,365 samples
+                              198.07s (3.30 minutes) - int16 dtype - 7.87 MiB
+      file_path: /Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/spike2/130322-1LY.smr
+
+
+The :py:func:``~spikeinterface.extractors.read_mearec`` function returns
+two objects, a :py:class:``~spikeinterface.core.BaseRecording`` and a
+:py:class:``~spikeinterface.core.BaseSorting``:
+
+.. code:: ipython3
+
+    recording, sorting = se.read_mearec(mearec_folder_path)
+    print(recording)
+    print(type(recording))
+    print()
+    print(sorting)
+    print(type(sorting))
+
+
+
+.. parsed-literal::
+
+    MEArecRecordingExtractor: 32 channels - 32.0kHz - 1 segments - 320,000 samples - 10.00s
+                              float32 dtype - 39.06 MiB
+      file_path: /Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/mearec/mearec_test_10s.h5
+    <class 'spikeinterface.extractors.neoextractors.mearec.MEArecRecordingExtractor'>
+
+    MEArecSortingExtractor: 10 units - 1 segments - 32.0kHz
+      file_path: /Users/christopherhalcrow/spikeinterface_datasets/ephy_testing_data/mearec/mearec_test_10s.h5
+    <class 'spikeinterface.extractors.neoextractors.mearec.MEArecSortingExtractor'>
+
+
+SI objects (:py:class:``~spikeinterface.core.BaseRecording`` and
+:py:class:``~spikeinterface.core.BaseSorting``) can be plotted quickly
+with the :py:mod:``spikeinterface.widgets`` submodule:
+
+.. code:: ipython3
+
+    import spikeinterface.widgets as sw
+
+    w_ts = sw.plot_traces(recording, time_range=(0, 5))
+    w_rs = sw.plot_rasters(sorting, time_range=(0, 5))
+
+    plt.show()
+
+
+
+.. image:: read_various_formats_files/read_various_formats_12_0.png
+
+
+
+.. image:: read_various_formats_files/read_various_formats_12_1.png

doc/how_to/read_various_formats_files/read_various_formats_12_0.png

@@ -276,10 +276,8 @@ The :py:func:`~spikeinterface.comparison.compare_two_sorters()` returns the comp
     import spikeinterface.comparisons as sc
     import spikinterface.widgets as sw
 
-    # First, let's download a simulated dataset
-    local_path = si.download_dataset(remote_path='mearec/mearec_test_10s.h5')
-    recording, sorting = se.read_mearec(local_path)
-
+    # First, let's generate a simulated dataset
+    recording, sorting = si.generate_ground_truth_recording()
     # Then run two spike sorters and compare their outputs.
     sorting_HS = ss.run_sorter(sorter_name='herdingspikes', recording=recording)
     sorting_TDC = ss.run_sorter(sorter_name='tridesclous', recording=recording)
@@ -332,9 +330,8 @@ Comparison of multiple sorters uses the following procedure:
 
 .. code-block:: python
 
-    # Download a simulated dataset
-    local_path = si.download_dataset(remote_path='mearec/mearec_test_10s.h5')
-    recording, sorting = se.read_mearec(local_path)
+    # Generate a simulated dataset
+    recording, sorting = si.generate_ground_truth_recording()
 
     # Then run 3 spike sorters and compare their outputs.
     sorting_MS4 = ss.run_sorter(sorter_name='mountainsort4', recording=recording)