Merge branch 'main' into add-tetrodes-doc

Author: chrishalcrow

doc/api.rst

@@ -176,13 +176,16 @@ spikeinterface.preprocessing
     .. autofunction:: clip
     .. autofunction:: common_reference
     .. autofunction:: correct_lsb
+    .. autofunction:: compute_motion
     .. autofunction:: correct_motion
     .. autofunction:: get_motion_presets
     .. autofunction:: get_motion_parameters_preset
     .. autofunction:: load_motion_info
     .. autofunction:: save_motion_info
     .. autofunction:: depth_order
     .. autofunction:: detect_bad_channels
+    .. autofunction:: detect_and_interpolate_bad_channels
+    .. autofunction:: detect_and_remove_bad_channels
     .. autofunction:: directional_derivative
     .. autofunction:: filter
     .. autofunction:: gaussian_filter

doc/how_to/index.rst

@@ -18,5 +18,6 @@ Guides on how to solve specific, short problems in SpikeInterface. Learn how to.
     auto_curation_training
     auto_curation_prediction
     physical_units
+    unsigned_to_signed
     customize_a_plot
     ../tutorials/forhowto/plot_1_working_with_tetrodes

doc/how_to/physical_units.rst

@@ -1,3 +1,5 @@
+.. _physical_units:
+
 Working with physical units in SpikeInterface recordings
 ========================================================
 

doc/how_to/unsigned_to_signed.rst

@@ -0,0 +1,102 @@
+Unsigned to Signed Data types
+=============================
+
+As of version 0.103.0 SpikeInterface has changed one of its defaults for interacting with
+:code:`Recording` objects. We no longer autocast unsigned dtypes to signed implicitly. This
+means that some users of SpikeInterface will need to add one additional line of code to their scripts
+to explicitly handle this conversion.
+
+
+Why this matters?
+-----------------
+
+For those that want a deeper understanding of dtypes `NumPy provides a great explanation <https://numpy.org/doc/stable/reference/arrays.dtypes.html>`_.
+For our purposes it is important to know that many pieces of recording equipment opt to store their electrophysiological data as unsigned integers
+(e.g., Intan, Maxwell Biosystems, 3Brain Biocam).
+Similarly to signed integers, in order to convert to real units these file formats only need to store a :code:`gain`
+and an :code:`offset`. Our :code:`RecordingExtractor`'s maintain the dtype that the file format utilizes, which means that some of our
+:code:`RecordingExtractor`'s will have unsigned dtypes.
+
+The problem with using unsigned dtypes is that many types of functions (including the ones we use from :code:`SciPy`) perform poorly with unsigned integers.
+This is made worse by the fact that these failures are silent (i.e. no error is triggered but the operation leads to nonsensical data). So the
+solution required is to convernt unsigned integers into signed integers. Previously we did this under the hood, automatically for users that had
+a :code:`Recording` object with an unsigned dtype.
+
+We decided, however, that implicitly performing this action was not the best course of action, since:
+
+1) *explicit* is always better than *implicit*
+2) some functions would *magically* change the dtype of the :code:`Recording` object, which can cause confusion
+
+So from version 0.103.0, users will now explicitly have to perform this transformation of their data. This will help users better understand how they are
+processing their data during an analysis pipeline as well as better understand the provenance of their pipeline.
+
+
+Using :code:`unsigned_to_signed`
+--------------------------------
+
+For users that receive an error because their :code:`Recording` is unsigned, their is one additional step that must be done:
+
+.. code:: python
+
+    import spikeinterface.extractors as se
+    import spikeinterface.preprocessing as spre
+
+    # Intan is an example of unsigned data
+    recording = se.read_intan('path/to/my/file.rhd', stream_id='0')
+    # to get a signed version of our Recording we use the following function
+    recording_signed = spre.unsigned_to_signed(recording)
+    # we can now apply any preprocessing functions like normal, e.g.
+    recording_filtered = spre.bandpass_filter(recording_signed)
+
+
+Now with the signed dtype of the :code:`Recording` one can use a SpikeInterface pipeline as usual.
+
+
+If you are curious if your :code:`Recording` is unsigned you can simply check the repr or use :code:`get_dtype()`
+
+.. code:: python
+
+    # the repr automatically displays the dtype
+    print(recording)
+    # use method on the Recording object
+    print(recording.get_dtype())
+
+In either case, if the dtype displayed has a :code:`u` at the beginning (e.g. :code:`uint16`) then your recording is
+unsigned. If it doesn't have the :code:`u` (e.g. :code:`int16`) then it is signed and would not need this preprocessing step.
+
+
+Bit depth
+---------
+
+One final important piece of information for some users is the concept of bit depth, which is the number of bits used to
+sample the data. The :code:`bit_depth` argument that can be fed into the :code:`unsigned_to_signed` function.
+This should be used in cases where the ADC bit depth does not match the bit depth of the data type (e.g., if the data is
+stored as :code:`uint16` but the ADC is 12 bits).
+Let's make a concrete example: the Biocam acquisition system from 3Brain uses a 12-bit ADC and stores the data as
+:code:`uint16`. This means that the data is stored in a 16-bit unsigned integer format, but the actual data
+only covers a 12-bit range. Therefore, that the "zero" of the data is not at 0, nor at half of the :code:`uint16` range (i.e. 2^15),
+but rather at 2048 (i.e., 2^12).
+In this case, setting the :code:`bit_depth` argument to 12 will allow the :code:`unsigned_to_signed` function to
+correctly convert the unsigned data to signed data and offset the data to be centered around 0, by subtracting 2048
+while converting the data from unsigned to signed.
+
+.. code:: python
+
+    recording_unsigned = se.read_biocam('path/to/my/file.brw')
+    # we can now convert to signed with the correct bit depth
+    recording_signed = spre.unsigned_to_signed(recording_unsigned, bit_depth=12)
+
+
+Additional Notes
+----------------
+
+1) Some sorters make use of SpikeInterface preprocessing either
+   within their wrappers or within their own code base. So remember to use the "signed" version of
+   your recording for the rest of your pipeline.
+
+2) Using :code:`unsigned_to_signed` in versions less than 0.103.0 does not hurt your scripts. This
+   option was available previously along with the implicit option. Adding this into scripts with old
+   versions of SpikeInterface will still work and will "future-proof" your scripts for when you
+   update to a version greater than or equal to 0.103.0.
+
+3) For additional information on units and scaling in SpikeInterface see :ref:`physical_units`.

doc/modules/sorters.rst

@@ -469,6 +469,7 @@ Here is the list of external sorters accessible using the run_sorter wrapper:
 * **Klusta** :code:`run_sorter(sorter_name='klusta')`
 * **Mountainsort4** :code:`run_sorter(sorter_name='mountainsort4')`
 * **Mountainsort5** :code:`run_sorter(sorter_name='mountainsort5')`
+* **RT-Sort** :code:`run_sorter(sorter_name='rt-sort')`
 * **SpyKING Circus** :code:`run_sorter(sorter_name='spykingcircus')`
 * **Tridesclous** :code:`run_sorter(sorter_name='tridesclous')`
 * **Wave clus** :code:`run_sorter(sorter_name='waveclus')`

doc/references.rst

@@ -43,6 +43,7 @@ please include the appropriate citation for the :code:`sorter_name` parameter yo
 - :code:`herdingspikes` [Muthmann]_ [Hilgen]_
 - :code:`kilosort`  [Pachitariu]_
 - :code:`mountainsort` [Chung]_
+- :code:`rt-sort` [van_der_Molen]_
 - :code:`spykingcircus` [Yger]_
 - :code:`wavclus` [Chaure]_
 - :code:`yass` [Lee]_
@@ -154,6 +155,8 @@ References
 
 .. [UMS] `UltraMegaSort2000 - Spike sorting and quality metrics for extracellular spike data. 2011. <https://github.com/danamics/UMS2K>`_
 
+.. [van_der_Molen] `RT-Sort: An action potential propagation-based algorithm for real time spike detection and sorting with millisecond latencies. 2024. <https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0312438>`_
+
 .. [Varol] `Decentralized Motion Inference and Registration of Neuropixel Data. 2021. <https://ieeexplore.ieee.org/document/9414145>`_
 
 .. [Watters] `MEDiCINe: Motion Correction for Neural Electrophysiology Recordings. 2025. <https://www.eneuro.org/content/12/3/ENEURO.0529-24.2025>`_

installation_tips/README.md

@@ -1,70 +1,105 @@
 ## Installation tips
 
-If you are not (yet) an expert in Python installations (conda vs pip, mananging environements, etc.),
-here we propose a simple recipe to install `spikeinterface` and several sorters inside an anaconda
-environment for windows/mac users.
-
-This environment will install:
- * spikeinterface full option
+If you are not (yet) an expert in Python installations, the first major hurdle is choosing the installation procedure.
+
+Some key concepts you need to know before starting:
+ * Python itself can be distributed and installed many, many ways.
+ * Python itself does not contain many features for scientific computing, so you need to install "packages". For example
+   numpy, scipy, matplotlib, spikeinterface, ... These are all examples of Python packages that aid in scientific computation.
+ * All of these packages have their own dependencies which requires figuring out which versions of the dependencies work for
+   the combination of packages you as the user want to use.
+ * Packages can be distributed and installed in several ways (pip, conda, uv, mamba, ...) and luckily these methods of installation
+   typically take care of solving the dependencies for you!
+ * Installing many packages at once is challenging (because of their dependency graphs) so you need to do it in an "isolated environment" to    not destroy any previous installation. You need to see an "environment" as a sub-installation in a dedicated folder.
+
+Choosing the installer + an environment manager + a package installer is a nightmare for beginners.
+
+The main options are:
+  * use "uv", a new, fast and simple package manager. We recommend this for beginners on every operating system.
+  * use "anaconda" (or its flavors-mamba, miniconda), which does everything. Used to be very popular but theses days it is becoming
+    harder to use because it is slow by default and has relatively strict licensing on the default channel (not always free anymore).
+    You need to play with "community channels" to make it free again, which is complicated for beginners.
+    This way is better for users in organizations that have specific licensing agrees with anaconda already in place.
+  * use Python from the system or Python.org + venv + pip: good and simple idea for Linux users, but does require familiarity with
+    the Python ecosystem (so good for intermediate users).
+
+Here we propose a step by step recipe for beginners based on [**"uv"**](https://github.com/astral-sh/uv).
+We used to recommend installing with anaconda. It will be kept here for a while but we do not recommend it anymore.
+
+
+This recipe will install:
+ * spikeinterface `full` option
  * spikeinterface-gui
- * phy
- * tridesclous
+ * kilosort4
 
-Kilosort, Ironclust and HDSort are MATLAB based and need to be installed from source.
+into our uv venv environment.
 
-### Quick installation
 
-Steps:
+### Quick installation using "uv" (recommended)
 
-1. Download anaconda individual edition [here](https://www.anaconda.com/download)
-2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners.
-3. Download with right click + save the file corresponding to your OS, and put it in "Documents" folder
-    * [`full_spikeinterface_environment_windows.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_windows.yml)
-    * [`full_spikeinterface_environment_mac.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_mac.yml)
-4. Then open the "Anaconda Command Prompt" (if Windows, search in your applications) or the Terminal (for Mac users)
-5. If not in the "Documents" folder type `cd Documents`
-6. Then run this depending on your OS:
-    * `conda env create --file full_spikeinterface_environment_windows.yml`
-    * `conda env create --file full_spikeinterface_environment_mac.yml`
+1. On macOS and Linux. Open a terminal and do
+   `curl -LsSf https://astral.sh/uv/install.sh | sh`
+2. On Windows. Open an instance of the Powershell (Windows has many options this is the recommended one from uv)
+   `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"`
+3. Exit the session and log in again.
+4. Download with right click and save this file in your "Documents" folder:
+    * [`beginner_requirements_stable.txt`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/beginner_requirements_stable.txt) for stable release
+5. Open terminal or powershell and run:
+6. `uv venv si_env --python 3.12`
+7. Activate your virtual environment by running:
+   - For Mac/Linux: `source si_env/bin/activate` (you should see `(si_env)` in your terminal)
+   - For Windows: `si_env\Scripts\activate`
+8. Run `uv pip install -r Documents/beginner_requirements_stable.txt`
 
 
-Done! Before running a spikeinterface script you will need to "select" this "environment" with `conda activate si_env`.
+## Installing before release (from source)
 
-Note for **linux** users : this conda recipe should work but we recommend strongly to use **pip + virtualenv**.
+Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, neo).
+We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `beginner_requirements_rolling.txt` file to create the environment instead of the `beginner_requirements_stable.txt` file. This will install the packages of the ecosystem from source.
+This is a good way to test if a patch fixes your issue.
 
 
 ### Check the installation
 
-
 If you want to test the spikeinterface install you can:
 
 1. Download with right click + save the file [`check_your_install.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/check_your_install.py)
     and put it into the "Documents" folder
-
-2. Open the Anaconda Command Prompt (Windows) or Terminal (Mac)
-3. If not in your "Documents" folder type `cd Documents`
-4. Run this:
-    ```
-    conda activate si_env
-    python check_your_install.py
-    ```
-5. If a windows user to clean-up you will also need to right click + save [`cleanup_for_windows.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/cleanup_for_windows.py)
-Then transfer `cleanup_for_windows.py` into your "Documents" folder. Finally run :
+2. Open the CMD Prompt (Windows)[^1] or Terminal (Mac/Linux)
+3. Activate your si_env : `source si_env/bin/activate` (Max/Linux), `si_env\Scripts\activate` (Windows)
+4. Go to your "Documents" folder with `cd Documents` or the place where you downloaded the `check_your_install.py`
+5. Run `python check_your_install.py`
+6. If you are a Windows user, you should also right click + save [`cleanup_for_windows.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/cleanup_for_windows.py). Then transfer `cleanup_for_windows.py` into your "Documents" folder and finally run:
    ```
    python cleanup_for_windows.py
    ```
 
-This script tests the following:
+This script tests the following steps:
   * importing spikeinterface
-  * running tridesclous
-  * running spyking-circus (not on mac)
-  * running herdingspikes (not on windows)
+  * running tridesclous2
+  * running kilosort4
   * opening the spikeinterface-gui
-  * exporting to Phy
 
 
-## Installing before release
+### Legacy installation using Anaconda (not recommended anymore)
+
+Steps:
+
+1. Download Anaconda individual edition [here](https://www.anaconda.com/download)
+2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners.
+3. Download with right click + save the environment YAML file ([`beginner_conda_env_stable.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/beginner_conda_env_stable.yml)) and put it in "Documents" folder
+4. Then open the "Anaconda Command Prompt" (if Windows, search in your applications) or the Terminal (for Mac users)
+5. If not in the "Documents" folder type `cd Documents`
+6. Run this command to create the environment:
+   ```bash
+   conda env create --file beginner_conda_env_stable.yml
+   ```
+
+Done! Before running a spikeinterface script you will need to "select" this "environment" with `conda activate si_env`.
+
+Note for **Linux** users: this conda recipe should work but we recommend strongly to use **pip + virtualenv**.
+
+
 
-Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, python-neo, sortingview).
-We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `full_spikeinterface_environment_rolling_updates.yml` file to create the environment. This will install the packages of the ecosystem from source.
-This is a good way to test if patch fix your issue.
+[^1]: Although uv installation instructions are for the Powershell, our sorter scripts are for the CMD Prompt. After the initial installation with Powershell, any session that will have sorting requires the CMD Prompt. If you do not
+plan to spike sort in a session either shell could be used.

…ikeinterface_environment_linux_dandi.yml …ation_tips/beginner_conda_env_stable.ymlinstallation_tips/full_spikeinterface_environment_linux_dandi.yml renamed to installation_tips/beginner_conda_env_stable.yml installation_tips/full_spikeinterface_environment_linux_dandi.yml renamed to installation_tips/beginner_conda_env_stable.yml

@@ -3,7 +3,7 @@ channels:
   - conda-forge
   - defaults
 dependencies:
-  - python=3.11
+  - python=3.12
   - pip
   - numpy
   - scipy
@@ -13,7 +13,7 @@ dependencies:
   - h5py
   - pandas
   - xarray
-  - zarr
+  - zarr<3
   - scikit-learn
   - hdbscan
   - networkx
@@ -30,8 +30,6 @@ dependencies:
   - libxcb
   - pip:
     - ephyviewer
-    - MEArec
     - spikeinterface[full,widgets]
     - spikeinterface-gui
-    - tridesclous
-    # - phy==2.0b5
+    - kilosort>4.0.30

installation_tips/beginner_requirements_rolling.txt

@@ -0,0 +1,10 @@
+https://github.com/NeuralEnsemble/python-neo/archive/master.zip
+https://github.com/SpikeInterface/probeinterface/archive/main.zip
+https://github.com/SpikeInterface/spikeinterface/archive/main.zip[full,widgets]
+https://github.com/SpikeInterface/spikeinterface-gui/archive/main.zip
+jupyterlab
+PySide6<6.8
+hdbscan
+pyqtgraph
+ephyviewer
+kilosort>4.0.30

installation_tips/beginner_requirements_stable.txt

@@ -0,0 +1,8 @@
+spikeinterface[full,widgets]
+jupyterlab
+PySide6<6.8
+hdbscan
+pyqtgraph
+ephyviewer
+spikeinterface-gui
+kilosort>4.0.30