Merge branch 'main' into append-concat

Author: zm711

doc/modules/sorters.rst

@@ -469,6 +469,7 @@ Here is the list of external sorters accessible using the run_sorter wrapper:
 * **Klusta** :code:`run_sorter(sorter_name='klusta')`
 * **Mountainsort4** :code:`run_sorter(sorter_name='mountainsort4')`
 * **Mountainsort5** :code:`run_sorter(sorter_name='mountainsort5')`
+* **RT-Sort** :code:`run_sorter(sorter_name='rt-sort')`
 * **SpyKING Circus** :code:`run_sorter(sorter_name='spykingcircus')`
 * **Tridesclous** :code:`run_sorter(sorter_name='tridesclous')`
 * **Wave clus** :code:`run_sorter(sorter_name='waveclus')`

doc/references.rst

@@ -43,6 +43,7 @@ please include the appropriate citation for the :code:`sorter_name` parameter yo
 - :code:`herdingspikes` [Muthmann]_ [Hilgen]_
 - :code:`kilosort`  [Pachitariu]_
 - :code:`mountainsort` [Chung]_
+- :code:`rt-sort` [van_der_Molen]_
 - :code:`spykingcircus` [Yger]_
 - :code:`wavclus` [Chaure]_
 - :code:`yass` [Lee]_
@@ -154,6 +155,8 @@ References
 
 .. [UMS] `UltraMegaSort2000 - Spike sorting and quality metrics for extracellular spike data. 2011. <https://github.com/danamics/UMS2K>`_
 
+.. [van_der_Molen] `RT-Sort: An action potential propagation-based algorithm for real time spike detection and sorting with millisecond latencies. 2024. <https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0312438>`_
+
 .. [Varol] `Decentralized Motion Inference and Registration of Neuropixel Data. 2021. <https://ieeexplore.ieee.org/document/9414145>`_
 
 .. [Watters] `MEDiCINe: Motion Correction for Neural Electrophysiology Recordings. 2025. <https://www.eneuro.org/content/12/3/ENEURO.0529-24.2025>`_

installation_tips/README.md

@@ -1,70 +1,105 @@
 ## Installation tips
 
-If you are not (yet) an expert in Python installations (conda vs pip, mananging environements, etc.),
-here we propose a simple recipe to install `spikeinterface` and several sorters inside an anaconda
-environment for windows/mac users.
-
-This environment will install:
- * spikeinterface full option
+If you are not (yet) an expert in Python installations, the first major hurdle is choosing the installation procedure.
+
+Some key concepts you need to know before starting:
+ * Python itself can be distributed and installed many, many ways.
+ * Python itself does not contain many features for scientific computing, so you need to install "packages". For example
+   numpy, scipy, matplotlib, spikeinterface, ... These are all examples of Python packages that aid in scientific computation.
+ * All of these packages have their own dependencies which requires figuring out which versions of the dependencies work for
+   the combination of packages you as the user want to use.
+ * Packages can be distributed and installed in several ways (pip, conda, uv, mamba, ...) and luckily these methods of installation
+   typically take care of solving the dependencies for you!
+ * Installing many packages at once is challenging (because of their dependency graphs) so you need to do it in an "isolated environment" to    not destroy any previous installation. You need to see an "environment" as a sub-installation in a dedicated folder.
+
+Choosing the installer + an environment manager + a package installer is a nightmare for beginners.
+
+The main options are:
+  * use "uv", a new, fast and simple package manager. We recommend this for beginners on every operating system.
+  * use "anaconda" (or its flavors-mamba, miniconda), which does everything. Used to be very popular but theses days it is becoming
+    harder to use because it is slow by default and has relatively strict licensing on the default channel (not always free anymore).
+    You need to play with "community channels" to make it free again, which is complicated for beginners.
+    This way is better for users in organizations that have specific licensing agrees with anaconda already in place.
+  * use Python from the system or Python.org + venv + pip: good and simple idea for Linux users, but does require familiarity with
+    the Python ecosystem (so good for intermediate users).
+
+Here we propose a step by step recipe for beginners based on [**"uv"**](https://github.com/astral-sh/uv).
+We used to recommend installing with anaconda. It will be kept here for a while but we do not recommend it anymore.
+
+
+This recipe will install:
+ * spikeinterface `full` option
  * spikeinterface-gui
- * phy
- * tridesclous
+ * kilosort4
 
-Kilosort, Ironclust and HDSort are MATLAB based and need to be installed from source.
+into our uv venv environment.
 
-### Quick installation
 
-Steps:
+### Quick installation using "uv" (recommended)
 
-1. Download anaconda individual edition [here](https://www.anaconda.com/download)
-2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners.
-3. Download with right click + save the file corresponding to your OS, and put it in "Documents" folder
-    * [`full_spikeinterface_environment_windows.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_windows.yml)
-    * [`full_spikeinterface_environment_mac.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/full_spikeinterface_environment_mac.yml)
-4. Then open the "Anaconda Command Prompt" (if Windows, search in your applications) or the Terminal (for Mac users)
-5. If not in the "Documents" folder type `cd Documents`
-6. Then run this depending on your OS:
-    * `conda env create --file full_spikeinterface_environment_windows.yml`
-    * `conda env create --file full_spikeinterface_environment_mac.yml`
+1. On macOS and Linux. Open a terminal and do
+   `curl -LsSf https://astral.sh/uv/install.sh | sh`
+2. On Windows. Open an instance of the Powershell (Windows has many options this is the recommended one from uv)
+   `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"`
+3. Exit the session and log in again.
+4. Download with right click and save this file in your "Documents" folder:
+    * [`beginner_requirements_stable.txt`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/beginner_requirements_stable.txt) for stable release
+5. Open terminal or powershell and run:
+6. `uv venv si_env --python 3.12`
+7. Activate your virtual environment by running:
+   - For Mac/Linux: `source si_env/bin/activate` (you should see `(si_env)` in your terminal)
+   - For Windows: `si_env\Scripts\activate`
+8. Run `uv pip install -r Documents/beginner_requirements_stable.txt`
 
 
-Done! Before running a spikeinterface script you will need to "select" this "environment" with `conda activate si_env`.
+## Installing before release (from source)
 
-Note for **linux** users : this conda recipe should work but we recommend strongly to use **pip + virtualenv**.
+Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, neo).
+We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `beginner_requirements_rolling.txt` file to create the environment instead of the `beginner_requirements_stable.txt` file. This will install the packages of the ecosystem from source.
+This is a good way to test if a patch fixes your issue.
 
 
 ### Check the installation
 
-
 If you want to test the spikeinterface install you can:
 
 1. Download with right click + save the file [`check_your_install.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/check_your_install.py)
     and put it into the "Documents" folder
-
-2. Open the Anaconda Command Prompt (Windows) or Terminal (Mac)
-3. If not in your "Documents" folder type `cd Documents`
-4. Run this:
-    ```
-    conda activate si_env
-    python check_your_install.py
-    ```
-5. If a windows user to clean-up you will also need to right click + save [`cleanup_for_windows.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/cleanup_for_windows.py)
-Then transfer `cleanup_for_windows.py` into your "Documents" folder. Finally run :
+2. Open the CMD Prompt (Windows)[^1] or Terminal (Mac/Linux)
+3. Activate your si_env : `source si_env/bin/activate` (Max/Linux), `si_env\Scripts\activate` (Windows)
+4. Go to your "Documents" folder with `cd Documents` or the place where you downloaded the `check_your_install.py`
+5. Run `python check_your_install.py`
+6. If you are a Windows user, you should also right click + save [`cleanup_for_windows.py`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/cleanup_for_windows.py). Then transfer `cleanup_for_windows.py` into your "Documents" folder and finally run:
    ```
    python cleanup_for_windows.py
    ```
 
-This script tests the following:
+This script tests the following steps:
   * importing spikeinterface
-  * running tridesclous
-  * running spyking-circus (not on mac)
-  * running herdingspikes (not on windows)
+  * running tridesclous2
+  * running kilosort4
   * opening the spikeinterface-gui
-  * exporting to Phy
 
 
-## Installing before release
+### Legacy installation using Anaconda (not recommended anymore)
+
+Steps:
+
+1. Download Anaconda individual edition [here](https://www.anaconda.com/download)
+2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners.
+3. Download with right click + save the environment YAML file ([`beginner_conda_env_stable.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/main/installation_tips/beginner_conda_env_stable.yml)) and put it in "Documents" folder
+4. Then open the "Anaconda Command Prompt" (if Windows, search in your applications) or the Terminal (for Mac users)
+5. If not in the "Documents" folder type `cd Documents`
+6. Run this command to create the environment:
+   ```bash
+   conda env create --file beginner_conda_env_stable.yml
+   ```
+
+Done! Before running a spikeinterface script you will need to "select" this "environment" with `conda activate si_env`.
+
+Note for **Linux** users: this conda recipe should work but we recommend strongly to use **pip + virtualenv**.
+
+
 
-Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, python-neo, sortingview).
-We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `full_spikeinterface_environment_rolling_updates.yml` file to create the environment. This will install the packages of the ecosystem from source.
-This is a good way to test if patch fix your issue.
+[^1]: Although uv installation instructions are for the Powershell, our sorter scripts are for the CMD Prompt. After the initial installation with Powershell, any session that will have sorting requires the CMD Prompt. If you do not
+plan to spike sort in a session either shell could be used.

…ikeinterface_environment_linux_dandi.yml …ation_tips/beginner_conda_env_stable.ymlinstallation_tips/full_spikeinterface_environment_linux_dandi.yml renamed to installation_tips/beginner_conda_env_stable.yml installation_tips/full_spikeinterface_environment_linux_dandi.yml renamed to installation_tips/beginner_conda_env_stable.yml

@@ -3,7 +3,7 @@ channels:
   - conda-forge
   - defaults
 dependencies:
-  - python=3.11
+  - python=3.12
   - pip
   - numpy
   - scipy
@@ -13,7 +13,7 @@ dependencies:
   - h5py
   - pandas
   - xarray
-  - zarr
+  - zarr<3
   - scikit-learn
   - hdbscan
   - networkx
@@ -30,8 +30,6 @@ dependencies:
   - libxcb
   - pip:
     - ephyviewer
-    - MEArec
     - spikeinterface[full,widgets]
     - spikeinterface-gui
-    - tridesclous
-    # - phy==2.0b5
+    - kilosort>4.0.30

installation_tips/beginner_requirements_rolling.txt

@@ -0,0 +1,10 @@
+https://github.com/NeuralEnsemble/python-neo/archive/master.zip
+https://github.com/SpikeInterface/probeinterface/archive/main.zip
+https://github.com/SpikeInterface/spikeinterface/archive/main.zip[full,widgets]
+https://github.com/SpikeInterface/spikeinterface-gui/archive/main.zip
+jupyterlab
+PySide6<6.8
+hdbscan
+pyqtgraph
+ephyviewer
+kilosort>4.0.30

installation_tips/beginner_requirements_stable.txt

@@ -0,0 +1,8 @@
+spikeinterface[full,widgets]
+jupyterlab
+PySide6<6.8
+hdbscan
+pyqtgraph
+ephyviewer
+spikeinterface-gui
+kilosort>4.0.30

installation_tips/check_your_install.py

@@ -1,131 +1,115 @@
 from pathlib import Path
 import platform
-import os
 import shutil
 import argparse
+import warnings
+
+warnings.filterwarnings("ignore")
+
+
+job_kwargs = dict(n_jobs=-1, progress_bar=False, chunk_duration="1s")
+
 
 def check_import_si():
     import spikeinterface as si
 
+
 def check_import_si_full():
     import spikeinterface.full as si
 
 
 def _create_recording():
     import spikeinterface.full as si
-    rec, sorting = si.toy_example(num_segments=1, duration=200, seed=1, num_channels=16, num_columns=2)
-    rec.save(folder='./toy_example_recording')
+
+    rec, _ = si.generate_ground_truth_recording(
+        durations=[200.0], sampling_frequency=30_000.0, num_channels=16, num_units=10, seed=2205
+    )
+    rec.save(folder="./toy_example_recording", verbose=False, **job_kwargs)
 
 
 def _run_one_sorter_and_analyzer(sorter_name):
-    job_kwargs = dict(n_jobs=-1, progress_bar=True, chunk_duration="1s")
     import spikeinterface.full as si
-    recording = si.load_extractor('./toy_example_recording')
-    sorting = si.run_sorter(sorter_name, recording, folder=f'./sorter_with_{sorter_name}', verbose=False)
 
-    sorting_analyzer = si.create_sorting_analyzer(sorting, recording,
-                                                format="binary_folder", folder=f"./analyzer_with_{sorter_name}",
-                                                **job_kwargs)
+    si.set_global_job_kwargs(**job_kwargs)
+
+    recording = si.load("./toy_example_recording")
+    sorting = si.run_sorter(sorter_name, recording, folder=f"./sorter_with_{sorter_name}", verbose=False)
+
+    sorting_analyzer = si.create_sorting_analyzer(
+        sorting, recording, format="binary_folder", folder=f"./analyzer_with_{sorter_name}"
+    )
     sorting_analyzer.compute("random_spikes", method="uniform", max_spikes_per_unit=500)
-    sorting_analyzer.compute("waveforms", **job_kwargs)
+    sorting_analyzer.compute("waveforms")
     sorting_analyzer.compute("templates")
     sorting_analyzer.compute("noise_levels")
     sorting_analyzer.compute("unit_locations", method="monopolar_triangulation")
-    sorting_analyzer.compute("correlograms", window_ms=100, bin_ms=5.)
-    sorting_analyzer.compute("principal_components", n_components=3, mode='by_channel_global', whiten=True, **job_kwargs)
+    sorting_analyzer.compute("correlograms", window_ms=100, bin_ms=5.0)
+    sorting_analyzer.compute("principal_components", n_components=3, mode="by_channel_global", whiten=True)
     sorting_analyzer.compute("quality_metrics", metric_names=["snr", "firing_rate"])
 
 
-def run_tridesclous():
-    _run_one_sorter_and_analyzer('tridesclous')
-
 def run_tridesclous2():
-    _run_one_sorter_and_analyzer('tridesclous2')
+    _run_one_sorter_and_analyzer("tridesclous2")
 
 
+def run_kilosort4():
+    _run_one_sorter_and_analyzer("kilosort4")
+
 
 def open_sigui():
     import spikeinterface.full as si
-    import spikeinterface_gui
-    app = spikeinterface_gui.mkQApp()
-
-    sorter_name = "tridesclous2"
-    folder = f"./analyzer_with_{sorter_name}"
-    analyzer = si.load_sorting_analyzer(folder)
+    from spikeinterface_gui import run_mainwindow
 
-    win = spikeinterface_gui.MainWindow(analyzer)
-    win.show()
-    app.exec_()
-
-def export_to_phy():
-    import spikeinterface.full as si
     sorter_name = "tridesclous2"
     folder = f"./analyzer_with_{sorter_name}"
     analyzer = si.load_sorting_analyzer(folder)
 
-    phy_folder = "./phy_example"
-    si.export_to_phy(analyzer, output_folder=phy_folder, verbose=False)
-
-
-def open_phy():
-    os.system("phy template-gui ./phy_example/params.py")
+    win = run_mainwindow(analyzer, start_app=True)
 
 
 def _clean():
     # clean
     folders = [
         "./toy_example_recording",
-        "./sorter_with_tridesclous",
-        "./analyzer_with_tridesclous",
         "./sorter_with_tridesclous2",
         "./analyzer_with_tridesclous2",
-        "./phy_example"
+        "./sorter_with_kilosort4",
+        "./analyzer_with_kilosort4",
     ]
     for folder in folders:
         if Path(folder).exists():
             shutil.rmtree(folder)
 
+
 parser = argparse.ArgumentParser()
 # add ci flag so that gui will not be used in ci
 # end user can ignore
-parser.add_argument('--ci', action='store_false')
+parser.add_argument("--ci", action="store_false")
 
-if __name__ == '__main__':
+if __name__ == "__main__":
 
     args = parser.parse_args()
 
     _clean()
     _create_recording()
 
     steps = [
-        ('Import spikeinterface', check_import_si),
-        ('Import spikeinterface.full', check_import_si_full),
-        ('Run tridesclous', run_tridesclous),
-        ('Run tridesclous2', run_tridesclous2),
-        ]
+        ("Import spikeinterface", check_import_si),
+        ("Import spikeinterface.full", check_import_si_full),
+        ("Run tridesclous2", run_tridesclous2),
+        ("Run kilosort4", run_kilosort4),
+    ]
 
     # backwards logic because default is True for end-user
     if args.ci:
-        steps.append(('Open spikeinterface-gui', open_sigui))
-
-    steps.append(('Export to phy', export_to_phy)),
-        # phy is removed from the env because it force a pip install PyQt5
-        # which break the conda env
-        # ('Open phy', open_phy),
-
-    # if platform.system() == "Windows":
-    #     pass
-    # elif platform.system() == "Darwin":
-    #     pass
-    # else:
-    #     pass
+        steps.append(("Open spikeinterface-gui", open_sigui))
 
     for label, func in steps:
         try:
             func()
-            done = '...OK'
+            done = "...OK"
         except Exception as err:
-            done = f'...Fail, Error: {err}'
+            done = f"...Fail, Error: {err}"
         print(label, done)
 
     if platform.system() == "Windows":