Merge branch 'main' of github.com:SpikeInterface/spikeinterface into fix-node-pipeline

Author: alejoe91

doc/development/development.rst

@@ -61,7 +61,7 @@ for the :code:`spikeinterface.extractors` module, you can use the following comm
 
 The markers are located in the :code:`pyproject.toml` file in the root of the repository.
 
-Note that you should install the package before running the tests. You can do this by running the following command:
+Note that you should install spikeinterface before running the tests. You can do this by running the following command:
 
 .. code-block:: bash
 
@@ -72,7 +72,7 @@ You can change the :code:`[test,extractors,full]` to install only the dependenci
 The specific environment for the CI is specified in the :code:`.github/actions/build-test-environment/action.yml` and you can
 find the full tests in the :code:`.github/workflows/full_test.yml` file.
 
-The extractor tests require datalad for some of the tests. Here are instructions for installing datalad:
+Some of the extractor tests require datalad. Here are instructions for installing datalad:
 
 Installing Datalad
 ------------------
@@ -87,13 +87,13 @@ Stylistic conventions
 
 SpikeInterface maintains a consistent coding style across the project. This helps to ensure readability and
 maintainability of the code, making it easier for contributors to collaborate. To facilitate code style
-for the developer we use the follwing tools and conventions:
+for the developer we use the following tools and conventions:
 
 
 Install Black and pre-commit
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-We use the python formatter Black, with defaults set in the :code:`pyproject.toml`. This allows for
+We use the Python formatter Black, with defaults set in the :code:`pyproject.toml`. This allows for
 easy local formatting of code.
 
 To install Black, you can use pip, the Python package installer. Run the following command in your terminal:
@@ -218,7 +218,7 @@ Adding new documentation
 
 Documentation can be added as a
 `sphinx-gallery <https://sphinx-gallery.github.io/stable/index.html>`_
-python file ('tutorials')
+Python file ('tutorials')
 or a
 `sphinx rst <https://sphinx-tutorial.readthedocs.io/step-1/>`_
 file (all other sections).
@@ -242,7 +242,7 @@ To run code coverage locally, you can use the following command:
 
 This will run the tests and generate a report in the :code:`htmlcov` folder. You can open the :code:`index.html` file in your browser to see the report.
 
-Note, however, that the running time of the command above will be slow. If you want to run the tests for a specific module, you can use the following command:
+Note, however, that the running time of the command above will be quite long. If you want to run the tests for a specific module, you can use the following command:
 
 .. code-block:: bash
 
@@ -252,21 +252,21 @@ Implement a new extractor
 -------------------------
 
 SpikeInterface already supports over 30 file formats, but the acquisition system you use might not be among the
-supported formats list (***ref***). Most of the extractors rely on the `NEO <https://github.com/NeuralEnsemble/python-neo>`_
+supported formats list (****ref****). Most of the extractors rely on the `NEO <https://github.com/NeuralEnsemble/python-neo>`_
 package to read information from files.
 Therefore, to implement a new extractor to handle the unsupported format, we recommend making a new :code:`neo.rawio.BaseRawIO` class (see `example <https://github.com/NeuralEnsemble/python-neo/blob/master/neo/rawio/examplerawio.py#L44>`_).
 Once that is done, the new class can be easily wrapped into SpikeInterface as an extension of the
 :py:class:`~spikeinterface.extractors.neoextractors.neobaseextractors.NeoBaseRecordingExtractor`
 (for :py:class:`~spikeinterface.core.BaseRecording` objects) or
 :py:class:`~spikeinterface.extractors.neoextractors.neobaseextractors.NeoBaseRecordingExtractor`
 (for :py:class:`~spikeinterface.core.BaseSorting` objects) or with a few lines of
-code (e.g., see reader for `SpikeGLX <https://github.com/SpikeInterface/spikeinterface/blob/0.96.1/spikeinterface/extractors/neoextractors/spikeglx.py>`_
-or `Neuralynx <https://github.com/SpikeInterface/spikeinterface/blob/0.96.1/spikeinterface/extractors/neoextractors/neuralynx.py>`_).
+code (e.g., see reader for `SpikeGLX <https://github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/neoextractors/spikeglx.py>`_
+or `Neuralynx <https://github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/neoextractors/neuralynx.py>`_).
 
-**NOTE:** implementing a `neo.rawio` Class is not required, but recommended. Several extractors (especially) for :code:`Sorting`
+**NOTE:** implementing a `neo.rawio` class is not required, but recommended. Several extractors (especially) for :code:`Sorting`
 objects are implemented directly in SpikeInterface and inherit from the base classes.
-As examples, see the `CompressedBinaryIblExtractor <https://github.com/SpikeInterface/spikeinterface/blob/0.96.1/spikeinterface/extractors/cbin_ibl.py>`_
-for a :py:class:`~spikeinterface.core.BaseRecording` object, or the `SpykingCircusSortingExtractor <https://github.com/SpikeInterface/spikeinterface/blob/0.96.1/spikeinterface/extractors/spykingcircusextractors.py>`_
+As examples, see the `IblRecordingExtractor <https://github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/iblextractors.py>`_
+for a :py:class:`~spikeinterface.core.BaseRecording` object, or the `SpykingCircusSortingExtractor <https://github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/spykingcircusextractors.py>`_
 for a a :py:class:`~spikeinterface.core.BaseSorting` object.
 
 
@@ -309,7 +309,7 @@ Then, you can start creating a new class:
 
     class MySpikeSorter(BaseSorter):
     """
-    Brief description (optional)
+    Brief description
     """
 
     sorter_name = 'myspikesorter'
@@ -367,7 +367,7 @@ Now you can start filling out the required methods:
     def _check_apply_filter_in_params(cls, params):
         return False
 
-        #  optional
+        # optional
         # can be implemented in subclass to check if the filter will be applied
 
 
@@ -388,10 +388,12 @@ Now you can start filling out the required methods:
         return sorting
 
 When your spike sorter class is implemented, you have to add it to the list of available spike sorters in the
-`sorterlist.py`. Then you need to write a test in **tests/test_myspikesorter.py**. In order to be tested, you can
-install the required packages by changing the **pyproject.toml**. Note that MATLAB based tests cannot be run at the moment,
-but we recommend testing the implementation locally.
+`sorterlist.py <https://github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/sorters/sorterlist.py>`_ .
+Then you need to write a test in **tests/test_myspikesorter.py**. In order to be tested, you can
+install the required packages by changing the `pyproject.toml <https://github.com/SpikeInterface/spikeinterface/blob/main/pyproject.toml>`_.
+Note that MATLAB based tests cannot be run at the moment,but we recommend testing the implementation locally.
 
-After this you need to add a block in **doc/sorters_info.rst** to describe your sorter.
+After this you need to add a block in `Install Sorters <https://github.com/SpikeInterface/spikeinterface/blob/main/doc/get_started/install_sorters.rst>`_
+to describe your sorter.
 
 Finally, make a pull request so we can review the code and incorporate into the sorters module of SpikeInterface!

doc/get_started/quickstart.rst

@@ -287,7 +287,7 @@ available parameters are dictionaries and can be accessed with:
      'detect_threshold': 5,
      'freq_max': 5000.0,
      'freq_min': 400.0,
-     'max_threads_per_process': 1,
+     'max_threads_per_worker': 1,
      'mp_context': None,
      'n_jobs': 20,
      'nested_params': None,

pyproject.toml

@@ -61,7 +61,7 @@ changelog = "https://spikeinterface.readthedocs.io/en/latest/whatisnew.html"
 extractors = [
     "MEArec>=1.8",
     "pynwb>=2.6.0",
-    "hdmf-zarr>=0.5.0",
+    "hdmf-zarr>=0.11.0",
     "pyedflib>=0.1.30",
     "sonpy;python_version<'3.10'",
     "lxml", # lxml for neuroscope
@@ -80,7 +80,7 @@ streaming_extractors = [
     "fsspec",
     "aiohttp",
     "requests",
-    "hdmf-zarr>=0.5.0",
+    "hdmf-zarr>=0.11.0",
     "remfile",
     "s3fs"
 ]

src/spikeinterface/core/__init__.py

@@ -91,7 +91,14 @@
     write_python,
     normal_pdf,
 )
-from .job_tools import ensure_n_jobs, ensure_chunk_size, ChunkRecordingExecutor, split_job_kwargs, fix_job_kwargs
+from .job_tools import (
+    get_best_job_kwargs,
+    ensure_n_jobs,
+    ensure_chunk_size,
+    ChunkRecordingExecutor,
+    split_job_kwargs,
+    fix_job_kwargs,
+)
 from .recording_tools import (
     write_binary_recording,
     write_to_h5_dataset_format,

src/spikeinterface/core/globals.py

@@ -97,7 +97,9 @@ def is_set_global_dataset_folder() -> bool:
 
 
 ########################################
-_default_job_kwargs = dict(n_jobs=1, chunk_duration="1s", progress_bar=True, mp_context=None, max_threads_per_process=1)
+_default_job_kwargs = dict(
+    pool_engine="thread", n_jobs=1, chunk_duration="1s", progress_bar=True, mp_context=None, max_threads_per_worker=1
+)
 
 global global_job_kwargs
 global_job_kwargs = _default_job_kwargs.copy()