Development =========== How to contribute ----------------- There are various ways to contribute to SpikeInterface as a user or developer. Some tasks you can help us with include: * Developing a new extractor for a different file format. * Creating a new spike sorter. * Designing a new post-processing algorithm. * Enhancing documentation, including docstrings, tutorials, and examples. * Crafting tutorials for common workflows (e.g., spike sorting, postprocessing, etc.). * Writing unit tests to expand code coverage and use case scenarios. * Reporting bugs and issues. We use a forking workflow ``_ to manage contributions. Here's a summary of the steps involved, with more details available in the provided link: * Fork the SpikeInterface repository. * Create a new branch (e.g., :code:`git switch -c my-contribution`). * Modify the code, commit, and push changes to your fork. * Open a pull request from the "Pull Requests" tab of your fork to :code:`spikeinterface/main`. * By following this process, we can review the code and even make changes as necessary. While we appreciate all the contributions please be mindful of the cost of reviewing pull requests ``_ . How to run tests locally ------------------------- Before submitting a pull request, we recommend running the tests locally. In the CI we use pytest to run the tests so it is a good idea to do the same. To run the tests locally, you can use the following command: .. code-block:: bash pytest From your local repository. This will run all the tests in the repository. If you want to run a specific test, you can use the following command: .. code-block:: bash pytest path/to/test.py For example, if you want to run the tests for the :code:`spikeinterface.extractors` module, you can use the following command: .. code-block:: bash pytest src/spikeinterface/extractors If you want to run a specific test in a specific file, you can use the following command: .. code-block:: bash pytest src/spikeinterface/core/tests/test_baserecording.py::specific_test_in_this_module We also mantain pytest markers to run specific tests. For example, if you want to run only the tests for the :code:`spikeinterface.extractors` module, you can use the following command: .. code-block:: bash pytest -m "extractors" The markers are located in the :code:`pyproject.toml` file in the root of the repository. Note that you should install spikeinterface before running the tests. You can do this by running the following command: .. code-block:: bash pip install -e .[extractors,full] --group test-all You can change the :code:`[extractors,full]` extras to install only the runtime dependencies you need, and swap :code:`--group test-all` for a per-module test group (for example :code:`--group test-postprocessing`). Both lists are defined in :code:`pyproject.toml`: feature extras live under :code:`[project.optional-dependencies]` and test groups under :code:`[dependency-groups]` (PEP 735). Note that :code:`--group` requires pip 25.1+ or uv. With :code:`uv`, you can run the tests for a single module without creating and activating a virtual environment explicitly. :code:`uv run` resolves the project, extras, and dependency groups on the fly. The general pattern is: .. code-block:: bash uv run --extra --group test- pytest src/spikeinterface//tests For example: .. code-block:: bash uv run --extra postprocessing --group test-postprocessing pytest src/spikeinterface/postprocessing/tests uv run --extra sortingcomponents --group test-sortingcomponents pytest src/spikeinterface/sortingcomponents/tests To replicate the nightly CI environment and run the whole suite at once: .. code-block:: bash uv run --extra extractors --extra streaming_extractors --extra full --group test-all pytest Both :code:`--extra` and :code:`--group` lists are enumerated in :code:`pyproject.toml`. Some modules' tests happen to work with just :code:`--group test-` because the group pulls the runtime deps transitively (preprocessing is one such case), but declaring the matching :code:`--extra` explicitly is the reliable pattern. The specific environment for the CI is specified in the :code:`.github/actions/build-test-environment/action.yml` and you can find the full tests in the :code:`.github/workflows/full_test.yml` file. Some of the extractor tests require datalad. Here are instructions for installing datalad: Installing Datalad ------------------ In order to get datalad for your OS please see the `datalad instruction `_. For more information on datalad visit the `datalad handbook `_. Note, this will also require having git-annex. The instruction links above provide information on also downloading git-annex for your particular OS. Stylistic conventions --------------------- SpikeInterface maintains a consistent coding style across the project. This helps to ensure readability and maintainability of the code, making it easier for contributors to collaborate. To facilitate code style for the developer we use the following tools and conventions: Install ruff and pre-commit ^^^^^^^^^^^^^^^^^^^^^^^^^^^ We use `ruff `_ both to format code and to lint it (catch errors such as undefined names or unused variables). Its settings live in the :code:`[tool.ruff]` section of :code:`pyproject.toml`. We use pre-commit to run ruff, and a few other checks, automatically before every commit. You can install both with pip: .. code-block:: bash pip install ruff pre-commit Then set up the pre-commit hooks for your local repository by running, from the repository root: .. code-block:: bash pre-commit install Now, each time you make a commit, pre-commit will run :code:`ruff format` and :code:`ruff check --fix` on the changed files under :code:`src/`. If the hooks make changes or find issues, the commit is stopped so you can review the changes, add them and commit again. You can also run ruff yourself at any time: .. code-block:: bash ruff format src # format the code ruff check src # report lint errors ruff check src --fix # report lint errors and apply the safe automatic fixes A few things to know about the lint configuration: * Test files (:code:`tests/` folders, :code:`test_*.py` and :code:`conftest.py`) are excluded from linting. * Unused imports (:code:`F401`) and unused variables (:code:`F841`) are reported but never removed automatically, because they are sometimes needed: side-effect imports, re-exports, or function calls that read from a file. Fix them by hand, or keep them and mark the line with :code:`# noqa: F401`. * Pytest fixtures shared by several test modules belong in a :code:`conftest.py`, so pytest finds them without an import (an import that ruff would see as unused). Since tests are not linted on every commit, it is worth checking them every now and then. A manual pre-commit hook lints everything under :code:`src/`, tests included: .. code-block:: bash pre-commit run ruff-check-tests --hook-stage manual --all-files or, equivalently, with ruff directly: .. code-block:: bash ruff check src --config 'lint.exclude = []' If you want ruff to leave a line unformatted, add the following comment to the end of the line: .. code-block:: python # fmt: skip To leave a block of code unformatted, put these two comments around it: .. code-block:: python # fmt: off code here # fmt: on To silence a single lint error on one line, add :code:`# noqa: ` (for example :code:`# noqa: F401`) to the end of that line. Docstring Conventions ^^^^^^^^^^^^^^^^^^^^^ For docstrings, SpikeInterface generally follows the `numpy docstring standard `_. This includes providing a one line summary of a function, and the standard NumPy sections including :code:`Parameters`, :code:`Returns`, etc. The format used for providing parameters, however is a little different. The project prefers the format: .. code-block:: bash parameter_name : type, default: default_value This allows users to quickly understand the type of data that should be input into a function as well as whether a default is supplied. A full example would be: .. code-block:: python def a_function(param_a, param_b=5, param_c="mean"): """ A function for analyzing data Parameters ---------- param_a : dict A dictionary containing the data param_b : int, default: 5 A scaling factor to be applied to the data param_c : "mean" | "median", default: "mean" What to calculate on the data Returns ------- great_data : dict A dictionary of the processed data """ There should be a space between each parameter and the colon following it. This is neccessary for using the `numpydoc validator `_. In the above example we demonstrate two other docstring conventions followed by SpikeInterface. First, that all string arguments should be presented with double quotes. This is the same stylistic convention enforced by ruff format in the pre-commit for the repo. Second, when a parameter is a string with a limited number of values (e.g. :code:`mean` and :code:`median`), rather than give the type a value of :code:`str`, please list the possible strings so that the user knows what the options are. Miscelleaneous Stylistic Conventions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #. Avoid using abbreviations in variable names (e.g. use :code:`recording` instead of :code:`rec`). It is especially important to avoid single letter variables. #. Use index as singular and indices for plural following the NumPy convention. Avoid idx or indexes. Plus, id and ids are reserved for identifiers (i.e. channel_ids) #. We use file_path and folder_path (instead of file_name and folder_name) for clarity. #. For the titles of documentation pages, only capitalize the first letter of the first word and classes or software packages. For example, "How to use a SortingAnalyzer in SpikeInterface". #. For creating headers to divide sections of code we use the following convention (see issue `#3019 `_): .. code:: python ######################################### # A header ######################################### How to build the documentation ------------------------------ We use Sphinx to build the documentation. To build the documentation locally, you can use the following command: .. code-block:: bash sphinx-build -b html doc ./doc/_build/ This will build the documentation in the :code:`doc/_build/html` folder. You can open the :code:`index.html` file in your browser to see the documentation. Adding new documentation ------------------------ Documentation can be added as a `sphinx-gallery `_ Python file ('tutorials') or a `sphinx rst `_ file (all other sections). To add a new tutorial, add your ``.py`` file to ``spikeinterface/examples``. Then, update the ``spikeinterface/doc/tutorials_custom_index.rst`` file to make a new card linking to the page and an optional image. See ``tutorials_custom_index.rst`` header for more information. For other sections, write your documentation in ``.rst`` format and add the page to the appropriate ``index.rst`` file found in the relevant folder (e.g. ``how_to/index.rst``). How to run code coverage locally -------------------------------- To run code coverage locally, you can use the following command: .. code-block:: bash pytest --cov=spikeinterface --cov-report html This will run the tests and generate a report in the :code:`htmlcov` folder. You can open the :code:`index.html` file in your browser to see the report. Note, however, that the running time of the command above will be quite long. If you want to run the tests for a specific module, you can use the following command: .. code-block:: bash pytest src/spikeinterface/core/ --cov=spikeinterface/core --cov-report html Integrate your software with SpikeInterface ------------------------------------------- You've made a great analysis tool for ephys data - congrats! It's popular and people are using it. Why would you integrate it into SpikeInterface? And how can do you do that? Why integrate it ^^^^^^^^^^^^^^^^ SpikeInterface is designed to allow users to 1) read in any recording format 2) apply a wide variety of preprocessing steps to the recording 3) run any sorter on the data 4) compute standard postprocessing data about the sorting (and more!!). We work hard to be general and allow users to do anything they'd like with their data. Hence if your tool works with SpikeInterface, it works with an enormous array of possible analysis pipelines. When they're first written, analysis tools are usually designed in a specific lab and are tested with a small number of collaborators and datasets. It's technically difficult to design a tool which can be applied to data from a wide variety of devices, set-ups and sorters. There are many edge cases and "gotcha"s to consider. But the SpikeInterface team have extensive experience with exactly this problem. We know how to write code so that it doesn't depend on data-specific or sorter-specific quirks. Hence we can help generalize the tool and widen its potential userbase. Often, the first step when using an analysis tool is to wrangle the data into a specific format: "please save your recording to a binary file which is preprocessed in a certain way" or "ensure your sorting output is in this format". SpikeInterface deals with this painful data wrangling for you: we load many formats and internally represent them in a consistent way. We also apply all preprocessing steps lazily, meaning that users never need to save a copy of a preprocessed recording: they can work directly from the raw recording. We believe this reduces the barrier to entry for the end user, which makes your tool more likely to be used by many researchers. How to integrate it ^^^^^^^^^^^^^^^^^^^ If you'd like to integrate your tool into SpikeInterface: amazing! Just `raise an issue `_. There are two main ways to integrate your software into SpikeInterface. 1) Wrapping You keep your code in your own codebase, and write a thin wrapper in SpikeInterface which interacts with your library. Most external sorters that SpikeInterface can call are organized in this way. To use the tool, the user will have to install your package and the maintenance of the software is primarily on your shoulders. 2) Integration You re-implement your code and add it to the SpikeInterface package. Many preprocessing steps, motion correction algorithms and postprocessing steps are implemented like this. Your method and code become part of the SpikeInterface package and we take on the maintenance burden (with your help, we hope!). In this case, we don't expect the re-implementation to be a perfect copy of the code, since we may need to change the code to work for a more general set of data. Credit and Citation ^^^^^^^^^^^^^^^^^^^ We want to ensure the tools that SpikeInterface wrap and integrate are properly cited. We'll ensure that credit is given when the tool is mentioned in the docs, in the docstring of the function, and we will add it to the :doc:`../references` page. When added, we will include your software on the front-page of the SpikeInterface GitHub for at least 1 year. If we fail to cite you properly, this is certainly a mistake. Please `raise an issue `_. See all the packages and projects which have been wrapped or integrated into SpikeInterface in our :doc:`../references` page. Implement a new extractor ------------------------- SpikeInterface already supports over 30 file formats, but the acquisition system you use might not be among the supported formats list (****ref****). Most of the extractors rely on the `NEO `_ package to read information from files. Therefore, to implement a new extractor to handle the unsupported format, we recommend making a new :code:`neo.rawio.BaseRawIO` class (see `example `_). Once that is done, the new class can be easily wrapped into SpikeInterface as an extension of the :py:class:`~spikeinterface.extractors.neoextractors.neobaseextractors.NeoBaseRecordingExtractor` (for :py:class:`~spikeinterface.core.BaseRecording` objects) or :py:class:`~spikeinterface.extractors.neoextractors.neobaseextractors.NeoBaseRecordingExtractor` (for :py:class:`~spikeinterface.core.BaseSorting` objects) or with a few lines of code (e.g., see reader for `SpikeGLX `_ or `Neuralynx `_). **NOTE:** implementing a `neo.rawio` class is not required, but recommended. Several extractors (especially) for :code:`Sorting` objects are implemented directly in SpikeInterface and inherit from the base classes. As examples, see the `IblRecordingExtractor `_ for a :py:class:`~spikeinterface.core.BaseRecording` object, or the `SpykingCircusSortingExtractor `_ for a a :py:class:`~spikeinterface.core.BaseSorting` object. Implement a spike sorter ------------------------ Implementing a new spike sorter for a specific file format is as simple as creating a new subclass based on the predefined base class :code:`BaseSorter`. To enable standardization among subclasses, the :code:`BaseSorter` is base class which require a new subclass to override a few methods. The contributed extractors are in the **spikesorters** folder. You can fork the repo and create a new folder **myspikesorter** there. In the folder, create a new file named **myspikesorter.py**. Additional configuration files must be placed in the same folder. You can start by importing the base class: .. code-block:: python import spikeinterface.extractors as se from ..basesorter import BaseSorter In order to check if your spike sorter is installed, a :code:`try` - :code:`except` block is used. For example, if your sorter is implemented in Python (installed with the package :code:`myspikesorter`), this block will look as follows: .. code-block:: python import importlib.util if importlib.util.find_spec("myspikesorter"): HAVE_MYSORTER = True else: HAVE_MYSORTER = False Then, you can start creating a new class: .. code-block:: python class MySpikeSorter(BaseSorter): """ Brief description """ sorter_name = 'myspikesorter' installed = HAVE_MYSORTER _default_params = { 'param1': None, 'param2': 2, } _params_description = { 'param1': 'Description for param1', 'param1': 'Description for param1', } installation_mesg = """ >>> pip install myspikesorter More information on MySpikesorter at: https://myspikesorterwebsite.com """ Now you can start filling out the required methods: .. code-block:: python def __init__(self, **kargs): BaseSorter.__init__(self, **kargs) # optional @classmethod def get_sorter_version(cls): return myspikesorter.__version__ @classmethod def is_installed(cls): # Fill code to check sorter installation. It returns a boolean return HAVE_MYSORTER @classmethod def _setup_recording(cls, recording, output_folder, params, verbose): # Fill code to set up the recording: convert to required file, parse config files, etc. # The files should be placed in the 'output_folder' @classmethod def _check_params(cls, recording, output_folder, params): # optional # can be implemented in subclass for custom checks return params @classmethod def _check_apply_filter_in_params(cls, params): return False # optional # can be implemented in subclass to check if the filter will be applied @classmethod def _run_from_folder(cls, output_folder, params, verbose): # Fill code to run your spike sorter based on the files created in the _setup_recording() # You can run CLI commands (e.g. klusta, spykingcircus, tridesclous), pure Python code (e.g. Mountainsort5, # Herding Spikes), or even MATLAB code (e.g. Kilosort, Kilosort2, Ironclust) @classmethod def _get_result_from_folder(cls, output_folder): # If your spike sorter has a specific file format, you should implement a SortingExtractor in spikeextractors. # Let's assume you have done so, and the extractor is called MySpikeSorterSortingExtractor sorting = se.MySpikeSorterSortingExtractor(output_folder) return sorting When your spike sorter class is implemented, you have to add it to the list of available spike sorters in the `sorterlist.py `_ . Then you need to write a test in **tests/test_myspikesorter.py**. In order to be tested, you can install the required packages by changing the `pyproject.toml `_. Note that MATLAB based tests cannot be run at the moment,but we recommend testing the implementation locally. After this you need to add a block in `Install Sorters `_ to describe your sorter. Finally, make a pull request so we can review the code and incorporate into the sorters module of SpikeInterface! How to make a release --------------------- Checklist ^^^^^^^^^ * pyproject.toml: check that the version is ahead of current release. Also, comment out the @ (git dependencies) * In the top level ``__init__`` (located at ``src/spikeinterface/__init__.py``) set ``DEV_MODE`` to ``False`` (this is used for the docker installations) * Create a new release note for the appropriate version on doc/releases/new_version_tag. It can be a large release like: ``doc/releases/0.101.0.rst`` Which contain a section called "Main Changes" and minor releases which include only bug fixes like: ``doc/releases/0.101.2.rst`` To collect all the PRs and bug fixes we have a bash script in: ``doc/scripts/`` called ``auto-release-notes.sh``. Run it with ``bash auto-release-notes.sh`` and it will create the release notes for the module specific changes. The first time you run the script, GitHub will guide you through an authorization process if you've not already done so. The signature of the script is: .. code-block:: bash bash auto-release-notes.sh Where the start date is the date of the last release and the end date is the current date. Dates are in YYYY-MM-DD format The date of the last release can be found on `PyPI `_. As a specific example: .. code-block:: bash bash auto-release-notes.sh 2025-02-19 2025-03-24 * Add the release to `doc/whatisnew.rst`, by adding it to the list of releases and providing a short description of the release. * Finish the release notes and merge * Locally tag the main branch with the newly merged release notes with the new version * Push the tag to the remote repository which will trigger the release action (.github/workflows/publish-to-pypi.yml) * Do an after-release `PR `_: - Uncomment the git installs in pyproject - Set ``DEV_MODE`` to ``True`` in the top level ``__init__`` (located at ``src/spikeinterface/__init__.py``) - Update `pyproject.toml` version one patch ahead or one minor if it is larger one.