Release notes

Spyking CIRCUS 0.8

This is the 0.8 release of the SpyKING CIRCUS, a new approach to the problem of spike sorting. The code is based on a smart clustering with sub sampling, and a greedy template matching approach, such that it can resolve the problem of overlapping spikes. The publication about the software is available at https://elifesciences.org/articles/34518

../_images/launcher1.png

The software can be used with command line, or a dedicated GUI

Warning

The code may still evolve. Even if results are or should be correct, we can expect some more optimizations in a near future, based on feedbacks obtained on multiple datasets. If you spot some problems with the results, please be in touch with pierre.yger@inserm.fr

Contributions

Code and documentation contributions (ordered by the number of commits):

  • Pierre Yger

  • Marcel Stimberg

  • Baptiste Lebfevre

  • Christophe Gardella

  • Olivier Marre

  • Cyrille Rossant

  • Joze Guzman

  • Ariel Burman

  • Ben Acland

Release 0.9.6

  • fixes in the smart search (not all rare cases were covered in 0.9.5)

  • fix a bug if multi file is activated with very small chunks

  • speeding up the estimation of the templates: less snippets, closer to centroids

  • speeding up the estimation of the amplitudes: less noise snippets

  • speeding up isolation step during the smart search

  • number of bins is adapted during the smart search as function of noise levels

  • add the possibility to hide the status bars (for SpikeInterface logs)

Release 0.9.5

  • speeding up the optimization of the amplitudes with MPI

  • speeding up the processing of numpy datafiles (SpikeInterface)

  • speeding up the smart search step (pre-generation of random numbers)

  • speeding up the clustering step

  • fix a bug while filtering in the preview mode introduced in 0.9.2

  • speeding up the fitting step

Release 0.9.4

  • speeding up the optimization of the amplitudes with MPI

  • speeding up the processing of numpy datafiles (SpikeInterface)

  • speeding up the smart search step (pre-generation of random numbers)

Release 0.9.2

  • speeding up the algorithm

  • fixing a bug in the clustering while assigining labels

  • better detection of noise snippets discarded during clustering

  • cosmetic changes in the sanity plots (clusters)

  • better handling of overlapping chunks while filtering, removing filtering artefacts

  • templates are restricted within shanks

  • optimization of the amplitudes once all templates have been found

  • export of a purity value, for phy, to assess how good a cluster is (between 0 and 1)

  • display the purity value in MATLAB

  • fix a (silent) bug in the supports introduced in 0.9.0, preventing mixture removal

  • nb_chances is automatically adapted during the fitting procedure

  • drifts are now automatically handled by the meta merging procedure

  • enhancement in the automatic merging of drifts

Release 0.9.1

  • Minor bug fixes in spyking-circus-launcher

  • fix a bug in the amplitude display. Values were shuffled when several CPU were used

  • add the option to ignore second component [clustering]->two_components

Release 0.9.0

  • can now fit spikes below detection threshold (with spike_thresh_min)

  • templates are now estimated without any spatial restrictions

  • display a warning if N_t is not optimally chosen

Release 0.8.9

  • fix a small bug in the smart search, introduced while refactoring in 0.8.7

Release 0.8.8

  • fix a regression introduced in 0.8.7 for non contiguous probe indices

Release 0.8.7

  • new methods to detect the peaks, more robust when low thresholds are fixed

  • more accurate fitting procedure, slightly slower

  • minor bug fixes

  • addition of a sparsity_limit parameter in the meta merging GUI, to remove noise more precisely

  • new parameter file is properly copied

  • enhancement of the smoothing/alignement procedure, more accurate estimation of noisy templates

  • better estimation of the amplitudes boundaries used during fitting

  • optimization while removing mixtures and important bug fixes

  • fix a bug in the thresholding method

  • minor updates to get more refined spikes during whitening and clustering

  • tests with SpikeInterface, showing clear increase in performance

  • some cleaning in the parameter file

  • default value for cc_merge is now 0.95, since merging functions are more robust

  • noisy templates are removed by default while meta merging with a lower threshold (0.75)

  • speeding up whitening and clustering steps

Release 0.8.6

  • Export from manual sorting with MATLAB to phy is now possible

  • Modification to pass SpikeSorters test suite

Release 0.8.5

  • fix a bug while removing noisy templates in meta merging

  • refactoring of the meta merging GUI, addition of bhatta distances

  • meta merging more robust for non stationary recordings

  • enhance logging if parameters are missing and/or not defined

  • can now display the electrode labels in preview GUI

  • detects if a wrong install of MPI is present (linking with mpi4py)

  • conda install overwrites the old parameter file

  • raw dispay of the MUA in the result GUI (to be improved)

  • display an error if not all nodes on a cluster can read the datafiles

  • fix a bug for thresholding method using dead times

Release 0.8.4

  • fix if no spikes are found on some electrodes

  • fix as mean/median-pca methods were broken (albeit not used)

  • fix to prevent a rare crash while loading too sparse overlaps

  • fix a bug with the new dip method in python 2.7

  • add the thresholding method to extract only MUA activity (requested by users)

  • channel lists in probe files can be non sorted

  • memory usage is dynamically adapted to reduce memory footprint

  • hdf5 and npy file format can now work with 3D arrays (x, y, time) or (time, x, y)

  • fix a bug if basis for pos and neg spikes have different sizes

  • add some docstrings (thanks to Jose Guzman)

  • sparse export for phy is now the default

  • comments can now be added in the trigger/dead times files

  • 4096 channels can now run on a single machine, with low memory consumption

  • basic support for 3d probes, without any visualization

  • more robust to saturating channels with nan_to_num

  • cc_merge set to 1 automatically if templates on few channels are detected

  • fix a bug if only one artefact type is given

  • fix a bug if only 2 spikes are found on a single electrode

  • former parameters sim_same_elec and dip_threshold renamed into merge_method and merge_param

  • sanity plots for local merges can now be produced during clustering (debug_plots in [clustering])

Release 0.8.3

  • automatic suppression, during meta merging, of noisy templates (for SpikeToolKit/Forest)

  • during the phy export, we can automatically pre-assign labels to neurons

  • fix a bug when converting to phy with dead channels

  • fix a bug when converting to phy with file formats without data_offset

  • speedup the estimation of the amplitude distribution

  • minor fixes for clusters

  • smoothing of the templates thanks to Savitzky-Golay filtering

  • fix a bug when launching GUIs for file format without data offset

  • can now work with scipy 1.3 and statsmodels 0.10

  • isolation mode is improved, set as default and leading to better performance

  • reducing overclustering with the Hartigan dip-test of unimodality

  • can now set the number of dimensions for local PCA (10 by default)

Release 0.8.2

  • add a docker file to build the software

  • add support for shanks in phy 2.0

  • add support for deconverting in the qt launcher

  • do not create a Qt App if merging in auto mode

  • waveforms are convolved with a Hanning window to boost PCA

  • oversampling in now adapted as function of the sampling rate

  • reduction of I/O while oversampling

  • speed improvement with undersampling while cleaning the dictionary

  • automation of the software for SpikeForest/SpikeToolkit benchmarks

  • merging is now included in the default pipeline

  • normalization of the metrics in the meta merging GUI

Release 0.8.0

  • major improvement in the clustering. No more max_clusters parameters

  • much faster clustering (thanks to Ruben Herzog)

  • added the statsmodels library as a required dependency

  • enhancement of the smart search mode

  • enhancement of the bicubic spline interpolation

  • fix a typo when using dead times and the collect mode

  • fix a minor bug when small amount of spikes are found during smart search

  • fix a bug in the wrapper for BRW files

  • support for phy 2.0 and phylib

  • remove the strongly time shifted templates

  • additing of a wrapper for MDA file format

  • amplitudes for unfitted spikes is now 1 when exporting to phy

  • default install is now qt5, to work with phy 2.0

Release 0.7.6

  • cosmetic changes in the GUI

  • adding a deconverting method to switch back from phy to MATLAB

  • support for the lags between templates in the MATLAB GUI

  • warn user if data are corrupted because of interrupted filtering

  • reduction of the size for saved clusters

  • display the file name in the header

  • fix a nasty bug allowing spikes at the border of chunks to be fitted even during dead periods

Release 0.7.5

  • fix a bug for MPICH when large dictionaries.

  • fix a bug for numpy files when used with new numpy versions

  • add the possibility to subtract one channel as a reference channel from others

  • native support for blackrock files (only .ns5 tested so far)

  • simplifications in the parameter file

  • fix for display of progress bars with tqdm

  • addition of a multi-folders mode for openephys

  • hide GPU support for now, as this is not actively maintained and optimized

  • fix in the MATLAB GUI for float32 data

  • fix the broken log files

  • default cpu number is now half the available cores

Release 0.7.4

  • fix a regression with spline interpolation, more investigation needed

Release 0.7.0

  • fix a possible rounding bug if triggers are given in ms

  • artefacts are computed as medians and not means over the signal

  • can turn off shared memory if needed

  • a particular pattern can be specified for neuralynx files

  • fix bugs with output_dir, as everything was not saved in the folder

  • add a circus-folders script to process virtually files within several folders as a single recording

  • add a circus-artefacts script to concatenate artefact files before using stream mode

  • multi-files mode is now enabled for Neuralynx data

  • fixes for conversion of old dataset with python GUI

  • smooth exit if fitting with 0 templates (thanks to Alex Gonzalez)

  • enhance the bicubic spline interpolation for oversampling

  • spike times are now saved as uint32 for long recordings

Release 0.6.7

  • optimizations for clusters (auto blosc and network bandwith)

  • addition of a dead_channels option in the [detection] section, as requested

  • prevent user to remove median with only 1 channel

  • fix for parallel writes in HDF5 files

  • hide h5py FutureWarning

Release 0.6.6

  • fix for matplotlib 2.2.2

  • fix a bug when loading merged data with phy GUI

  • faster support for native MCD file with pyMCStream

  • more robust whitening for large arrays with numerous overlaps

  • add an experimental mode to refine coreset (isolated spikes)

  • put merging units in Hz^2 in the merging GUI

  • add a HDF5 compression mode to greatly reduce disk usage for very large probe

  • add a Blosc compression mode to save bandwith for clusters

  • fix a display bug in the merging GUI when performing multiple passes

Release 0.6.5

  • reduce memory consumption for mixture removal with shared memory

  • made an explicit parameter cc_mixtures for mixture removal in the [clustering] section

  • Minor fixes in the MATLAB GUI

  • fix in the exact times shown during preview if second is specified

  • prevent errors if filter is False and overwrite is False

Release 0.6.4

  • fix a bug in the BEER for windows platforms, enhancing robustness to mpi data types

  • speed up the software when using ignore_dead_times

  • ensure backward compatibility with hdf5 version for MATLAB

  • fix a rare bug in clustering, when no spikes are found on electrodes

  • fix a bug in the MATLAB GUI when reloading saved results, skipping overlap fixes

Release 0.6.3

  • fix a bug if the parameter file have tabulations characters

  • add a tab to edit parameters directly in the launcher GUI

  • fix dtype offset for int32 and int64

  • minor optimizations for computations of overlaps

  • explicit message displayed on screen if filtering has already been performed

  • can specify a distinct folder for output results with output_dir parameter

  • fix a bug when launching phy GUI for datafiles without data_offset parameter (HDF5)

  • fix a memory leak when using dead_times

  • fix a bug for BRW and python3

  • fix a bug in the BEER

  • pin HDF5 to 1.8.18 versions, as MATLAB is not working well with 1.10

  • fix a bug when relaunching code and overwrite is False

  • fix a bug when peak detection is set on both with only one channel

Release 0.6.2

  • fix for openephys and new python syntax

  • fix in the handling of parameters

  • fix a bug on windows with unclosed hdf5 files

  • fix a bug during converting with multi CPU on windows

  • minor optimization in the fitting procedure

  • support for qt5 (and backward compatibility with qt4 as long as phy is using Qt4)

Release 0.6.1

  • fix for similarities and merged output from the GUIs

  • fix for Python 3 and HDF5

  • fix for Python 3 and launcher GUI

  • fix for maxlag in the merging GUI

  • optimization in the merging GUI for pairs suggestion

  • addition of an auto_mode for meta merging, to suppress manual curation

  • various fixes in the docs

  • fix a bug when closing temporary files on windows

  • allow spaces in names of probe files

  • collect_all should take dead times into account

  • patch to read INTAN 2.0 files

  • fix in the MATLAB GUI when splitting neurons

  • fix in the MATLAB GUI when selecting individual amplitudes

Release 0.6.0

  • fix an IMPORTANT BUG in the similarities exported for phy/MATLAB, affect the suggestions in the GUI

  • improvements in the neuralynx wrapper

  • add the possibility to exclude some portions of the recordings from the analysis (see documentation)

  • fix a small bug in MS-MPI (Windows only) when shared memory is activated and emtpy arrays are present

Release 0.5.9

  • The validating step can now accept custom spikes as inputs

  • Change the default frequency for filtering to 300Hz instead of 500Hz

Release 0.5.8

  • fix a bug for int indices in some file wrappers (python 3.xx) (thanks to Ben Acland)

  • fix a bug in the preview gui to write threshold

  • fix a bug for some paths in Windows (thanks to Albert Miklos)

  • add a wrapper for NeuraLynx (.ncs) file format

  • fix a bug in the installation of the MATLAB GUI

  • fix a bug to see results in MATLAB GUI with only 1 channel

  • fix a bug to convert data to phy with only positive peaks

  • add builds for python 3.6

  • optimizations in file wrappers

  • fix a bug for MCS headers in multifiles, if not all with same sizes

  • add the possibility (with a flag) to turn off parallel HDF5 if needed

  • fix a bug with latest version of HDF5, related to flush issues during clustering

Release 0.5.7

  • Change the strsplit name in the MATLAB GUI

  • Fix a bug in the numpy wrapper

  • Fix a bug in the artefact removal (numpy 1.12), thanks to Chris Wilson

  • Fixes in the matlab GUI to ease a refitting procedure, thanks to Chris Wilson

  • Overlaps are recomputed if size of templates has changed (for refitting)

  • Addition of a “second” argument for a better control of the preview mode

  • Fix when using the phy GUI and the multi-file mode.

  • Add a file wrapper for INTAN (RHD) file format

Release 0.5.6

  • Fix in the smart_search when only few spikes are found

  • Fix a bug in density estimation when only few spikes are found

Release 0.5.5

  • Improvement in the smart_select option given various datasets

  • Fix a regression for the clustering introduced in 0.5.2

Release 0.5.2

  • fix for the MATLAB GUI

  • smart_select can now be used [experimental]

  • fix for non 0: DISPLAY

  • cosmetic changes in the clustering plots

  • ordering of the channels in the openephys wrapper

  • fix for rates in the MATLAB GUI

  • artefacts can now be given in ms or in timesteps with the trig_unit parameter

Release 0.5rc

  • fix a bug when exporting for phy in dense mode

  • compatibility with numpy 1.12

  • fix a regression with artefact removal

  • fix a display bug in the thresholds while previewing with a non unitary gain

  • fix a bug when filtering in multi-files mode (overwrite False, various t_starts)

  • fix a bug when filtering in multi-files mode (overwrite True)

  • fix a bug if matlab gui (overwrite False)

  • fix the gathering method, not working anymore

  • smarter selection of the centroids, leading to more clusters with the smart_select option

  • addition of a How to cite section, with listed publications

Release 0.5b9

  • switch from progressbar2 to tqdm, for speed and practical issues

  • optimization of the ressources by preventing numpy to use multithreading with BLAS

  • fix MPI issues appearing sometimes during the fitting procedure

  • fix a bug in the preview mode for OpenEphys files

  • slightly more robust handling of openephys files, thanks to Ben Acland

  • remove the dependency to mpi4py channel on osx, as it was crashing

  • fix a bug in circus-multi when using extensions

Release 0.5b8

  • fix a bug in the MATLAB GUI in the BestElec while saving

  • more consistency with “both” peak detection mode. Twice more waveforms are always collect during whitening/clustering

  • sparse export for phy is now available

  • addition of a dir_path parameter to be compatible with new phy

  • fix a bug in the meta merging GUI when only one template left

Release 0.5b7

  • fix a bug while converting data to phy with a non unitary gain

  • fix a bug in the merging gui with some version of numpy, forcing ucast

  • fix a bug if no spikes are detected while constructing the basis

  • Optimization if both positive and negative peaks are detected

  • fix a bug with the preview mode, while displaying non float32 data

Release 0.5b6

  • fix a bug while launching the MATLAB GUI

Release 0.5b3

  • code is now hosted on GitHub

  • various cosmetic changes in the terminal

  • addition of a garbage collector mode, to collect also all unfitted spikes, per channel

  • complete restructuration of the I/O such that the code can now handle multiple file formats

  • internal refactoring to ease interaction with new file formats and readibility

  • because of the file format, slight restructuration of the parameter files

  • N_t and radius have been moved to the [detection] section, more consistent

  • addition of an explicit file_format parameter in the [data] section

  • every file format may have its own parameters, see documentation for details (or –info)

  • can now work natively with open ephys data files (.openephys)

  • can now work natively with MCD data files (.mcd) [using neuroshare]

  • can now work natively with Kwik (KWD) data files (.kwd)

  • can now work natively with NeuroDataWithoutBorders files (.nwb)

  • can now work natively with NiX files (.nix)

  • can now work natively with any HDF5-like structure data files (.h5)

  • can now work natively with Arf data files (.arf)

  • can now work natively with 3Brain data files (.brw)

  • can now work natively with Numpy arrays (.npy)

  • can now work natively with all file format supported by NeuroShare (plexon, blackrock, mcd, …)

  • can still work natively with raw binary files with/without headers :)

  • faster IO for raw binary files

  • refactoring of the exports during multi-file/preview/benchmark: everything is now handled in raw binary

  • fix a bug with the size of the safety time parameter during whitening and clustering

  • all the interactions with the parameters are now done in the circus/shared/parser.py file

  • all the interactions with the probe are now done in the circus/shared/probes.py file

  • all the messages are now handled in circus/shared/messages.py

  • more robust and explicit logging system

  • more robust checking of the parameters

  • display the electrode number in the preview/result GUI

  • setting up a continuous integration workflow to test all conda packages with appveyor and travis automatically

  • cuda support is now turned off by default, for smoother install procedures (GPU yet do not bring much)

  • file format can be streamed. Over several files (former multi-file mode), but also within the same file

  • several cosmetic changes in the default parameter file

  • clustering:smart_search and merging:correct_lag are now True by default

  • fix a minor bug in the smart search, biasing the estimation of densities

  • fix a bug with the masks and the smart-search: improving results

  • addition of an overwrite parameter. Note that any t_start/t_stop infos are lost

  • if using streams, or internal t_start, output times are on the same time axis than the datafile

  • more robust parameter checking

Release 0.4.3

  • cosmetic changes in the terminal

  • suggest to reduce chunk sizes for high density probes (N_e > 500) to save memory

  • fix a once-in-a-while bug in the smart-search

Release 0.4.2

  • fix a bug in the test suite

  • fix a bug in python GUI for non integer thresholds

  • fix a bug with output strings in python3

  • fix a bug to kill processes in windows from the launcher

  • fix graphical issues in the launcher and python3

  • colors are now present also in python3

  • finer control of the amplitudes with the dispersion parameter

  • finer control of the cut off frequencies during the filtering

  • the smart search mode is now back, with a simple True/False flag. Use it for long or noisy recordings

  • optimizations in the smart search mode, now implementing a rejection method based on amplitudes

  • show the mean amplitude over time in the MATLAB GUI

  • MATLAB is automatically closed when closing the MATLAB GUI

  • mean rate is now displayed in the MATLAB GUI, for new datasets only

  • spike times are now saved as uint32, for new datasets only

  • various fixes in the docs

  • improvements when peak detection is set on “both”

  • message about cc_merge for low density probes

  • message about smart search for long recordings

  • various cosmetic changes

  • add a conda app for anaconda navigator

Release 0.4.1

  • fix a bug for converting millions of PCs to phy, getting rid of MPI limitation to int32

  • fix bugs with install on Windows 10, forcing int64 while default is int32 even on 64bits platforms

  • improved errors messages if wrong MCS headers are used

  • Various cosmetic changes

Release 0.4

First realease of the software