cortex-lab / phy Goto Github PK
View Code? Open in Web Editor NEWphy: interactive visualization and manual spike sorting of large-scale ephys data
License: BSD 3-Clause "New" or "Revised" License
phy: interactive visualization and manual spike sorting of large-scale ephys data
License: BSD 3-Clause "New" or "Revised" License
For performance and memory considerations, it may be needed to delete noisy spikes to save memory.
We need an efficient structure for per-cluster data.
We'll probably need a dynamic array implementation on top of NumPy (inspired by this for example). For dual cluster axis (CCGs) we'll need something specific as well.
Ideally, this structure would contain a cluster_map
variable with the cluster assignements for all spikes. When this variable is changed, the internal arrays are updated.
cc @nippoo
phy.cluster.masked_em._stats
.To do later: support sparse structures.
Create a io/h5.py
module implementing a simple HDF5 API (on top of h5py).
with open_h5(filename, 'r') as f:
data = f.read('/path/to/node')
value = f.read_attr('/path/to/node', 'myattr')
with open_h5(filename, 'w') as f:
f.write('/path/to/node', data)
f.write_attr('/path/to/node', 'myattr', value)
Should offer the same experience as Qt's docking panels (resizable, drag-and-drop, fullscreen widgets).
A few links:
simple.html
, but maybe too limited: not possible to move widgets around)We should experiment with a few of these libraries and try to implement a prototype (using PNG screenshots of KlustaViewa's views for example).
_repr_html_()
of the MEA classInspired by scikit-learn:
# We launch the spike detection.
# This will automatically use multiple CPUs if
# multiple engines have been launched with IPython.parallel.
# This call is asynchronous: the user can continue to work in the notebook,
# and request the task's status.
phy.spikedetect.run(model, algorithm="spikedetekt", ipp_view=c.load_balanced_view())
# Launch clustering.
phy.cluster.run(model, algorithm="klustakwik2", ipp_view=c.load_balanced_view())
('move', [2], 0)
: move cluster 2 to group 0('merge', [3, 4], [10])
: merge clusters 3 and 4 to cluster 10next_best()
next_candidate()
next()
: call next_candidate()
or next_best()
if there's no candidate leftmerge(clusters, to)
: called by the Session controllermove(clusters, to)
: called by the Session controllerdefaultdict
(cl1, cl2) ==> similarity, default value=0. When the pair doesn't exist, the structure returns 0. We just have to compute the similarity for clusters that have similar channel masks.See here. Should be in:
phy/datasets/mock.py
phy/datasets/tests/test_mock.py
After #49 is merged.
phy/utils/array.py
will contain array-related utility functions.
Experimental VisPy code here: https://github.com/kwikteam/experimental/tree/master/plot
_update()
method which updates cluster counts and masksAn object that represents a selection of spikes.
spike_clusters
See the API on the wiki.
Possible starting point. Based on Vispy.
(nchannels, nsamples)
Locally and on Travis.
Based on Vispy.
key = value
pairs~/.phy/config.py
~/.phy/filename/config.py
to_dense()
from_dense
(possibly: to be partially merged into VisPy later)
Benchmarks need to be done in order to find efficient on-disk formats for the features.
Example size (high estimate): a (n_spikes, n_features)
numerical matrix with:
n_spikes = 100,000,000
n_features = 10,000
Access patterns:
Possibilities:
Notes:
In phy.cluster.manual.stats.py
.
See this issue
See this issue.
In the vertex shader, a_time + offset
with different offset for every spike ==> vertex
(spike_changed, cluster)
(works for both merge and split; those are actually similar actions)We need specific data structures to represent temporal data (like in the file format, but for in-memory structures). To be implemented in a specific package phy.time
.
What are the different types of temporal data?
We could subclass ndarray
to represent temporal data.
one array + metadata:
two arrays:
one array + metadata
(2, N)
array with start and endjust a Time Series + another array with the indices (e.g. neuron number for every spike)
(proposed by Adrien Peyrache)
Ping @kdharris101 @nippoo Adrien.
Each view for clustering will be an IPython widget exposing specific traitlet attributes:
A base widget will implement those, custom widgets will derive from it.
In the final interface in IPython, we'll link all these traitlets together using IPython's link()
function. When a spike selection changes in one widget, it will also change in the others.
To make this work, we'll need to implement specific traitlet types:
Provisional list of clustering widgets:
phy.io.kwik.model
BaseModel
.Implement a user-level class with control actions:
class Session:
def merge(clusters)
def move(clusters, group)
def undo()
def redo()
def start_wizard(self)
def pause_wizard(self)
def reset_wizard(self)
This class uses Clustering
, ClusterMetadata
, and Selection
instances, and uses a GlobalHistory
to track a unique undo stack with both clustering (merge, split, etc.) and cluster metadata (cluster moved, cluster color changed, etc.) actions.
This class can also update all views through the Selection
instance. The different instances communicate with UpdateInfo
instances.
Locally and on Travis.
Different classes that work with files should use this singleton to access the files.
opener = KwikOpener()
fh = opener.open('myfile.kwik')
opener.close('myfile.kwik')
We need to standardise:
What format the error, warn, log should take (present tense, capitalisation, line breaks, when they should be used, etc)
Also a standard for breaking from functions after an error.
A structure that handles:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.