swharden / pyabf Goto Github PK
View Code? Open in Web Editor NEWpyABF is a Python package for reading electrophysiology data from Axon Binary Format (ABF) files
Home Page: https://swharden.com/pyabf
License: MIT License
pyABF is a Python package for reading electrophysiology data from Axon Binary Format (ABF) files
Home Page: https://swharden.com/pyabf
License: MIT License
When using a custom stimulus waveform, ATF stimulus waveforms are supported with abf.py#L35 which uses atf_reader.py and atf_storage.py to cache ATF files and load them in as an ABF class argument.
Should atfStorage=ATFStorage
really be an ABF class argument?
This feels odd because this seems like a rare case (ABF files with ATF stimulus files), yet its made its way all the way to become an argument in the primary class of this entire project. It works as it is, but I'm preparing to port this project to C# (re-writing it from scratch) so I want to make sure everything is in its most logical spot.
An alternative way to get the same functionality is just to have the user do the work of reading and caching these files themselves. The tools can still be available with pyABF, just imported outside the ABF class. Perhaps even an ATF class would be useful.
import pyabf
abf = pyabf.ABF("data.abf")
abf.epochsByChannel = pyabf.ATF("stim.atf").epochsByChannel
# then all the usual python code works
print(abf.sweepX, abf.sweepY)
print(abf.sweepX, abf.sweepC)
Although, does it really make sense to call it epochs at all at that point? Why not just keep the stimulus data outside the ABF class forever? This would mean you'd write use-case-based analysis code as needed:
import pyabf
stim = pyabf.ATF("stim.atf")
abf = pyabf.ABF("data.abf")
# then write your custom program as needed
print(abf.sweepX, abf.sweepY)
print(abf.sweepX, stim.sweepC)
I'm leaning toward this, because it makes it easy to maintain the ATF code and ATF code doesn't live inside of the ABF class...
According to the struct format, "i"
and "l"
are both 4-byte integers. Make it clear in the source code these are to be read identically. Currently, i
and l
are used interchangeably. Maybe it means something to C89, but not to Python.
https://docs.python.org/2/library/struct.html#format-characters
Lines 279 to 307 in 02ef67d
Hi Scott,
We have an ABF file with 2 channels. Channel 0 is membrane potential which is recovered correctly. Channel 1 is an analog output from a temperature recorder.
The temperature is scaled as 2.3 + 10*T and when the ABF file is read into the software that generated it, it displays and formats correctly. However when it's read into python /w pyabf the temperature is not transformed and the units appear to be set as mV
I've included the ABF file in question along with an example use case
import pyabf
abf = pyabf.ABF("example.abf")
abf.setSweep(0, channel=0)
Vm = abf.dataY
t = abf.dataX
abf.setSweep(0, channel=1)
Temp = abf.dataY
#Temp = 2.3 + 10* abf.dataY
It's obviously not a critical issue - but it would be a nice feature (or if I'm doing something wrong, if you could let me know that would be great).
Cheers,
Aaron
How hard would it be to make the current code base back-compatible with Python 2.7? Assess feasibility and follow-up here. Perhaps a single release supporting Python 2.7 would be useful.
pyABF uses lots of f-strings which according to PEP 4098 means Python <3.6 isn't supported. Python 3.6 is probably the minimum supported by pyABF.
Check how this fails if run on Python 2. I got an email from someone saying this happens (which is unclear).
File "/Users/dir/Library/Python/2.7/lib/python/site-packages/pyabf/__init__.py", line 9, in <module>
from ._version import __version__
File "/Users/dir/Library/Python/2.7/lib/python/site-packages/pyabf/_version.py", line 28
errMsg = f"pyABF version {__version__} < required {versionNeeded}"
Strip/add to ensure trailing zeros to 3 spaces for the seconds timestamp
Loading 16426005.abf (6/1570) ...
2016-04-26T13:48:58.265000
Loading 16426006.abf (7/1570) ...
2016-04-26T14:19:26
Loading 16426007.abf (8/1570) ...
2016-04-26T14:47:20.375000
Rather than using the mean limits, use the min/max limits.
This behavior is defined here:
https://github.com/swharden/pyABF/blob/master/dev/autoanalysis/analysisByProtocol.py#L223
abf.epochPoints
and abf.epochValues
are now simple to access. If abf.epochTypes
is added, epoch waveform synthesis can be dramatically simplified (by simply reading these 3 lists). stimulus.py could be greatly simplified by refactoring to this model.
currently all debugging is done with warnings.warn()
. This module would greatly benefit from careful implementation of the logging module. This would be especially useful for providing additional information when tests fail.
Graphs on the quickstart guide are also used on the pypi pyABF index page. Currently they use the BMH style, but that complicates imports and isn't visually recognizable as a Python/matplotlib graph in the same way that the default styles are (this is important since google ImageSearch is indexing these GitHub and PyPi pages).
Modify all quickstart code to use the default styling and ensure the PyPi page gets updated too.
when looking at a grid of tens of images, yellow backdrop would help you shoot right to the drug-application experiments
Kim emailed me an ABF which crashes pyABF when it is loaded. Upon inspection I realized I never tested pyABF against a gap-free file, since all work I do (and all example files to date) involves episodic files (with no time between sweeps, so it's effectively gap-free).
This abf currently crashes pyABF:
/data/16d22006_kim_gapfree.abf
Header:
16d22006_kim_gapfree.md
I'm using pyABF in https://github.com/AllenInstitute/ipfx/blob/master/ipfx/x_to_nwb/ABFConverter.py in version 2.0.18.
Is the new API stable enough so that upgrading to it now makes sense?
If yes, could you create a pypi release before?
When attempting to read in an .abf file with only one comment, the following error is shown:
" 'int' object is not iterable ", from line 77 in abf.py.
My crude solution so far is to replace lines 77-79 with the following (there should be a more elegant way, I think):
if isinstance(self.commentTimes, int):
self.commentTimesSec = self.commentTimes*self._abfHeader.header["fSynchTimeUnit"]/1e6
self.commentTimeMin = self.commentTimesSec * 60
self.commentSweeps = int(self.commentTimesSec/self.sweepLengthSec)
else:
self.commentTimesSec=[x*self._abfHeader.header["fSynchTimeUnit"]/1e6 for x in self.commentTimes]
self.commentTimesMin=[x*60 for x in self.commentTimesSec]
self.commentSweeps=[int(x/self.sweepLengthSec) for x in self.commentTimesSec]
On a related note, I would greatly appreciate a fix to issue #3 :D
since autoanalysis runs on a headless server, log its exceptions to file.
https://github.com/swharden/pyABF/tree/master/dev/autoanalysis
Close this ticket when numpy is imported normally.
Initially a lot of effort was put into making the abf header class numpy-agnostic. This just complicates code maintenance, and since non-numpy data arrays are so slow, let's just go ahead and require numpy for pyABF. If a user wants a non-numpy ABF header class, they can modify it like this.
try:
import numpy as np # use Numpy if we have it
except:
np=False
fb=open("someFile.abf",'rb')
fb.seek(firstBytePosition)
scaleFactor = self.header['lADCResolution'] / 1e6
if np:
data = np.fromfile(fb, dtype=np.int16, count=pointCount)
data = np.multiply(data,scaleFactor,dtype='float32')
else:
print("WARNING: data is being retrieved without numpy (this is slow). See docs.")
data = struct.unpack("%dh"%(pointCount), fb.read(pointCount*2)) # 64-bit int
data = [point*scaleFactor for point in data] # 64-bit int * 64-bit floating point
fb.close()
abf1 files do support tags. fix this.
Often I forget to add a tag or later want to change the time or comment of a tag in an ABF file. Since the tag structure is known, create a module to modify tags by changing the specific bytes of the file associated with tags.
This might be a little complex with files which don't contain tags, as there will be no bytes specifically allocated to store tag information. Investigation will require finding unused space in the ABF file which can be modified, or seeing if adding extra bytes at the end of the file is destructive.
I'm not sure what currently happens when a 5 second sweep is set to record every 10 seconds. 50% of that time should be empty.
Locate or create an ABF with known empty space and ensure it's handled properly by pyABF
I just noticed it's not right
https://github.com/swharden/pyABF/blob/master/data/headers/130618-1-12.md
fEpochInitLevel should be -70, -80, -70, ...
I found a bug in ABF-Tag-Editor which makes me think tag times in pyABF might be wrong in ABFs with space between their sweeps (non-minimum inter-sweep interval?)
Find an ABF with tags which meets this criteria and confirm.
The current method of generating a text-formatted header report isn't intuitive:
abf = pyabf.ABF("demo.abf")
abf.getInfoPage().showText()
Consider something like:
abf = pyabf.ABF("demo.abf")
print(abf.headerText)
Then add that to the quickstart guide.
I just noticed that the units are wrong on channel 2 of 18702001-ramp.abf https://github.com/swharden/pyABF/tree/master/data - it's a model cell, and I think Ch2 units should be mV
In neo with have also a AxonIO which is basically what pyabf do.
@Emerson207 is reporting a strange problem here:
NeuralEnsemble/python-neo#533
Maybe we can try to resolve it evrywhere (neo + pyabf).
Same problem in neo.
I open an issue here for collborating.
NeuralEnsemble/python-neo#544
Currently a new instance of ATF is created for every loaded protocol. This takes quite some time and makes it unbearable slow here for ATF files with 20MB.
My old approach of caching the results needs to be resurrected so that this is usable again.
I'll provide a PR.
one graph with all 4
Tested on pyabf-2.0.20 and pyabf-2.0.19. Returns DateTime object in 2.0.9, which I chose at random because I really need to get some files processed.
Version 2.0.17 returns a DateTime object correctly.
For a while I used winter
, then I switched to Dark2
. I just now realized Dark2
is stepped, not a gradient. Find a gradient color map which is versatile in many situations.
Examples where demo is used: https://github.com/swharden/pyABF/tree/master/docs/getting-started#advanced-plotting-with-the-pyabfplot-module
Change the default colormap here:
Line 43 in 232826a
There aren't any great options:
https://github.com/swharden/pyABF/blob/master/docs/advanced/v1%20cookbook/2017-11-12%20colormaps/colormaps.pdf
... so maybe a custom gradient to mimic Dark2?
just regenerating the data index isn't working because the output is flagged as different based on numpy version number.
make a test that ensures header values don't change when core restructuring happens.
every time I regenerate the header github wants to commit a change
bottom right graph units should be pF not pA.
pyABF/dev/autoanalysis/analysisByProtocol.py
Line 430 in 4ef9e19
171116sh_0017 displays a memtest pattern, when really the waveform should be entirely disabled
https://github.com/swharden/pyABF/tree/master/data
The spirit of the ABF class is to read ABF files... not to analyze them.
Currently things like baseline subtraction and averaging are in the ABF class. These "advanced" features might best be moved outside the core ABF class. Perhaps an "analysis" class which you could feed ABF objects would be a better way to approach analytical processes which lie outside the scope of ABF file reading (and hence the ABF class). This ticket notes on these ideas.
override abfFilePath so it's always something generic.
Also make abfDateTimeString contain the "T"
A lot of people seem to use software like MiniAnalysis which expects data to arrive as ABF1 files. It may be useful to create a pyABF module which allows the creation of ABF1 files from scratch.
The goal of this ticket is to track progress toward exporting data as ABF1 files that MiniAnalysis can import. In the process a sweep-synthesis module will be created to create simulated data which contains events suitable for detection.
Always, thank you for your contribution.
I found minor issue while handling my data with datetime of recording.
To get right into the point, abfDateTime seem to have minor problem
I have checked my file with your getInfoPage().launchTempWebpage() function .
import pyabf
abf = pyabf.ABF('sample.abf')
abf.getInfoPage().launchTempWebpage()
and it returns
abfDateTime = 2018-01-05 18:28:13
uFileStartDate = 20180305
uFileStartTimeMS = 66313103
above date time does not match and I know second one is correct.
Therefore, I looked up your code and I found some minor issue.
In your code, it is written like below.
startDate = str(self._headerV2.uFileStartDate)
startTime = round(self._headerV2.uFileStartTimeMS/1000)
startDate = datetime.datetime.strptime(startDate, **"%Y%M%d"**)
startTime = datetime.timedelta(seconds=startTime)
self.abfDateTime = startDate+startTime
as you can see, strptime format has little mistypo I guess.
In my guess, you have to change
"%Y%M%d" into "%Y%m%d" .
Currently only epoch types 0 (disabled), 1 (step), and 2 (ramp) are supported. Determine what additional epochs are available and be able to synthesize their waveforms.
Stress test this against new multi-channel ABFs with epochs of variable duration and variable levels. This is likely best done in conjunction with a full overall of the epoch processing procedure.
A lot of places in core.py I do things like if self.abfFileFormat == 2
. Currently the abfFileFormat
property of the ABF class is just a 1 or a 2 depending on if abfVersion
starts with a 1 or 2. The abfFileFormat
property can probably be removed entirely, replaced by a small functions which determines if the file format is ABF1 or ABF2 based on looking at abfVersion
.
Currently fSignalOffset
and fInstrumentOffset
are screwing with scaleFactors
. Modify core.py to move these operations into the _loadAndScaleData()
section. This problem doesn't affect ABFs where fSignalOffset
and fInstrumentOffset
are zero (most ABFs).
Identify any ABFs where fSignalOffset
and fInstrumentOffset
are not zero and ensure it is working.
https://github.com/swharden/pyABF/blob/master/src/pyabf/core.py#L240-L241
In the getting started guide go out of your way to emphasize that sweeps start at zero. For mult-sweep examples use sweeps [0, 5, 10, 15]
The autoanalysis project (/dev/autoanalysis) started as a small side-project initially intended to become a pyABF module (similar to the intrinsic property calculating memtest.py module, or action potential detection AP.py module). It has since exploded in complexity, is highly experimental, and probably merits its own repo instead of bundling all this code with the pyABF project. Frequent commits to the autoanalysis folder are also drowning-out meaningful commits to the core pyABF project.
Remove the autoanalysis project from pyABF, and give it its own repo.
This issue is described in NeuralEnsemble/python-neo#533
Update the ABFcore class and the ABF file format guide to reflect this understanding.
ensure they don't need to be divided by the number of channels
Currently the code looks to see if sADCUnit
has the label "mV", and if it does assumes it's voltage clamp (pA units) otherwise it assumes its current-clamp (mV units). This happens here:
https://github.com/swharden/pyABF/blob/master/src/pyabf/header.py#L132
This has caused problems in the past, because non-telegraphed instruments such as (axo-patch) don't include sADCUnit
in their header. Additionally, some ABFs have channels recording more than just voltage or current (some ABFs in the data section record temperature). In these cases, the lookup fails, and the default unit is pA.
Find an alternative field in the header to more reliably determine units. Make it ABF1 and ABF2 compatible.
protocol 0404
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.