Memory usage builds up when iteratively open abf files about pyabf

[javierzsm]

Thanks Scott!
The issue also raises when running on regular python from bash, that was the first thing I tried.
However, you seem to have spotted what the issue was as calling the garbage collector solved the issue (and made me learn about it!).
Thanks, this was very helpful!

[swharden]

Hi @javierzsm, thanks for reporting this! I can take a closer look this evening.

Can you share the code you are using to iteratively open many ABF files?

[javierzsm]

Hi @swharden

Thanks a lot for your quick reply. I copy below the piece of code that reproduce the issue.

I just noticed that it becomes appreciable when opening files => 60-70 Mb each; with smaller files it builds up but more slowly.

Thanks,
Javier

import pyabf
import os

path_data = r'/home/javier/Lab/Data/Ephys/'

for nom_file in os.listdir(path_data):
    if nom_file.endswith('.abf'):
        print('\nOpening file %s' % nom_file)
        abf = pyabf.ABF(path_data + '/' + nom_file)

[swharden]

Hi @javierzsm,

I do not think this is a bug. I think this is the natural management of memory by Python's garbage collector.

Here's sample code I used to benchmark how much memory is used as I re-load the same ABF files 5,000 times:

import sys
import pathlib
import tracemalloc
import os
import matplotlib.pyplot as plt
import pyabf

if __name__ == "__main__":

    abfPath = "2020_06_16_0001.abf"
    repetitions = 5000
    memoryPerLoad = []

    tracemalloc.start()
    for i in range(repetitions):
        abf = pyabf.ABF(abfPath)
        memory = tracemalloc.get_traced_memory()[0]/1e6
        memoryPerLoad.append(memory)
        print(f"{i+1} of {repetitions}... current memory: {memory} MB")
    tracemalloc.stop()

    plt.plot(memoryPerLoad)
    plt.title(os.path.basename(abfPath))
    plt.grid(ls='--', alpha=.5)
    plt.ylabel("Process Memory (MB)")
    plt.xlabel("ABF Read Count")
    plt.show()

If you want to force Python's garbage collector to run manually you can, but this is typically not recommended https://docs.python.org/3/library/gc.html

Do you think this explanation adequately addresses your issue? If you can provide or a sample ABF and sample code that lets me reproduce a bug on my side I'd be happy to look deeper into this. Thanks, and good luck! 👍 🚀

[swharden]

It may be worth trying to call the garbage collector manually after loading huge files to see if it helps... something like this:

import pyabf
import gc

for i in range(repetitions):
    abf = pyabf.ABF("test.abf")
    gc.collect()

[swharden]

Oh! Also I now see you're running that from inside Spyder/IPython.

Does this issue go away if you run it from a console in regular Python?