memory leak about reservoirpy

[nTrouvain]

Hello,
Depending on the version you are trying to run, the code can indeed requires a lot of memory to run. This is not a leak, it comes from the fact that the linear regression algorithm used from training is suboptimal, and needs to perform very big matrix concatenations and dot products. This behaviour is corrected on last versions of reservoirpy, but requires that you batch your data. Could you give more details on the task you are trying to perform ?

[FuzzyWuzzyIsABear]

So I encountered the growing memory usage using hyperopt to optimize the hyperparameters. I stopped using the hyperopt and instead transplanted the objective function code into an objective function for a genetic algorithm and the memory leak stopped. So it sounds like the memory leak is somewhere in research().

[nTrouvain]

research is just a wrapper around hyperopt fmin function. It only adds the ability to save the results as JSON files automaticaly. I will investigate this but the memory leak is more likely to come from the way your task is framed, and the way hyperopt handles concurrency and memory usage. It would be very useful to know a bit more about your task: are you training your model on a single big sequence of data ? Or is it a dataset made of several short sequences of data ?

[FuzzyWuzzyIsABear]

its 15-20 short sequences of data with roughly 100 time points per sequence, about 350 variables.

The memory leak came back. Its not the research function. It looks like its from either train or run function of an ESN object.

[nTrouvain]

What version of reservoirpy are you using ? And what is the size of your reservoir ?

[FuzzyWuzzyIsABear]

version 0.2.4

I'm using a genetic algorithm with reservoir sizes between 2500 and 10000. So far the best size is 5000 for my data. Anymore seems to overfit but I'm still playing with parameters.

[nTrouvain]

In this case, the memory error is very likely to come from the huge size of your reservoirs. 1000 units is already a lot, and usually efficient enough for most cases. 2500 is usually already too much, and 10000 is probably unreasonable in almost all cases. Note that performance of ESNs does not increase linearly with the number of units, but with a factor close to the log of units. Hence, going from 100 units to 300 increases performance significantly, but going from 1000 to 10000 not much.
Could you try again with a more reasonable size ? For an hyperopt search we usually use a reduced number of units, between 100 and 300. Indeed, the number of units is (under a rough approximation) independent from all other hyperparameters except the ridge parameter for the readout learning. Once you have found all needed parameters (except ridge), you can then create a final version of your model with more units and adapt the ridge regularization to this number (i.e. to make the (X . X^T + ridge . Id) matrix full rank, where X is the concatenation of all reservoir states computed during training). This will increase the performance of your final model, if it can be increased. Sometimes, 100 units is enough, and is even preferable to a bigger size, which could lower the generalization capabilities of your model.

[FuzzyWuzzyIsABear]

Thanks for the advice. I am trying out smaller sizes. I just skipped over anything lower than 2000 because literature suggests high number of units for reservoir. Ill see if it helps with memory leakage.

[nTrouvain]

I am closing this issue as the problem seems to be solved. Let us know if it is not the case.