ESN Parameter Effects about reservoirpy

[PAUL-BERNARD]

Hello,
Indeed, specifying the leaking rate to 1.0 shouldn't change anything, as this is already the default in ReservoirPy.
The difference in performance comes from the spectral radius :

• if not specified, a random sparse matrix W is generated, and its non-zero values are normally distributed (you can also change the distribution).
• if specified, the same matrix is created, and is then scaled to have a spectral radius of sr

In your case, the spectral radius you have if you don't specify its value is around ~10. You can get its value by using the reservoirpy.observables.spectral_radius method:

from reservoirpy.nodes import Reservoir
from reservoirpy.observables import spectral_radius
import numpy as np

my_reservoir = Reservoir(
    units=1000,
    # sr=1.0
)

my_reservoir.initialize(np.random.normal(size=(12, 1)))

spectral_radius(my_reservoir.W)

The spectral radius of the recurrent connection weights has a significant impact on the performances of the task, so it's not a surprise that you have such bad performances if you don't specify it. You can read more about its impact in the documentation.

[neuronalX]

Hello,

Thank you for testing new tasks with ReservoirPy.
Reservoir Computing rarely works just out-of-the-box, as it is a machine learning tool with few trained parameters (only the output layer), one need to find optimal hyperparameters (parameters not trained) for each kind of task.

Here there are several factors that could influence your performances.

1. Normalize your data if not done. Between -1 and 1 by default, or maybe between 0 and 1 if you have only positive values.
2. Start with the simplest model, i.e. an ESN that do not have a feedback from the read-out layer and the reservoir. This makes the training more complicated and more unstable, in particular if you use offline learning (ridge) and not online learning (RLS, FORCE, ...).
3. Make an extensive hyperparameter search, including changing the ridge parameter (regularization parameter). It is also important to keep fixed the number of units inside the reservoir while doing this search, because results will be less interpretable. It is better to make several searches with fixed number of units for each search, but increase the number of units for different searches instead.
4. Look at the results of hyperparameter search to understand what are the most robust set of hyperparameters, and do not just take the "best" result. If you just want to take the best, then be careful to take also the same seed, to be sure to obtain the same results. You can find several exemples of the influence of hyperparameters and how to look at the hyperparameter search results:
   https://github.com/reservoirpy/reservoirpy/blob/master/tutorials/4-Understand_and_optimize_hyperparameters.ipynb

This kind of plot will help you to understand what are the hyperparameters that give the most robust results.

I hope this helps. If you show us this kind of plot for all the hyperparameters, we could help you interpret them.

[neuronalX]

Additionally, we provide some hints of how to optimise hyperparemeters in this paper:

Which Hype for My New Task? Hints and Random Search for Echo State Networks Hyperparameters. ICANN 2021 HTML HAL PDF

[DJIDM]

Hello Paul,

Thank you for your response.

I actually achieved my best response when I left the spectral radius unspecified. I even attempted a hyperparameter optimization using Hyperopt, but my ESN with just the number of units specified outperformed the ESN with the optimized hyperparameters. I've attached some figures to illustrate what I mean.

Despite the better result, the unspecified predictions appear very noisy. I am guessing this has to do with the chaotic nature of the reservoir due to the high spectral radius. I was wondering if there was a way to smooth this out. I thought about using a filter, but my initial attempt proved unsuccessful.

*All the "Zoom In" figures are from the ESN with unspecified leaking rate and spectral radius.

[PAUL-BERNARD]

Hello,
Its difficult to tell why you don't have better results with an hyper-parameter exploration. Can you provide more details about it ? What kind of performances does the hyper-parameter exploration gives you around the default parameters ?

It seems you have more units in the default reservoir (500) than in the optimized version (150). That could explain the performance decrease.

For the smoothness of your output, many parameters comes into play, with high inter-dependencies.

[DJIDM]

Hello Paul,

I attached the hyperparameter optimization script, as well as the data I used to train the ESN. When I ran it, I got the values stated in the rctest_hyperopt figure that I shared previously.

As far as the number of units in my reservoir, I started with 150 and found that the performance increased as I increased the number of units.

[DJIDM]

Hello,

To illustrate my question further, in a previous experiment, I was able to create a ESN model (rpy_PK4GA7P1_ESN.py) that performed well in predicting future dynamics of a piezoelectric actuator.

When I rerun the same experiment with a new set of data from an electric linear actuator (rpy_RCP.py), keeping the structure of the model the same, I get very different performance results (RMSE: 14.22177, R^2: -16.21228). Furthermore, when I rerun the experiment again on the electric linear actuator data, but leave the leaking rate and spectral radius unspecified, the performance improves (RMSE: 0.75309, R^2: 0.95170), but still not to the level that was achieved with the piezoelectric data. I am wondering what adjustments I should make in order to achieve similar regression results to the piezoelectric data with the electric linear data.