Initial Guess - SR3 - Custom Library [BUG] about pysindy

[IamKFJ]

Thanks for the quick reply!

I chose library ensembling because i saw that it can lead to better approximation in one of the youtube tutorials. I didn't think about how ensembling interfeers with the initial guesses.
I don't remember why I chose the SR3. I think i got it from building on an existing code i found online.
I want to use initial guesses because i wanted to prime the algorithm towars a choice of parameters i know to be somewhat physical (mainly setting them positive/negative/zero). The ones in the initial guesse i put into the code are just placeholders.

[Jacob-Stevens-Haas]

Good catch, and thanks for providing this! I can reproduce on 1.7.5 and on master. The bug was introduced when creating EnsembleOptimizer was added to replace the old method of ensembling via model.fit(). It appears that the inner optimizer requires a 3x5 initial guess (correctly), whereas the EnsembleOptimizer requires a 3x4 (incorrectly), which is why you were getting errors for either one. See next comment.

Modified your script, since yours had some name errors:

import pysindy as ps
from pysindy import SINDy
import numpy as np

feature_names = ['x', 'v', 'F', 'u']

x_library_functions = [
    lambda: 1,
    lambda x: x,
]
x_library_functions_names = [
    lambda: 1,
    lambda x: x,
]
X_Data = np.zeros((5, 3))
U_Data = np.zeros((5, 1))
dT = .1

initial_guess_manual = np.array([[0, 0, 1, 0, 0], [1, 1, 1, 1, 0], [0, 0, 0, 0, 1]])
lib = ps.feature_library.CustomLibrary(library_functions=x_library_functions,function_names=x_library_functions_names,interaction_only=True)
model = SINDy(feature_library= lib,
              feature_names=feature_names,
              optimizer=ps.SR3(initial_guess=initial_guess_manual, normalize_columns=False, threshold=.1, thresholder='l1', max_iter=50))
model.fit(x=X_Data, t=dT, u=U_Data, library_ensemble=True, ensemble=True, quiet=True)

But even obeying the deprecation warning and creating the EnsembleOptimizer yourself gives the same error, and that error is on master as well. E.g.:

opt_inner = ps.SR3(initial_guess=initial_guess_manual, normalize_columns=False, threshold=.1, thresholder='l1', max_iter=50)
opt_outer = ps.EnsembleOptimizer(opt_inner, bagging=True, library_ensemble=True)

model = SINDy(feature_library= lib,
              feature_names=feature_names,
              optimizer=opt_outer)
model.fit(x=X_Data, t=dT, u=U_Data, quiet=True) # remove quiet=True for `master` branch

[Jacob-Stevens-Haas]

Actually, it looks like the bug predates EnsembleOptimizer. I believe it happens when dropping terms out of a library ensemble, but not adjusting the initial guess. I'm not sure what the right solution is. Should library ensembling w/initial guess (a) be allowed, and EnsembleOptimizer removes terms that correspond to features left out of the library bag, or (a) be disallowed, since it's unclear whether an initial guess is meaningful when modified in that manner? Obviously the error only occurs in optimizers that actually use the provided initial guess.

I looked at the Ensemble-SINDy paper of @urban-fasel, and the only time it mentions an initial guess is the one STLSQ generates for itself. So it feels like both (a) and (b) are reasonable - (a) allows people to investigate if this method would be useful, but (b) makes it easier for newer users to identify what SINDy methods make sense for their problem.

@urban-fasel @akaptano - what are your thoughts on the matter? And @IamKFJ, can you let us know, in your use case, how you chose library ensembling, SR3, and your initial guess?