Failure to Sparsify Forced Linear System with Large Parameter Differences about pysindy

[akaptano]

I tend to think you are right that the 10^3 order of magnitude difference in the parameters is the issue here. Certainly it is preventing you from systematically turning the threshold up and down, since the identified coefficients are very different for each ODE. I am not sure if this will fix the issue, or you have tried it already, but you could try normalizing each of the x_1,...,x_4 to the unit ball, before building the SINDy library. So x_1 -> x_1/max(|x_1|) and so on. If the difference in parameter sizes in representative of a mix of "slow" and "fast" dynamics, take a look at https://arxiv.org/pdf/2006.00940.pdf.

If you would like an easier but perhaps more ad-hoc approach, look at the SR3 branch. There we have SR3 implemented with initial guesses and variable thresholding. That means for each x_i you can set a different threshold lambda_i. You can even set higher thresholds only on quadratic terms.

Also, as a first check to your identified model that you posted above, could you attach the plots of "true" and "identified" Xdot? It's possible that even though it does not capture the true ODE system well, it is still approximating Xdot pretty well. A picture like this could help inform what is going wrong here.

[JGPrior]

Hi Alan, thanks for the rapid response.

Firstly, for the plots of predicted vs computed:

As you said, the model seems well fit to the data, it just doesn't capture the original ODE. Just as a side note: the above plots are for the training data. I haven't generated a validation set yet, but I'm sure this would demonstrate some degree of overfitting, particularly in x2_dot.

I've also attempted to fit SINDy to data normalised in a unit hypersphere, as you suggested:

Again, when I plot the derivatives for a model based on this data the fit is good, arguably better than originally in x2_dot:

But the ODEs are somewhat baffling...

This result comes using a threshold of 10, but there are a lot of remaining parameters that exceed that threshold somehow, even with alpha set to 0... I Have a feeling that I've messed something up here but it's hard to tell. I guess this is also in part because even a well-sparsified result is going to be fundamentally different to the original ODEs I used to generate the data, owing to the normalisation.

I'm definitely interested in the multi-scale DMD + SINDy solution you've proposed here. From the plots it's clear to see that my system has different speed dynamics, which would be entirely expected of a damped oscillating system such as a suspension model. However I may need to read over it a bit more before attempting, as I'm sure there are some differences in my application to some of the basic cases presented (forced, damped oscillations etc). Out of interest, is there a recommended repository for a DMD package?

Apologies for the long winded response, I'll be interested to know if this has provoked any further thoughts.

[akaptano]

Yeah the threshold not working could be a red flag that you're not using the module right. Could you post your code (at least the SINDy relevant part)? And get rid of the normalization for now so we can compare the coefficient values to the true model directly by eye.

[JGPrior]

Both the training data and the code:

demo_Qcar.zip

[akaptano]

Okay, it looks like this works fine with poly_order = 1 (since this is a linear system). With poly_order = 2, it also fits Xdot very well, but the odeint(Xdot) is liable to blow up if you overfit with quadratic terms. Attached are my results for poly_order = 1, with nothing else changed (except I integrate the identified model to check the fit on x1,...,x4 and not just on x1dot,...,x4dot). Note that the model still does not agree with your ground truth, but notably it disagrees on precisely the locations where the coefficients are very large. For whatever reason, the fit of X is still very good -- the model cares little about the values where it is wrong. This appears to be why your model can identify a very different looking set of coefficients.

My guess is that you are not training on data for which the uncertainty in these coefficients can be reduced significantly (the "fast" dynamics are not resolved...?). It is quite suspicious that the coefficients in the "large" locations can be off by orders of magnitude, and the prediction of X is still excellent. I haven't looked at the physics so I'll leave that to you.

Lastly, there is a small issue during model.simulate (if you want to compute the prediction of X rather than just Xdot). The algorithm sends u = u_train to scipy.interp1d, which requires the extra argument fill_value="extrapolate" in order to extrapolate beyond the values that are already contained in u_train. So all you need to do is change the line in pysindy.py to
"u_fun = interp1d(t, u, axis=0, kind="cubic", fill_value='extrapolate')". Changing this behavior to be the default may be a good change, @briandesilva.

Let me know if you have any more questions.

Qcar_x.pdf
Qcar_xdot.pdf

[briandesilva]

@akaptano, thanks for your work on this. I'll look into changing the default behavior of SINDy.simulate. Other people have reported similar problems in the past.

[briandesilva]

Out of interest, is there a recommended repository for a DMD package?

PyDMD is the best one that I'm aware of. It implements lots of different DMD variants and has a good set of tutorials for getting started.

As for the issues with normalization, there's a bit of subtlety when it comes to normalizing the library terms. For example, if you use a STLSQ optimizer with a threshold of 0.5 and set normalize to true

STLSQ(threshold=0.5, normalize=True)

you don't necessarily get an ODE with coefficients all larger than 0.5 back. The reason for this is because the normalization process boils down to

1. Compute the library features Theta(X) (each column corresponds to a different function);
2. Divide each column of Theta(X) by its norm, saving the norms for later. Let Theta*(X) be the resulting array;
3. Solve X' = Theta*(X) Xi using STLSQ;
4. Multiply the rows of Xi by the norms saved in step 2

All the nonzero values of Xi that are found in step 3 will be larger (in magnitude) than the threshold you specified, but after step 4, this property no longer holds.

[akaptano]

Thanks Brian, also @JGPrior, just so you are aware, the thresholding can be a bit subtle, especially with SR3, which does not hard-threshold. You can look in the published SR3 papers by Kathleen and see that sparsity is not enforced on Xi, but rather on a matrix W, and then there is an extra term in the optimization to approximately enforce W = Xi. The sparsity enforcement is "soft" or "relaxed" in this sense, and in fact the term promoting sparsity lambda * R(W), requires that lambda = threshold^2/nu, even though the L0 term (or L1 term) in R(W) is still chopping off terms in W > threshold. So changing threshold indeed changes where the L0 thresholding takes place (as desired), but it also scales the sparsity promotion term lambda * R(W) like O(lambda^2). Changing the "nu" parameter also has some subtlety in this way, so in general I recommend using STLSQ or other simpler algorithms rather than SR3. The primary advantage of the SR3 algorithm, as it appears in the current PySINDy code base, is that it allows for the variable thresholding, initial guesses, and linear equality constraints C * Xi = d, as I mentioned above. I believe Brian just merged this SR3 branch with the main branch, so you should have access to these advanced functionalities if you desire.