is kneed picking the right knee point? about kneed

[Peterdha]

Hi @arvkevi , have been looking a bit closer at the code, I can't see anything wrong with your implementation, however I do believe the algorithm is flawed.
In the case where there is some jitter on the curve we want to find a knee for, it means we have a lot of local maxima and minima. This results in always resetting the threshold, resulting in outputting as a knee the last local max that was found.
My guess is the smoothing step is the culprit here, as a choice for a spline will always pass through the datapoints themselves. In my case that means the jitter stays in!
So I hacked "exponentially weighted moving average" in, as mentioned in the article to be an alternative for splines,
works like a charm now :-) !
As illustrated below.
Using the solution from Divakar on https://stackoverflow.com/questions/42869495/numpy-version-of-exponential-weighted-moving-average-equivalent-to-pandas-ewm

However the S-factor doesn't do a whole lot for me...

[arvkevi]

Thanks Peter, I'll look into this and I'm especially interested to see if S is working as expected.

[arvkevi]

Thanks for spending some time with the code @Peterdha

My guess is the smoothing step is the culprit here, as a choice for a spline will always pass through the datapoints themselves. In my case that means the jitter stays in!
This is what I originially thought could be the issue -- that perhaps your function was in a local maxima.

Do you think it's worthwhile to implement a better spline method? Or provide a keyword argument to implement ewma as an option?

I tried modifying the source code to kind='cubic' and kind='quadratic'

uspline = interpolate.interp1d(self.x, self.y, kind='cubic')

with no success eliminating the steps in your function.

[Peterdha]

hi @arvkevi

Do you think it's worthwhile to implement a better spline method? Or provide a keyword argument to implement ewma as an option?

I'm not sure if ewma is a good solution here, as it requires some tweaking to get a nice smooth curve.
Instead I would try to fit a higher order polynomial through it, for the use-cases I can imagine 5-7 order polynomial should suffice. (https://scikit-learn.org/stable/auto_examples/linear_model/plot_polynomial_interpolation.html#sphx-glr-auto-examples-linear-model-plot-polynomial-interpolation-py)

But then, if you go through the trouble of fitting a nice curve, you could just as well calculate the max of second derivative to get the knee point...

[arvkevi]

Thanks @Peterdha
I'm thinking to leave the algorithm as-is for now.
I'll create a branch to work on this a bit, reference this issue and maybe others will chime in 😄

[arvkevi]

fixed by #26