interpolate() should incorporate samples immediately before or after the query range, when available about victoriametrics

[mactyr]

@hagen1778, do you have a sense whether this is actionable? It's currently causing embarrassingly incomplete/incorrect charts in our customer-facing web application, so I'd like to understand whether a fix might be forthcoming or if I need to hunt for workarounds. 😅

I've noticed that keep_last_value() and keep_next_value() also don't appear to utilize samples outside of the query window (if I'm reading the code correctly). I don't use those functions in my application so that's not an immediate problem for me, but I would have expected them to look outside the query window in the same way I expected interpolate() to.

[hagen1778]

Hello @mactyr! In the current implementation, all transform functions have no knowledge about datapoints outside the interval. I don't know yet if this is actionable - I need feedback from @valyala who originally added this function.

[hagen1778]

Hey @mactyr! It is unlikely we're going to change transform functions to capture data before and/or after the requested interval, in the same fashion as rollup functions do. This could be a severe change.
But I think it could be possible to use data samples on the requested interval in order to extrapolate missing data points. It should work perfectly with monotonically increasing counter from your example, but will be partially correct for other cases. wdyt?

[mactyr]

Thanks for offering that change, @hagen1778, but extrapolating from the samples within the interval definitely isn't safe with our real data, where the rate of change varies quite a bit over time.

My first impulse is to ask if there could be a version of interpolate that is a rollup (like interpolate_rollup()?) but I see how it's a little unclear what the lookbehind window would mean for that function, which is probably why @valyala chose to implement interpolate() as a transform in the first place.

So my next question is whether you all could take another look at #3079 and see whether that might be actionable? What I'm really trying to do here is just get the increase() without assuming that the counter had a value of 0 if there's no sample found prior to the query window. Because I couldn't get the result I needed with the existing family of "increase" functions, I settled on the workaround of doing ascent_over_time(interpolate(metric{})) instead, which usually works well enough but has the problem described in this issue. If I had an increase-like function that worked as desired, I think I could get away from relying on interpolate() so much.