Add accumulator to pipeline about pond

[pjm17971]

I'd imagine that wouldn't work too well, since the interpolation between two events of the same time is not going to intersect an alignment boundary at all.

If you have a fixed timeseries (rather than one being streamed in), I'd probably suggest you de-dup the data first, since probably only you know what that means in your system. Sometimes you can say that the last point wins, sometimes they represent the same thing, sometimes there's just two points arrived at the same time and they need to be reconciled. None of these cases apply to our use cases, so I'm not exactly sure what the general solution is here.

There are a couple of helper functions on Collection (e.g. dedup()) that might help. In fact, our feeling is that duplicate timestamps in a TimeSeries is pretty much a no-go, to the extent that we're considering changing the backing data structure to not allow it at all (i.e. move to a Map).

BTW, here's the linear interpolation in the align code:
https://github.com/esnet/pond/blob/master/src/pond/lib/processors/aligner.js#L175

It takes Events in with addEvent() at the bottom of this file and processes them to emit new events. There's a few different modes that can happen in, but it should be relatively easy to follow. It's really just managing boundaries with getBoundaries() since the last event and interpolating across them, to some limit of boundaries.

[pjm17971]

For the align and fill functions, the logic is simple enough to explain:

We wait for an initial event (or point) (you have to have one good point to begin), then we wait for the next good point. On the boundaries (say every 30s) in between those two points we linearly interpolate the the estimated value at that point, up to a limit of a specified number of points. The math for this is simple linear interpolation, but I can point you to that piece of code if you'd like.

That code is used in our collection system and you can also find tests for it. We're pretty sure it does what we expect it to do, but if you have a specific example of something unexpected I can look at it for you.

For 2, sure that sounds entirely reasonable and has been asked about before, but other than looping over the timeseries, I don't think there's a way to output that right now. You can get that end result (i.e. aggregate over a window of globally).

I guess the question is if it should generically reduce a field (e.g. units) or ts.reduce("units", "total", (accum, value) => accum + value}, initValue), or should we to assume it's an accumulation and do something like ts.accumulate("units", "valid", value => value > 0 ? true : false) to events above 0, or in your case ts.accumulate("units", "total", value => value) (default). And then I suppose it should maybe do that per-window/per-key as well.

[jonathanwoahn]

Yeah, if you wouldn't mind pointing me to where the code is for the interpolation, that would be helpful. Also, I'll do a few more test cases and see if I can provide some concrete examples of what I'm talking about.

For 2, I'd guess the reduce and accumulate functions would be for different use case scenarios. For my particular case, ideally it would be the accumulated value up to the row for events above 0.

Just for some context, here's what we have going on. I have a bunch of time series data that records units produced on an infrequent interval--up to an hour or more. We have a number of events that take place in the intermediate time, but they don't always have intermediate unit values associated with them.

There are three things I'm really interested in this library for. First, the ability to use the fill method to complete the intermediate values. Second, the ability to align the results with a specific timeframe (i.e. we want to output the results on consistent 30 minute intervals). And third, finally, if we can get the accumulated value for each of the intermediate data points. This library looks like it'll easily be able to handle the first two, but I haven't been sure if it already had the capability for the third one.

Thanks for your responsiveness

[jonathanwoahn]

Related to my question 1 from above, how does the align function work when you have two events that have the exact same timestamp? This is one place where I'm seeing some really odd behavior that I'd like to understand better.