Track timeout time based on a defined timstamp field rather than the time of the platform about logstash-filter-aggregate

[c0da]

Hello @fbaligand ,
I'm trying it right now and processing logs from 2016 :) It seems it's working perfectly fine.. Thank you very much for this enhancement!

[fbaligand]

Nice !
Happy to see that it match needs to both of you @c0da and @SolomonShorser-OICR !
The way to implement this feature is now quite clear to me, So I will do it soon !

Stay tuned... :)

[fbaligand]

Hi @SolomonShorser-OICR and @c0da,

I have the pleasure to announce you that the feature is now released !
You can download right now v2.8.0 and use the brand new option "timeout_timestamp_field" that will allow you to process old logs !

Feel free to give feedback about the feature !

[SolomonShorser-OICR]

Hmmmm...

Maybe for timeouts that are based on a user-specified field, the timeout management is checked when each event (or n events) is processed.

Calculating timeouts would need to be based on the user-specified timestamp field. So when a user specifies that @log_based_timestamp should be used for the aggregation, new maps are created such that their "map create time" is set to the value of @log_based_timestamp when they are first created. A timeout is detected when the most recent @log_based_timestamp - "map create time" > "timeout amount".

Checking for expired maps when processing every n events might not be as performant as doing it every 5 minutes in real-time, but I might be willing to live with that if it lets me get aggregate data on old logs.

[fbaligand]

@SolomonShorser-OICR
First, sorry for my late answer.
So tell me if this implementation matches your need :

• add a new option named 'timeout_timetamp_field', that contains field name to use to compute timeout
• when this option is set, each time an event arrives in aggregate filter, aggregate filter computes difference between event timestamp and aggregate map creation timestamp. If timeout is reached and push_map_as_event_on_timeout=true, map is pushed as new event and then removed from aggregate filter maps.

Does it match your need ?

[fbaligand]

Hi @SolomonShorser-OICR,

In the very first version of aggregate plugin, timeout management was based on event's @timestamp field.
And because of that, I had problems when parsing old logs (several days before). So I changed the code to use "current" time.
The problem I had was that timeout (set to 30 minutes) occurred before all "one task" logs were processed.
That's why I changed that to use "current" time.

For this reason, I'm afraid it's not possible to manage to match your need.

So that I understand better your need, could you provide your aggregate configuration that do what you need ?

[SolomonShorser-OICR]

I was trying to do get a count of "website visitors" for some web applications. It is very similar to this example. After a certain number of "clicks" or visits, I set some fields, depending on which parts of the site they visited.

There is no real "end" event that I can use to end the aggregation, so I was relying on "timeout" to end aggregation - if a user has not "clicked" after n minutes, it is considered a single session. The aggregation will work on logs that are currently being written, but I've been tasked with analyzing data in old logs.

I'd also mention that I'm creating a @log_timestamp which is based on the timestamp field in the log files, and I'm using that as the main timestamp field for the index in kibana. Since this field contains the historic timestamps, it would be nice to be able to aggregate based on @log_timestamp, and not @timestamp.

[fbaligand]

I should explain timeout management :
timeout management is launched every 5 seconds and look at all pending aggregate maps to know if there are "expired". And so, to know if a map is "expired", I compute "current time" - "map create time". If it is superior to timeout, it is expired.
So the problem is : how to detect "timeout occurs" based on a timestamp field which is very old ?
I mean : given that I detect timeout based on difference between "current time" and aggregate map initial timestamp field time, the timeout will occur always.
If you have a way to detect timeout based on an old timestamp field, tell me, I would be happy to know it.

[c0da]

that would be absolutely great, @fbaligand!!

[fbaligand]

@c0da Great to see you enjoy the feature !

@SolomonShorser-OICR does it match your need ?

[SolomonShorser-OICR]

@fbaligand That sounds like it would let the aggregate plugin work on historic data! :)

[fbaligand]

@SolomonShorser-OICR @c0da
You can find documentation to use the feature here :
https://www.elastic.co/guide/en/logstash-versioned-plugins/current/v2.8.0-plugins-filters-aggregate.html#v2.8.0-plugins-filters-aggregate-timeout_timestamp_field

[SolomonShorser-OICR]

@fbaligand I saw, this looks great! I've had to put some of my Kibana experimental work on the back-burner for a while, but I hope to get back to this and try it out in the next few weeks.

[fbaligand]

Great !
Happy to see you enjoy the feature !

[fbaligand]

As feature works as expected, I close the issue.