Feature requests about logstash-filter-aggregate

[fbaligand]

OK, so let me answer you for each of your 3 points :

• Point 1 : Presently, aggregate plugin doesn't provide such a feature. That said, it is possible to add a feature that creates a new event tagged '_aggregatetimeout' each time an aggregate map falls in timeout.
  The event would contain taskid information and tag would be customizable by configuration.
  Then, using a "if" based on '_aggregatetimeout' tag, you could use the collate plugin to group events.
• Point 2 : if the metrics you speak are tied to a task with a task id, so aggregate plugin provides today such a feature. And the 2 samples in aggregate plugin documentation explain how to do this.
  If the metrics you speak are not tied to a task, but to an interval, so the metrics plugin provides such a feature.
• Point 3 : as for point 2, if your need is tied to a task with a task id, so aggregate plugin provides today such a feature. And yes, you can do that using code attribute.
  And as for point 2, if the metrics you speak are not tied to a task, but to an interval, so the metrics plugin provides such a feature to compute the count. Then you can simply use mutate filter to set a field (whatever its name) using the computed metrics.

[erik-stephens]

Tagging the event and letting it through instead of dropping should address the critical needs. I assume will want another config value to enable this new behavior. Preference on the parameter name? I'll start working on a PR unless anything else I should consider first.

Items 2 & 3 were more nice-to-have's and might not be worth complicating this plugin for. I get nervous about evaluating code blocks, especially if an exception there can lead to the pipeline being shut down but looks like this plugin has a catch-all exception handler.

[fbaligand]

Concerning point 1, I precise one thing. When timeout, a new event would be created with a tag. It is not the initial start task event which would be tagged.
Aggregate filter doesn't retain events, it just extracts informations from event, push it into aggregate map and let event go.
That said, concerning new fields for that feature, I think about 3 new fields :

• create_event_when_timeout (boolean, default value : false)
• timeout_tag (string, default value : _aggregatetimeout)
• timeout_taskid_field (string, default value : task_id)
  What do you think about ?

Concerning points 2 and 3, I can understand you get nervous about code field. But the code call is protected by a exception catch and all details are logged. Another protection is the field "map_action". It avoids to call code section when map is not in the good state (for example, do a update code only if aggregate map has already been created before).
That said, could you say how you would imagine plugin evolution to cover these needs ?

[erik-stephens]

Sounds good. I don't understand the need for timeout_taskid_field, though. As for my thoughts on how to implement:

• If create_event_when_timeout, create an aggregated event here before map data is expired and return it so it gets emitted.
• I don't know enough to know if changes to periodic flush will impact this or not.

Let me know if I'm overlooking something. Thanks.

[fbaligand]

To answer your 2 points :

• To me the update is more at line 248 :

expired_elements = remove_expired_elements()
@@last_eviction_timestamp = Time.now
if (create_event_when_timeout)
    return create_expired_events_from(expired_elements)
else
    return nil
end

• No impact. This feature allows to enable or disable periodical flush on plugin. For aggregate plugin, periodical flush is always enabled :
  https://github.com/logstash-plugins/logstash-filter-aggregate/blob/master/lib/logstash/filters/aggregate.rb#L235
• Finally, timeout_taskid_fieldallows to customize in "new aggregate timeout event", the field name where to store task_id value. So that "new aggregate timeout event" looks like :

{
"tags" => ["_aggregatetimeout"],
"task_id" => "<current_task_id>"
}

[erik-stephens]

I think we're on the same page regarding the gist of the patch. Regarding timeout_taskid_field, can that be handled by code block if the user wants that?

[fbaligand]

To be more precise, I would say to replace lines 248 and 249 (https://github.com/logstash-plugins/logstash-filter-aggregate/blob/master/lib/logstash/filters/aggregate.rb#L248) by the code block mentioned in my previous comment.

Concerning "timeout_taskid_field", code block can't process that : it is processed in other moment for another goal, in filter() method.
That said, we could say this attribute is optional, and if not filled, no taskid field is added in "aggregate timeout event".
Do you agree ?

[erik-stephens]

I was going to factor out the relevant bits from here when defining the expired aggregate event. Basically, create a new event and treat it as if it's an end-of-task event.

[fbaligand]

Hi,
When I click on your link, I got a http 404 error.
Could you check your link ?

[erik-stephens]

Sorry about that. I fixed the link.

[fbaligand]

To make the job, I suggest you draw inspiration from :

• few lines written in this comment
• code written in this plugin which does exactly what you want (create a new aggregate timeout event from expired event)

And if you need help, don't hesitate to ask :)

[erik-stephens]

Before I document and add unit tests, I would like to gauge how you feel about this change. My concern is that it complicates this plugin too much. It's been working ok for us but multiline event processing in logstash is tricky - maybe because I'm new. I may try a different approach if you think adds too much complexity to this plugin.

[suyograo]

@erik-stephens any update here..

[erik-stephens]

I've moved on, so it's ok to close.

[tony2944]

Hi

IN which version these changes are ?

[fbaligand]

This issue was a list of feature wishes.
It has been closed by his creator, but features have not been implemented.

If you want a particular feature, I invite you to create a new issue.

[tony2944]

Can you please share the Changes to be done in aggregate.rb and implementation in filter file....As we also want to implement same thing as mentioned in point 1.....I have done changes mentioned in your comment but I am getting method not found error for "create_event_when_timeout "

[erik-stephens]

@tony2944, it was hyperlinked in my comment but probably easy to miss. Here it is if interested:

master...erik-stephens:master

[tony2944]

Actually I am also looking for similar solution as mentioned in Point 1, When we don't know the end point of our aggregation so I want to use timeout and store the map value into event(New field). I don't want the count but I want particular map value to be added as field at the timeout.

I have also done the similar changes in my aggregate.rb plugin. I am getting new tag after timeout but I am not getting map value in it. Also I am facing one more issue after adding your changes in aggregate.rb i.e previously below mentioned code statement used to work
code => "map['New_message'] += event['internal_message']+'||' ; map['Service_Name'] = ''"
But now I am getting error couldn't able to find + method.

I am sharing my filter file

#Start the aggregation from the starting point as entry
if ("entry" in [internal_message]) {
aggregate {
task_id => "%{correlation_Id}"
code => "map['called_Service_Names'] = '' ; map['called_Service_Names_order'] = ''"
map_action => "create"
}
}
#Start the aggregation from the starting point as send
if("send" in [internal_message]) {
aggregate {
task_id => "%{correlation_Id}"
code => "'||'"
map_action => "update"
}

}
#Ending aggregation
if ([message] =~ /.+/ ) {
aggregate {
task_id => "%{correlation_Id}"
timeout => 4
timeout_tag => 'aggregated'
code => "
if map['_expired']
event['called_Service_Names_order'] = #map['called_Service_Names_order']
event.tag('keep')
end
"
create_event_when_timeout => true
}
if 'aggregated' not in [tags] {
drop {}
}
}

Please find the output below :
{
"@Version" => "1",
"@timestamp" => "2016-06-02T07:04:59.580Z",
"tags" => [
[0] "aggregated",
[1] "keep"
],
"host" => "XYZ",
"called_Service_Names_order" => [
[0] "aggregated",
[1] "keep"
]
}

And I am looking for something like this kind of response :
{
"@Version" => "1",
"@timestamp" => "2016-06-02T07:04:59.580Z",
"host" => "host name",
"called_Service_Names_order" => "||"
}

Could you please helping in resolving this issue.

aggregate.rb.txt

[tony2944]

HI erik-stephens

Is there any plan to fix these?.

[erik-stephens]

@tony2944 There are no plans to get these changes incorporated into the plugin, so I wouldn't expect much support and would treat it as dead code.

[alkuzad]

Why don't you implement that? Ruby is easy language
15.06.2016 16:58 "Erik Stephens" [email protected] napisał(a):

@tony2944 https://github.com/tony2944 There are no plans to get these
changes incorporated into the plugin, so I wouldn't expect much support and
would treat it as dead code.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#14 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADTZqtsevOI9p9gomxLPUroIn0-tD9tEks5qMBMdgaJpZM4Gd-DF
.

[tony2944]

Hi
If you guide me then i will fix it up.

[alkuzad]

@tony2944 can you pass full map into event for debugging ?

[tony2944]

Hi Alkuzad,
Thanks for your helping

@alkuzad I am also trying to move map to event at the timeout but it is not working for me. I have shared the ruby file and problem statement in my above post.
Could you please check from your side what needs to be added to fix this.
we already tried we need this fix to be made.

[fbaligand]

@tony2944
Do you agree that issue #33 matches your need ?

[tony2944]

Yes Agreed it matches our need

[fbaligand]

Ok.
It's nice if one feature can answer both needs.

[fbaligand]

Hi @tony2944

I have just released version 2.2.0, with a new feature : push_previous_map_as_event
I know this is not exactly what you describe in this issue.
But you might find it interesting.

You can find a good example here :
https://github.com/logstash-plugins/logstash-filter-aggregate#example-3

[fbaligand]

release 2.3.0 is done with timeout event generation !