airbus-cyber / graylog-plugin-aggregation-count

Alert condition plugin for Graylog to perform aggregation

WARNING:

This plugin is not maintained anymore.
Graylog has evolved and its built-in "Filter & Aggregation" feature allows to use aggregation in Event Definition.
So this plugin is not needed anymore.

Aggregation Count Plugin for Graylog

Alert condition plugin for Graylog to perform aggregation

The alert condition triggers whenever the stream received more or less than X messages matching the same values of some message fields and with distinct values of other message fields in the last Y minutes.

Perfect for example to be alerted when there are brute-force attempts on your platform. Create a stream that catches every authentification failure and be alerted when that stream exceeds a given threshold per user.

Also perfect for example to be alerted when there are network port scans on your platform. Create a stream that catches your network traffic and be alerted when that stream exceeds a given threshold per source and per destination and with distinct values of port.

Please also take note that only a single alert is raised for this condition during the alerting interval, although multiple messages containing different values for the message fields may have been received since the last alert.

Example of raised alert:

Version Compatibility

Plugin Version	Graylog Version
4.1.x	4.2.x
4.0.x	4.1.x
2.2.x	3.3.x
2.1.x	3.2.x
2.0.x	3.2.x
1.2.x	3.0.x
1.1.x	2.5.x
1.0.x	2.4.x

Installation

Download the plugin and place the .jar file in your Graylog plugin directory. The plugin directory is the plugins/ folder relative from your graylog-server directory by default and can be configured in your graylog.conf file.

Restart graylog-server and you are done.

Usage

First you have to select the alert type Aggregation Count Alert Condition

Then, you can configure the Grouping Fields and the Distinction Fields to count messages respectively with the same values and with distinct values.

Optionally you can add a Comment about the configuration of the condition.

You can also set all the common parameters : Threshold Type, Threshold, Search within the last, Execute search every and Search Query.

Build

This project is using Maven 3 and requires Java 8 or higher.

• Clone this repository.
• Run mvn package to build a JAR file.
• Optional: Run mvn jdeb:jdeb and mvn rpm:rpm to create a DEB and RPM package respectively.
• Copy generated JAR file in target directory to your Graylog plugin directory.
• Restart the Graylog.

License

This plugin is released under version 1 of the Server Side Public License (SSPL).

graylog-plugin-aggregation-count's Issues

Does it work on Graylog 3.0.0?

I am trying to use this plugin to trigger a condition based on cardinality.

In my case, I am examining a stream of DNS query errors from a farm of recursive DNS servers. The stream contains only DNS query errors and fields with the self explanatory names "dns_query_name", "dns_client_addr", etc.

I am trying this configuration but despite having more than 10 distinct failed names in a 5 minute interval the condition will not trigger.

Does it work? I am using Graylog 3.0.0 and at least the wizard to configure the condition loads properly.

backlog: 0 comment: <empty> distinction_fields: dns_req_name grace: 0 grouping_fields: <empty> query: <empty> repeat_notifications: false threshold: 5 threshold_type: MORE time: 5

Thanks!

can't use distinction/grouping fields

Hello,

I have Graylog 3.
I'm trying to be alerted when an IP is trying to bruteforce my mail server.

With your plugin, I add this configuration :

backlog: 10
comment: <empty>
distinction_fields: <empty>
grace: 60
grouping_fields: <empty>
query: message:("error=authentication failed for" OR "authentication failed: authentication failure") AND _exists_:clientIP AND NOT clientIP_country_code:FR AND _exists_:clientIP_country_code
repeat_notifications: false
threshold: 10
threshold_type: MORE
time: 30

It works but not grouping alert by IP.

If I change the alert by adding :
distinction_fields: clientIP

The alert sends me all my log, like as query doesn't used anymore :/

Could you help me please ?

Wrong query when email alert

Hello,
In GL 3.0.0, with aggregation plugin 1.1.0, when an alert is sent by mail, you can have the link to graylog webUI search :
${if stream_url}Stream URL: ${stream_url}${end}

In my case, the aggregation condition is :

backlog: 50
comment: <empty>
distinction_fields: <empty>
grace: 30
grouping_fields: clientIP
query: message: "Vous avez atteint la limite d'envoi"message: "Vous avez atteint la limite d'envoi" AND _exists_:clientIP_country_code AND clientIP_country_code:FR
repeat_notifications: false
threshold: 10
threshold_type: MORE
time: 60

But, when I receive a mail alert, I have :
Stream URL: https://xxxxxxxxxxxxxxxxxxxxxx/streams/5c5aa6f582e4200209a034cc/messages?rangetype=absolute&from=2019-03-20T13:47:31.456Z&to=2019-03-20T14:47:31.456Z&q=*

Query is the default one, not what I have wrote

Thank you for your help

Alert does not trigger if backlog is 0

If the backlog is equal to 0 the alert is not triggered.
Graylog 3.0.2.
graylog-plugin-aggregation-count 1.2.0

How query works?. I'm unable to filter using query on Graylog 3

Hello,

I'm trying to use your plugin to detect brute force attacks, but I want to filter some IP that are allowed and clients that are already blocked, but query doesn't work for me.

My configuration is:
backlog: 1 comment: <empty> distinction_fields: <empty> grace: 0 grouping_fields: remote_addr query: NOT status: 403 repeat_notifications: false threshold: 1000 threshold_type: MORE time: 10

but when I receive alerts from this plugin, are from an IP that is already blocked and gets an status 403.
I've tested that filter on graylog and works fine, that IP is filtered and doesn't appear, so looks like a problem with query configuration in plugin.

My graylog version is 3.0.0:
Version: 3.0.0+db6cf59, codename Space Moose JVM: PID 2470, Oracle Corporation 1.8.0_181 on Linux 4.9.0-8-amd64

i'm doing something wrong?

EDIT: I've tested the same filter on a simple message count alert and seems to be working, but this plugin doesn't.

Thanks.

Timestamp of events is at the start of the defined time period

v2.2 (Because I'm still running graylog 3.3x)

When creating an event definition and a "search within the last " setting of, say, 1 hour is defined then any resulting event triggered by the definition will have its timestamp as the beginning of that aggregation time range and not the time that the event was created

This seem wrong to me...the event pictured was actually created by the system at 16:57...

Throw the right exception to get more context

When an error occurred the wrong exception is thrown so Graylog is not able to give us more context to understand what happened.

I think this issue is also present in the Correlation Count plugin.

Related logs :

[EventProcessorExecutionJob] Event processor <aggregation-count/5f97c5d27768f006933560b5> failed to execute: Couldn't create events for: EventDefinitionDto{id=5f97c5d27768f006933560b5, title=abcdef12345, description=Generated by the alert wizard, priority=2, alert=true, config=AggregationCountProcessorConfig{type=aggregation-count, stream=5f97c5d27768f006933560ac, thresholdType=MORE, threshold=0, groupingFields=[], distinctionFields=[], comment=Generated by the alert wizard, searchQuery=*, searchWithinMs=60000, executeEveryMs=60000}, fieldSpec={}, keySpec=[], notificationSettings=EventNotificationSettings{gracePeriodMs=0, backlogSize=500}, notifications=[Config{notificationId=5f97c5d27768f006933560b3, notificationParameters=Optional.empty}], storage=[Config{type=persist-to-streams-v1, streams=[000000000000000000000002]}]} (retry in 5000 ms)

[EventProcessorEngine] Caught an unhandled exception while executing event processor <aggregation-count/abcdef12345/5f96fda37768f006932f0e36> - Make sure to modify the event processor to throw only EventProcessorExecutionException so we get more context!

Aggregation table in notification

Hello! Is that possible to create notification like this using this plugin?

Build is getting failed while compiling the source

I am trying to compile the source using "mvn package" on mac. But the build is getting failed and getting below errors / exceptions. Please help me to resolve the issue

[INFO] Unpacking /Users/divagu/.m2/repository/com/github/eirslett/yarn/1.22.5/yarn-1.22.5.tar.gz into /Users/divagu/Desktop/graylog-source/graylog-plugin-aggregation-count/node/yarn
[INFO] Installed Yarn locally.
[INFO]
[INFO] --- frontend-maven-plugin:1.9.1:yarn (yarn install) @ graylog-plugin-aggregation-count ---
[INFO] Running 'yarn install' in /Users/divagu/Desktop/graylog-source/graylog-plugin-aggregation-count
[INFO] yarn install v1.22.5
[INFO] warning ../../../package.json: No license field
[INFO] [1/4] Resolving packages...
[INFO] error Package "graylog-web-plugin" refers to a non-existing file '"/Users/divagu/Desktop/graylog-source/graylog2-server/graylog2-web-interface/packages/graylog-web-plugin"'.
[INFO] error Package "graylog-web-plugin" refers to a non-existing file '"/Users/divagu/Desktop/graylog-source/graylog2-server/graylog2-web-interface/packages/graylog-web-plugin"'.info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46:52 min
[INFO] Finished at: 2022-01-24T16:08:56+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.9.1:yarn (yarn install) on project graylog-plugin-aggregation-count: Failed to run task: 'yarn install' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

How to alert if the field value is larger then 50?

I have some log that struct like below
{"type":"ItemException","GameSvrId":"game-000-0001","EventTime":"2019-02-18 19:43:23","GameAppId":"loveworld-local","PlatId":2,"LoginChannel":"unity-editor","OpenId":"dt201901161950","RoleId":37017695861859,"Format":"1.0.0","Sequence":1550490199,"ItemId":121,"ItemType":12,"OperateType":0,"Num":5,"NumberNum":5}

and I want to be alertd if the value of "NumberNum" is larger then 50 and need to group by "RoleId"

I make the configure like below, but it seems not work.

so what should I do ? If I there is someting wrong with my config?

issue with "less than" threshold type

When I create a "less than" rule, the alert in graylog is never triggered.
To reproduce this issue, it's simple :

• Create a new rule with a matching condition that doesn't exist (ex: source must match 1.2.3.4)
• Configure less than 1 message in the last 1 minute

The alert is not trigerred and we can see these errors:

2022-10-07T18:12:50.158+02:00 DEBUG [JobExecutionEngine] Execute job: test rule less than/63404c4199c095396f04a043/event-processor-execution-v1 (job-class=EventProcessorExecutionJob trigger=63404c4199c095396f04a044 config=Config{type=event-processor-execution-v1, eventDefinitionId=63404c4199c095396f04a042, parameters=AggregationCountProcessorParameters{type=aggregation-count, timerange=AbsoluteRange{type=absolute, from=2022-10-07T15:55:49.413Z, to=2022-10-07T15:56:49.413Z}}, processingWindowSize=60000, processingHopSize=60000})
2022-10-07T18:12:50.158+02:00 DEBUG [EventProcessorEngine] Executing event processor <test rule less than/63404c4199c095396f04a042/aggregation-count>
2022-10-07T18:12:50.163+02:00 DEBUG [EventProcessorExecutionJob] Event processor <test rule less than/63404c4199c095396f04a042> couldn't be executed because of a failed precondition (retry in 5000 ms)

tested on Graylog 4.2.12 with wizard 4.2.2 and aggregation count 4.1.1 / Graylog 3.3.16 with wizard 3.3.0 and aggregation count 2.2.0

There is more than 1 event in input Graylog but for the rule test, there is no event in the concerned stream.

Not functional

Hi,

I tested this plugin for Graylog version 3.1.2-1 and it is not working (as if I wasn't even installed)
Checked the logs but no errors were noticed.

Thank you

Include messages in email alert notification

I am using v1.2.1 of the plugin and am trying to figure out how to get it to include the "grouped" messages that met the criteria in the email notification for the alert that is raised.

My alert configuration looks like this:

backlog: 10000
comment: Someone created more than 50 distinct files in 5 minutes
distinction_fields: ObjectName
grace: 0
grouping_fields: SubjectUserName
query: EventID:4663 AND AccessMask:0x100
repeat_notifications: false
threshold: 50
threshold_type: MORE
time: 5

My notification configuration looks like this:

body: ##########
Alert for: ${alertCondition.title}
Alert Description: ${check_result.resultDescription}
Date: ${check_result.triggeredAt}
##########

${if backlog}Last messages accounting for this alert:
${foreach backlog message}
##########
${message}
##########
${end}${else}<No backlog>
${end}
email_receivers: <empty>
sender: [email protected]
subject: Graylog alert: ${alertCondition.title}
user_receivers: johnd

So if I go create 50 files that causes the alert to be raised, I do get the email notification, but it shows:

##########
Alert for: File Creations
Alert Description: Stream had 0 messages in the last 5 minutes with trigger condition more than 50 messages with the same value of the fields SubjectUserName, and with distinct values of the fields ObjectName. (Current grace time: 0 minutes)
Date: 2019-07-19T12:26:09.618Z
##########

<No backlog>

In the end, I basically am trying to say if a user creates 50 files in 5 minutes (regardless of how many messages it needs to look through), I want to get an alert notification email showing me the 50 (or more) "messages" (as they include the filename) that met the criteria for that user.

What am I missing here to make this actually work? Or is it not possible in this scenario?