Aggregate filter has problem with JDBC input plugin with paging enabled about logstash-filter-aggregate

[fbaligand]

Some questions :

• When pagination is enabled, jdbc plugin makes a first sql query for first page, sends all results as events in Logstash pipeline and then immediately makes a second sql query for second page ?
• Your aggregate issue is that now you get one last tagged event for each page, and not anymore for each full query ?

[iti888]

•When pagination is enabled, jdbc plugin makes a first sql query for first page, sends all results as events in Logstash pipeline and then immediately makes a second sql query for second page ?

Yes. But since it is a separate query, it might take some time. So, immediately is no meaning.

•Your aggregate issue is that now you get one last tagged event for each page, and not anymore for each full query ?

Correct.

I think the root solution should be on JDBC plugin, need to issue a tag of "EOF" for determining the last record. Using timeout is not reliable. I was using 3 sec. but when system is too busy or something, last event timeout got triggered. I am using 20 sec now. But still might have problem when system hangs or CPU at 100%.

[fbaligand]

I agree that a new option in jdbc input to tag the last event would be the better and cleaner solution. Have you opened an issue for that ?

Then, concerning aggregate plugin, I have no better idea than increasing timeout, as you did.

[iti888]

I will open an issue with the jdbc input plugin in a minute.:)

[iti888]

added an issue to jdbc input plugin:
logstash-plugins/logstash-input-jdbc#188

[fbaligand]

I wish you this issue will be solved.

[iti888]

I hope so.