patrickjennings / logstash-pfsense Goto Github PK

Log attatched. Looks like there is a pattern issue for PFSENSE_LOG_DATA

I have reloaded the pfsense2-4.grok file and restarted logstash with no result.

testing.txt

Hello,
I don't why but in my case filterlog wasn't the first string of the line but the last word of the line.

If you have the same issue just modify the following line to :
if [prog] =~ /^filterlog$/ { => if [prog] =~ /filterlog/ {

I am sending logs from a pfsense 2.5.2 firewall, i got it working but my firewall messages were not getting Groked.

i did some investigation and the the cause was the Regex statement in this line.

if [prog] =~ /^filterlog$/ {

the problem was that the statement was just looking for the word "filterlog" in the [prog] field, when the real contents of the field was "filterlog[24523]", i assume the number is the PID of the process, so i added some regex to the if statment and now it works for most messages, im still trying to figure out an issue with Grok itself

for those of you who have this issue just replace the line with the following line and it should work

if [prog] =~ /^filterlog[([0-9]*)]$/ {

Hi there,
I had to change the PFSENSE_LOG_DATA pattern so it would match interface names with a vlan such as re0.1. The pattern %{WORD:iface} does not match properly so I changed it to %{DATA:iface} instead

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule})?,,(%{INT:tracker}),(%{DATA:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),

Seems to be working this side :)

I had some problem when i downloaded this file from github to logstash folder , but when i restart this services, it's appear error :
[2020-09-16T09:51:33,670][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, } at line 122, column 22 (byte 2959) after output {\n if [type] == \"syslog\" {\n elasticsearch {\n hosts => 10.1"}

Anyone have the way for fix it? Many thanks.

On multiple instances, I only see tcp_flags values of RA or FA. When the field has A, FPA, or PA (the only ones I can reliably find on search) tcp_flags doesn't populate at all in Elasticsearch. An example search result file (csv) is attached based on the following search:
tags:dcSense AND direction:in AND src_ip:(192.168.0.* OR 192.168.1.* OR 192.168.10.* OR 192.168.11.* OR 192.168.14.* OR 192.168.15.) AND !(src_ip: (fe80 OR 127.0.0.1)) AND !(dest_ip:( 192.168.0.* OR 192.168.1.* OR 192.168.0.255 OR 192.168.1.255 OR 192.168.10.255 OR 192.168.11.255 OR 192.168.14.255 OR 192.168.15.255 OR 255.255.255.255)) AND !(tcp_flags:(RA OR FA))
Outbound-Blocked-Traffic.txt

I am using your pfSense grok and conf.d files (though I've made some minor modifications along the way, including to GeoIP tag outgoing traffic). All of that is working fine, I just am not getting the tcp_flags field to populate on values other than RA or FA.

Hi
can you help me understand why such lines do not match the grok filter?

7,,,1000000105,igb1,match,block,in,6,0x00,0x00000,1,Options,0,32,fe80::f29f:c2ff:fe1a:2665,ff02::1,HBH,PADN,RTALERT,0x0000,
or
5,,,1000000103,ovpnc1,match,block,in,4,0x0,,64,65242,1480,none,17,udp,173,10.20.30.40,10.20.30.101,

As a OPNsense user https://github.com/fabianfrz/opnsense-logstash-config, works better and I have not tested logstash-pfsense. I wonder if it might make sense to merge. I have reworked https://github.com/fabianfrz/opnsense-logstash-config to use integration testing so logstash-pfsense could be merged into this as well we could have a unified OPNsense/pfSense Logstash config.

The Common Destination Ports visualization does not import into Kibana 6.5.4 - gives an error "Could not locate that index-pattern-field (id: dest_port.keyword)".

The Blocked IPs & Ports visualization also gives an error when imported into Kibana 6.5.4 "Could not locate that index-pattern-field (id: src_ip.keyword)".

Please advise how to fix these issues.

Could you please specify a license for this repo? I'd like to use the grok pattern in another project if the license allows.

Thanks!