patrickjennings / logstash-pfsense

Logstash configuration for pfSense syslog events.

logstash-pfsense

Logstash configuration for pfSense syslog events. Comes with a dashboard for displaying blocked events from the firewall.

Includes a modified logstash configuration to work with the latest pfSense release v2.5 and Elastic services release v7.

pfSense Configuration

Enable remote logging in the pfSense web UI by going to:

Status -> System Logs -> Settings

Make sure that the "Log Message Format" is set to "BSD (RFC 3164, default)".

In Remote Logging Options, check "Enable Remote Logging", and add your remote Logstash server to the "Remote log servers". For example:

192.168.1.100:5140

Finally, check the "Everything" checkbox for "Remote Syslog Contents".

Logstash Configuration

The directories pipelines and patterns are specific to the logstash configuration.

You will need to edit ./pipelines/30-outputs.conf to set the elasticsearch host connection.

Kibana

Import the visualizations.json and dashboard.json.

K8s

There is an example Kubernetes kustomization for running ECK through the Elastic Operator.

Configuring and deploying the Elastic services in this way is beyond the scope of this document. But the yaml resources in infrastructure should give an idea of how to get that deployed.

Logstash is no longer a default component of the deployed Elastic services. logstash.yaml will show a way to deploy Logstash along side the other containers.

logstash-pfsense's Issues

Issue with [prog] regex in 11-pfsense.conf

I am sending logs from a pfsense 2.5.2 firewall, i got it working but my firewall messages were not getting Groked.

i did some investigation and the the cause was the Regex statement in this line.

if [prog] =~ /^filterlog$/ {

the problem was that the statement was just looking for the word "filterlog" in the [prog] field, when the real contents of the field was "filterlog[24523]", i assume the number is the PID of the process, so i added some regex to the if statment and now it works for most messages, im still trying to figure out an issue with Grok itself

for those of you who have this issue just replace the line with the following line and it should work

if [prog] =~ /^filterlog[([0-9]*)]$/ {

Problem : Cannot create pipeline {:reason=>"Expected one of #, } at line 122, column 22 (byte 2959) after output {\n if [type] == \"syslog\" {\n elasticsearch {\n hosts => 10.1"}

I had some problem when i downloaded this file from github to logstash folder , but when i restart this services, it's appear error :
[2020-09-16T09:51:33,670][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, } at line 122, column 22 (byte 2959) after output {\n if [type] == \"syslog\" {\n elasticsearch {\n hosts => 10.1"}

Anyone have the way for fix it? Many thanks.

Pattern %{PFSENSE_LOG_DATA} not defined

Log attatched. Looks like there is a pattern issue for PFSENSE_LOG_DATA

I have reloaded the pfsense2-4.grok file and restarted logstash with no result.

testing.txt

Kibana visualizations "Could not locate that index-pattern-field"

The Common Destination Ports visualization does not import into Kibana 6.5.4 - gives an error "Could not locate that index-pattern-field (id: dest_port.keyword)".

The Blocked IPs & Ports visualization also gives an error when imported into Kibana 6.5.4 "Could not locate that index-pattern-field (id: src_ip.keyword)".

Please advise how to fix these issues.

Merge with opnsense-logstash-config

As a OPNsense user https://github.com/fabianfrz/opnsense-logstash-config, works better and I have not tested logstash-pfsense. I wonder if it might make sense to merge. I have reworked https://github.com/fabianfrz/opnsense-logstash-config to use integration testing so logstash-pfsense could be merged into this as well we could have a unified OPNsense/pfSense Logstash config.

Filterlog not begining the string for |prog]

Hello,
I don't why but in my case filterlog wasn't the first string of the line but the last word of the line.

If you have the same issue just modify the following line to :
if [prog] =~ /^filterlog$/ { => if [prog] =~ /filterlog/ {

_grokparsefailure on some lines

Hi
can you help me understand why such lines do not match the grok filter?

7,,,1000000105,igb1,match,block,in,6,0x00,0x00000,1,Options,0,32,fe80::f29f:c2ff:fe1a:2665,ff02::1,HBH,PADN,RTALERT,0x0000,
or
5,,,1000000103,ovpnc1,match,block,in,4,0x0,,64,65242,1480,none,17,udp,173,10.20.30.40,10.20.30.101,

license?

Could you please specify a license for this repo? I'd like to use the grok pattern in another project if the license allows.

Thanks!

Not populating all values for tcp_flags

On multiple instances, I only see tcp_flags values of RA or FA. When the field has A, FPA, or PA (the only ones I can reliably find on search) tcp_flags doesn't populate at all in Elasticsearch. An example search result file (csv) is attached based on the following search:
tags:dcSense AND direction:in AND src_ip:(192.168.0.* OR 192.168.1.* OR 192.168.10.* OR 192.168.11.* OR 192.168.14.* OR 192.168.15.) AND !(src_ip: (fe80 OR 127.0.0.1)) AND !(dest_ip:( 192.168.0.* OR 192.168.1.* OR 192.168.0.255 OR 192.168.1.255 OR 192.168.10.255 OR 192.168.11.255 OR 192.168.14.255 OR 192.168.15.255 OR 255.255.255.255)) AND !(tcp_flags:(RA OR FA))
Outbound-Blocked-Traffic.txt

I am using your pfSense grok and conf.d files (though I've made some minor modifications along the way, including to GeoIP tag outgoing traffic). All of that is working fine, I just am not getting the tcp_flags field to populate on values other than RA or FA.

PFSENSE_LOG_DATA doesn't match vlan interfaces

Hi there,
I had to change the PFSENSE_LOG_DATA pattern so it would match interface names with a vlan such as re0.1. The pattern %{WORD:iface} does not match properly so I changed it to %{DATA:iface} instead

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule})?,,(%{INT:tracker}),(%{DATA:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),

Seems to be working this side :)