ELSA not preserving original IP about enterprise-log-search-and-archive

[GoogleCodeExporter]

You need to make sure the spoof-source support is compiled into syslog-ng 
(check syslog-ng -V) and make sure that spoof_source(yes) is set on the 
destination from the forwarding server.  Also, are you sure you're running with 
the syslog-ng.conf in the nodes/conf directory?  You should copy that to 
/usr/local/syslog-ng/etc/syslog-ng.conf and make edits there so that code 
updates do not overwrite your local edits.

Original comment by [email protected] on 22 Mar 2012 at 1:02

[GoogleCodeExporter]

I'm going to close this for now, but I'll reopen if your issue is not fixed.

Original comment by [email protected] on 24 Mar 2012 at 3:29

• Changed state: Done

[GoogleCodeExporter]

Here is my 'syslog-ng -V' output:

Enable-Threads: off
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-Sun-STREAMS: off
Enable-IPv6: on
Enable-Spoof-Source: on   <-- this means I have spoof-source support
Enable-TCP-Wrapper: off
Enable-SSL: on
Enable-SQL: off
Enable-Linux-Caps: on
Enable-Pcre: on
Enable-Pacct: off

I have determined my syslog-ng install is using the conf file @ 
/usr/local/syslog-ng/etc/ . However, adding options for spoof_source(yes); 
causes a syntax error. Can you point me in another direction?

Thank you.

Original comment by [email protected] on 26 Mar 2012 at 10:49

[GoogleCodeExporter]

Can you attach the config or a snippet as well as the error?

Original comment by [email protected] on 27 Mar 2012 at 1:26

• Changed state: New

[GoogleCodeExporter]

It was my understanding of syslog, that I needed to set keep_hostname(yes) and 
chain_hostname(no). Do you know how they are different? (chain_hostname causes 
the same error)

syslog-ng.conf file:

@version: 3.2

options {
        spoof_source(yes);
};

source s_network {
        tcp();
        udp();
};

parser p_db {
        db-parser(file("/usr/local/elsa/node/conf/patterndb.xml"));
};

filter f_rewrite_cisco_program { match('^(%[A-Z]+\-\d\-[0-9A-Z]+): ([^\n]+)' 
value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); };
filter f_rewrite_cisco_program_2 { 
match('^[\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d
{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?: [A-Z]{3})?: (%[^:]+): ([^\n]+)' 
value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; 
filter f_rewrite_cisco_program_3 { match('^\d+[ywdh]\d+[ywdh]: (%[^:]+): 
([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); };


Error:

Stopping syslog-ng

Starting syslog-ng
Error parsing main, syntax error, unexpected LL_IDENTIFIER, expecting '}' in 
/usr/local/syslog-ng-3.2.4/etc/syslog-ng.conf at line 4, column 2:

    spoof_source(yes);
    ^^^^^^^^^^^^

syslog-ng documentation: 
http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng

Original comment by [email protected] on 27 Mar 2012 at 2:34

[GoogleCodeExporter]

My syslog-ng server is recieving syslog forwards from another syslog-ng server. 
The trouble is, my syslog-ng server keeps the IP of the other syslog-ng server 
instead of the original syslog messages sender. However, I can see the correct 
IP when I do a TCPdump of the network interface of my syslog-ng server.

Original comment by [email protected] on 27 Mar 2012 at 2:56

[GoogleCodeExporter]

Ah, you need spoof_source(yes) as an option in your destination, like this:

destination d_other_syslog {
        udp("10.0.2.24" port(514) spoof_source(yes));
};

Original comment by [email protected] on 27 Mar 2012 at 3:17

[GoogleCodeExporter]

I am not sending anything from my syslog-ng server, just recieving... I would 
think if I did that it wouldn't make a difference because I am sending it to 
ELSA.

Am I thinking about this correctly?


Thanks!

Original comment by [email protected] on 27 Mar 2012 at 6:45

[GoogleCodeExporter]

You don't need spoof_source at all on the syslog-ng instance that writes to 
ELSA.  ELSA's template uses $SOURCEIP which is the actual IP address it sees on 
the wire when it receives a log.  If you have logs chained through hosts, then 
the forwarding hosts need to spoof_source.

Original comment by [email protected] on 27 Mar 2012 at 6:52

[GoogleCodeExporter]

I just want to mention that we have a similar scenario; in our case we have 
hosts behind NAT, which send logs directly to the ELSA syslog-ng instance. All 
of these hosts end up with the $SOURCEIP of the NAT router.

Prior to using ELSA we had been using syslog_ng to write to flat files with the 
keep_hostnames option, which worked as expected.

I guess this is a feature request, but it would be nice to have some way for 
ELSA to do something with keep_hostnames, even if it means using name 
resolution to turn those hostnames back into IP addresses internally. 
spoof_source seems like it won't work in many network situations, and it will 
definitely not cover our particular situation (spoofed packets from the NAT 
router we own would be discarded by the real routers we don't control between 
it and our ELSA server).

(Yes, I know our network situation is bad, but it's outside of my control at 
the moment, and I'm sure there are others suffering similar fates)

Original comment by [email protected] on 30 Mar 2012 at 5:35

[GoogleCodeExporter]

This should be no problem as it's all just syslog-ng config.  You can use $HOST 
instead of $SOURCEIP in the syslog-ng template and turn on keep_hostnames(yes). 
 ELSA uses the inet_aton() function on the host given (which will be $HOST and 
not $SOURCEIP if you change it) which does DNS resolving on its own.  
Performance of this is untested, so I can't say if the DNS will be cached in 
any way or if that will significantly degrade throughput.

Original comment by [email protected] on 30 Mar 2012 at 6:14

[GoogleCodeExporter]

That sounds like a great solution for us - thanks for the prompt reply.

Original comment by [email protected] on 30 Mar 2012 at 6:56

[GoogleCodeExporter]

It looks like the correct IP address comes across in the Message section of the 
syslog packet (see below). Couldn't I have Syslog-ng look for that rewrite 
$SCOURCEIP to the IP I found?

sample tcpdump:

5:43:35.545651 IP (tos 0x0, ttl 64, id 15401, offset 0, flags [DF], proto UDP 
(17), length 176)
    ip-xxx-xxx-xxx-xxx.xxx.xxx.xxxx > ncc1701d.xx.xxx.xxx.syslog: [udp sum ok] SYSLOG, length: 148
    Facility local7 (23), Severity error (3)
    Msg: Apr  9 15:43:35 144.38.100.189/144.38.100.189 592905: Apr  9 15:43:34 MDT: %LINK-3-UPDOWN: Interface FastEthernet2/0/46, changed state to down\0x0a

Thanks.

Original comment by [email protected] on 9 Apr 2012 at 9:51

[GoogleCodeExporter]

Or... is there a quick way to increase the size of the MSGONLY field to include 
more than "%LINK-3-UPDOWN: Interface FastEthernet2/0/46, changed state to down" 
like the entire Msg: ?


Thanks!

Original comment by [email protected] on 9 Apr 2012 at 10:02

[GoogleCodeExporter]

The $MSG macro should be what you're looking for.  I think some sort of 
rewrite() function could work here, but it will take some work.

Original comment by [email protected] on 9 Apr 2012 at 10:13

[GoogleCodeExporter]

Original comment by [email protected] on 30 Apr 2012 at 1:54

• Changed state: Done