syslog-ng / syslog-ng Goto Github PK

I have a pdb file which triggers an action on a given condition:

<action condition='"${appacct.difference_dur}" &gt;= "604800"'>

pdbtool merge transforms this snippet into:

<action condition='&quot;${appacct.difference_dur}&quot; &gt;= &quot;604800&quot;'>

Problem is, the condition now matches all events, even those where ${appacct.difference_dur} < 604800.

I thought I must have just messed up some syntax here, but nothing I do seems to work. The issue is with my attempts to specify filter facility function ranges - here's the error message (from "syslog-ng -s"):

"Error parsing filter expression, syntax error, unexpected LL_DOTDOT, expecting ')' in /etc/syslog-ng.conf at line 69, column 35:

filter f_namain { facility(local0 .. local7) and program(namain); };

..."

Filter function specifications like "facility(kern, authpriv, cron)" and "level(warn .. emerg)" appear to work fine.

There is a strange problem in syslog-ng that the kernel timestamp drifts over time. I now have an uptime of 5d 13h and the kernel log timestamp has drifted by about 11 minutes and gets greater as time passes. If I remove syslog-ng and install rsyslog or busybox-klogd/syslogd then the timestamp is correct.

This only happens with kernel log messages and not other normal messages.

It seems it is possible to modify a message using actions without creating new ones by omitting the message element.
Is this an undocumented feature, or a bug?

#
@version: 3.5
#
source s_file{
    file("/dev/stdin" flags(no-parse)) ;
};
#
parser p_myparser_pdb {
  db_parser(
    file("/tmp/patterndb.xml")
  );
};
#
destination d_file {
   file("/dev/stdout" template("MESSAGE=``${MESSAGE}`` answer=``${answer}``\n") );
};

#
log {
  source(s_file)  ;
  parser(p_myparser_pdb);
  destination(d_file);
};

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2014-06-16'>
  <ruleset name='myname' id='123456789'>
    <rules>
      <rule id='123' class='myclass' provider='bcn'>
        <patterns>
            <pattern>@ESTRING:greeting: @@ANYSTRING:who@</pattern>
        </patterns>
        <actions>
          <action trigger='match'>
            <values>
              <value  name="answer">hi</value>
            </values>
          </action>
        </actions>
      </rule>
    </rules>
  </ruleset>
</patterndb>

This is somehow functional, but doesn't validate the xsd.

⚡ syslog-ng -Fv
syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'
hello mr oizo
MESSAGE=`hello mr oizo` answer=`hi`
this_aint_gonna_match
MESSAGE=`this_aint_gonna_match` answer=``

IMHO this is a useful feature, though maybe not very comprehensive in its "current" form. Maybe this would be more clear:

<action>
  <rewrite>
    ...
  </rewrite>
</action>

syslog-ng OSE admin guide writes about these options are obsolete.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guides/en/syslog-ng-ose-v3.5-guide-admin/html-single/index.html#reference-destination-mongodb

There is also missing a config warning for that options.

The issue:
When syslog-ng starts, the given file destination exists, but this is a directory thus syslog-ng cannot write it when a message incomes.
Than I remove the directory, but syslog-ng never tries to reopen the destination and to write the message again. After reloading syslog-ng, this message is lost.

output:
[2014-06-17T10:40:45.483910] syslog-ng starting up; version='3.6.0alpha0'
[2014-06-17T10:40:46.485061] Follow mode file still does not exist; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:47.486176] Follow mode file still does not exist; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:48.430951] EOF on control channel, closing connection;
[2014-06-17T10:40:48.487127] Follow mode file still does not exist; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:49.488325] Follow-mode file source moved, tracking of the new file is started; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:49.488961] Incoming log entry; line='<38>Feb 25 14:09:07 testhost testapp: test message'
[2014-06-17T10:40:49.535173] Requesting flow control; location='/home/pzolee/zwa/projects/syslo'
[2014-06-17T10:40:49.535294] Initializing destination file writer; template='/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d', filename='/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d'
[2014-06-17T10:40:49.535386] Error opening file for writing; filename='/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d', error='Is a directory (21)'
that's all, no more debug message, syslog-ng never tries to reopen the destination again.

config:
options {
keep_hostname(yes);
time_reopen(1);
threaded(yes);
};
source s_file_d99c49dff6c3455ab7ade6a135c31bd8 {
file("/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt");
};
destination d_file_7e7572d8447143d192718a19c213b28f {
file("/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d");
};
log {
source(s_file_d99c49dff6c3455ab7ade6a135c31bd8);
destination(d_file_7e7572d8447143d192718a19c213b28f);

flags(flow-control);
};

According to 470beca, throttle() should be supported by the MongoDB destination. However, due to an earlier patch (7c484af), we ignore that and print a warning.

We need to revisit whether throttling works now, and if so, remove the warning and the disabling code.

Viktor replies to my original query (below):
"If a log message does not match any pattern for a parser, syslog-ng db-parser sets its .classifier.class to "unknown" regardless of the field's previous state. So if it matched on a previous parser, the next parser will overwrite it if it doesn't match on that. I think it's a bug rather than a feature, so could you please open an issue for that on github?"

I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases):

log {
filter(f_auth);
parser("login");
parser("sshd");
parser("su");
parser("sudo");
log {
filter(f_class_system);
...
};
};

The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db other than the last referenced parser() db. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') does not match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter does match.

When not using syslog protocol it's not possible without some hackish template function to extract the priority and facility and/or their numeric versions from PRI. It would be nice to make that template function available, for instance for people who send json formatted logs, e.g.

{"PRI":42, "PROGRAM":"kernel","MESSAGE":"I/O error"}

That being said, same goes for legacy_msghdr: a template function to extract program and pid would come in handy.

syslog-ng 3.5.4.1 (freebsd)

config:
@Version: 3.5
log {
source { file("/dev/stdin" time_zone('PST8PDT')); };
destination { file("/dev/stdout" time_zone('UTC') template("$DATE $(format-json date=$DATE isodate=$ISODATE)\n")); };
};

$ syslog-ng -vFf ./syslogng-json-date.conf
Jun 24 00:00:01.295 host proc: message
Jun 24 07:00:01 {"isodate":"2014-06-24T00:00:01-07:00","date":"Jun 24 00:00:01"}

I cannot find any information about how to connect to a MongoDB database using SSL. If the MongoDB server has the sslOnNormalPorts configuration enabled, clients are required to connecto to it via SSL. I don't see any option in the mongodb destination to enable such a flag. Does it autodetect it? Or it's not supported at all?

Hello,

I'm trying to update syslog-ng for macports on Macos (10.9.4)
A patch is referenced in old bugzilla https://bugzilla.balabit.com/show_bug.cgi?id=155
so I fit it to current configure.ac in 3.5.5 but I fall on another issue

  CC       tests/loggen/loggen.o
tests/loggen/loggen.c:911:80: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
    (double) sum_count * USEC_PER_SEC / diff_usec, sum_count, sum_time.tv_sec, sum_time.tv_usec / 1000, raw_message_length,
                                                                               ^~~~~~~~~~~~~~~~~~~~~~~
  CC       syslog-ng/main.o
1 warning generated.
  CC       syslog-ng/syslog-ng-ctl.o
  AR       libtest/libsyslog-ng-test.a
syslog-ng/syslog-ng-ctl.c:98:60: warning: comparison of unsigned expression < 0 is always false [-Wtautological-compare]
      if ((len = read(control_socket, buff, BUFF_LEN - 1)) < 0)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~
  CCLD     lib/libsyslog-ng.la
1 warning generated.
  CCLD     tests/loggen/loggen
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
duplicate symbol _last_parser in:
    lib/.libs/lib_libsyslog_ng_la-cfg-grammar.o
    lib/parser/.libs/lib_libsyslog_ng_la-parser-expr-grammar.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:3682: recipe for target 'lib/libsyslog-ng.la' failed

outcome also presented here Homebrew/legacy-homebrew#22030

outside of *.y there are

extern LogParser *last_parser;

in modules/dbparser and modules/json.
var is used but not declared in modules/syslogformat and modules/csvparser so it could be that
it's only present as extern in one header lib/cfg-grammar.h

A possible explanation:
http://samwho.co.uk/blog/2013/12/08/duplicate-symbol-what/

Currently the patterndb parser enforces the use of the following macros:

• $PROGRAM for the ruleset patterns
• $MESSAGE for the rule patterns

The latter is not entirely true, as it seems to be possible according to the source to use template() in a db_parser block. It is however not officially possible, from reading the docs.

Nevertheless I have the strong feeling this is an unnecessary limitation that could be easily lifted without hacking the config (copy whatever is in $custom_message into $MESSAGE and same for *PROGRAM). This is particularily true when using e.g. JSON formatted input.

parser p_new_pdb {
  file("/var/lib/syslog-ng/patterndb/default.xml")
  message_template("$my_message")
  program_template("$my_program")
};

I feel a strong need to separate business logic (actions) from implementation logic (patterns). The reason is simple: patterns can and should be shared with the community. Actions however are very business specific, and thus should not be fused together with patterns, which would make sharing difficult. We need a way to have actions and patterns in different files.

I see two possibilities, both of which are somewhat backwards-compatible:

• Add new XML tag sets, e.g.:

<patterndb ...>
  <actions>
    <action rule_id='ruleid_to_apply_action_to'>
      ...
    </action>
  </actions>
</patterndb>

• Use existing tags but merge multiple rule tags with same id

The first possibility seems saner to me, as it would probably be easier to implement.

value-pairs(key("PRI")) does not do what one would expect, because PRI is not stored in the nvtable, but is a special macro. value-pairs should be clever enough to take those into account too.

In the following config the first server is unreachable, the second is up. syslog-ng does not fallback to the second server.

destination d_mongodb {
mongodb(
servers("localhost:27017", "localhost:28018")
safe_mode(yes)
);
};

console log:
Initializing MongoDB destination; address='localhost', port='27017', database='syslog', collection='messages'
Worker thread started; driver='d_mongodb#0'
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.5.3'
Error connecting to MongoDB;
Syslog connection accepted; fd='13', client='AF_INET(127.0.0.1:59023)', local='AF_INET(127.0.0.1:1112)'
Incoming log entry; line='<38>1 2014-02-27T16:11:21+02:00 localhost prg00000 1234 - - \xef\xbb\xbfseq: 0000000000, thread: 0000, runid: 1393513881, stamp: 2014-02-27T16:11:21 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPAD\x0a'
EOF occurred while reading; fd='13'
Syslog connection closed; fd='13', client='AF_INET(127.0.0.1:59023)', local='AF_INET(127.0.0.1:1112)'
Closing log transport fd; fd='13'
Error connecting to MongoDB;
Network error while inserting into MongoDB; time_reopen='60'

well, it seems that syslog-ng does not detect any files in this mode and/or at least not displaying any incoming message.
To reproduce the issue: create the source file with this content:

root@thor-t410:/home/pzolee/zwa/projects/syslog-ng-ose-3.6/install# cat /var/testdb_working_dir/073358cb-d1ab-415a-a2a3-a874a500b080.txt
Mar 18 12:07:04 testhost localprg PREFIX msg_with_prefix_without_garbage first line
msg_with_prefix_without_garbage second line
msg_with_prefix_without_garbage third line

then start syslog-ng with the config below.
The result:

[2014-06-18T09:43:28.842848] Module loaded and initialized successfully; module='syslogformat'
[2014-06-18T09:43:28.843001] Running application hooks; hook='1'
[2014-06-18T09:43:28.843019] Running application hooks; hook='3'
[2014-06-18T09:43:28.843054] syslog-ng starting up; version='3.6.0alpha0'

Thats all, syslog-ng does not recognize the file.

Expected result: syslog-ng processes the file then creates the proper multi line message when the timeout is reached:
"A message is considered complete if no new lines arrive to the message for 10 seconds, even if no line matching the multi-line-garbage() option is received."

config

@version: 3.6
options {
    threaded(yes);
};
source s_file_97358a54e6924af4826d9ab6ccb074fb {
    file("/var/testdb_working_dir/073358cb-d1ab-415a-a2a3-a874a500b080.txt" keep_hostname(yes)
 multi-line-mode(regexp)
 multi_line_prefix("PREFIX")
);
};
destination d_file_3f633b65908846c085561437c13d2ff8 {
    file("/var/testdb_working_dir/d333cb26-8a64-4bc1-8089-b04c1ace8a20.txt");
};
log {
source(s_file_97358a54e6924af4826d9ab6ccb074fb);
destination(d_file_3f633b65908846c085561437c13d2ff8);

flags(flow-control);
};

And it's default parameter is internal(aka git submodule), which is not fortunate if you are a distro packager. (mostly you want to use the system libmongo-client library).

Attempting to compile syslog-ng v3.4.8 from source on solaris sparc 8 fails with

logsource.c: In function ‘log_source_msg_ack’:
logsource.c:85: error: ‘CLOCK_MONOTONIC’ undeclared (first use in this function)
logsource.c:85: error: (Each undeclared identifier is reported only once
logsource.c:85: error: for each function it appears in.)

The configure.ac script has a libmongo-client >= 0.1.6 minimum requirement for the library, while the code relies on features and symbols added in >= 0.1.8. The minimum version should be bumped.

Since the system() source is implemented in C, it is kind of tiresome to figure out what it expands to on a given system. We should ship a tool that does that for us, and prints the expanded part only.

A shell script can do this just fine.

It will be good if I can use ./configure --enable-all-modules instead of:
./configure --enable-geoip --enable-systemd --enable-redis --enable-smtp --enable-stomp --enable-amqp --enable-json --enable-mongodb --enable-pacct --enable-sql --enable-spoof-source --enable-ipv6 --enable-ssl

Because without this, incubator couldn't compile when compiling syslog-ng from source with internal ivykis.

Suppose we have the following config:

@version: 3.6
source s_net { tcp(port(12345) max-connections(100)); };
destination d_mongo { mongodb(database("sng") collection("stress")); };
log { source(s_net); destination(d_mongo); flags(flow-control); };

If we start stressing this with loggen, like this: loggen -r 10000 -I 100 -i 127.0.0.1 12345 --active-connections=10, it will crash and burn:

ERROR:../lib/logqueue-fifo.c:220:log_queue_fifo_push_tail: assertion failed: (thread_id < 0 || log_queue_max_threads > thread_id)

The reason this happens is because both IO workers and the extra thread used by LogThreadedDestDriver count towards the maximum number of threads allowed. But we don't take the extra thread into consideration when starting worker threads.

I see multiple solutions here:

1. Migrate LogThreadedDestDriver to pthreads or plain ivykis, skipping the entire worker thread infrastructure
2. Whenever creating a thread from LogThreadedDestDriver, increase log_queue_max_threads
3. Create a separate API for these extra threads, and run separate bookkeeping for them, that's not based on thread_id
4. Have a soft and a hard limit on threads: IO workers would be limited to log_queue_max_threads, but other threads would only have the (already existing) 64 thread limit. This is basically a simplified version of the previous option.

@bazsi: I'd love to have some input on this: which option to go for, or if you have any other suggestions on how to proceed.

(Problem spotted by and reported internally by @mitzkia)

Simply put, when merging two pattern databases, the > symbol is converted to &gt in the output file. This is meant to match literally in my log line (from a Cisco ASA log).

  CC       modules/afsocket/modules_afsocket_libafsocket_notls_la-transport-unix-socket.lo
../modules/afsocket/transport-unix-socket.c:145:55: warning: declaration of 'struct ucred' will not be visible outside of this function [-Wvisibility]
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:147:40: error: incomplete definition of type 'struct ucred'
  _add_nv_pair_int(aux, ".unix.pid", uc->pid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:145:55: note: forward declaration of 'struct ucred'
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:148:40: error: incomplete definition of type 'struct ucred'
  _add_nv_pair_int(aux, ".unix.uid", uc->uid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:145:55: note: forward declaration of 'struct ucred'
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:149:40: error: incomplete definition of type 'struct ucred'
  _add_nv_pair_int(aux, ".unix.gid", uc->gid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:145:55: note: forward declaration of 'struct ucred'
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:188:64: error: use of undeclared identifier 'SCM_CREDENTIALS'
      if (cmsg->cmsg_level == SOL_SOCKET && cmsg->cmsg_type == SCM_CREDENTIALS) 
                                                               ^
../modules/afsocket/transport-unix-socket.c:192:40: error: incomplete definition of type 'struct ucred'
          _feed_aux_from_procfs(aux, uc->pid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:190:18: note: forward declaration of 'struct ucred'
          struct ucred *uc = (struct ucred *) CMSG_DATA(cmsg);
                 ^
../modules/afsocket/transport-unix-socket.c:193:37: warning: incompatible pointer types passing 'struct ucred *' to parameter of type 'struct ucred *'
      [-Wincompatible-pointer-types]
          _feed_aux_from_ucred(aux, uc);
                                    ^~
../modules/afsocket/transport-unix-socket.c:145:62: note: passing argument to parameter 'uc' here
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                             ^
../modules/afsocket/transport-unix-socket.c:248:30: error: use of undeclared identifier 'SO_PASSCRED'
  setsockopt(fd, SOL_SOCKET, SO_PASSCRED, &one, sizeof(one));
                             ^
2 warnings and 6 errors generated.

In short, ucred is a linux-ism, FreeBSD has cmsgcred with similar contents.

The OSE 3.5 documentation claims in Figure 8.3. Managing log messages in syslog-ng: "The log_fetch_limit() parameter can be set as a global option, or for every source individually.".

The syslog-ng daemon (3.5.3) doesn't agree:

Using a global log-fetch-limit() option was removed, please use a per-source log-fetch-limit();
Using a global log-iw-size() option was removed, please use a per-source log-iw-size();

As the title says: the zone file parser does not support tzif3 format files, which is produced by zic >= 2013. This causes test case failures on at least Fedora.

The updated manpage says:

For version-3-format time zone files, the POSIX-TZ-style string may use
two minor extensions to the POSIX TZ format, as described in
newtzset(3).  First, the hours part of its transition times may be
signed and range from -167 through 167 instead of the POSIX-required
unsigned values from 0 through 24.  Second, DST is in effect all year
if it starts January 1 at 00:00 and ends December 31 at 24:00 plus the
difference between daylight saving and standard time.

Originally reported by Peter Czanik [email protected]

There are multiple issues when trying to compile the grammar with bison 3:

• The configure script flags 3.0 as too old (I have an easy fix for this)
• YYID appears to be gone
• bison throws a ton of warnings, we previously grepped through (we need -Wno-other in AM_YFLAGS now)

For the moment, I reverted my Debian unstable and Ubuntu Trusty build environments to bison 2.7, but that won't do in the long run.

A recent discussion here in the syslog-ng room at work, the topic of ancient defaults came up, and we were wondering if we could tweak a few defaults for 3.6, to cater for the expectations of a wide number of installations:

• Up flush-lines() from 1 to maybe 100. This has a noticable performance impact in a good way, at the expense of possibly losing more messages. I would rather have a bit more performance by default, instead of trying very hard to not loose a single message and crawling (compared to others, with default configs).
• Switch to threaded(yes) by default.

There may be a couple of other options that may worth revisiting the default for, this two came up this morning.

We should make debian packages from the https://github.com/balabit/syslog-ng-patterndb
repo, to be able to distribute it easily. We should also make a link to this repo from the syslog-ng.org site.

Further plans: We should also consolidate them: some files has .pdb extensions, some
of them has .xml. And they are separated in different files, so it is hard to load them,
or manage them.

On FreeBSD, when in jail, /dev/klog does not exist, and should not be included in system(). We should detect if we're running in a jail, and not include that source in that case.

To detect the jail, sysctl security.jail.jailed returns 1 if in jail, 0 if not. Need to figure out how to do that from C, though.

relevant output for make check:
./test-driver: line 107: 8752 Segmentation fault (core dumped) "$@" > $log_file 2>&1
FAIL: modules/basicfuncs/tests/test_basicfuncs

bt for test_basicfuncs's core:
Program received signal SIGSEGV, Segmentation fault.
0xb7f49de0 in g_static_mutex_get_mutex_impl@plt ()
from /home/micek/syslog-ng-collector/checkout-dir/syslog-ng-ose-3.6/lib/.libs/libsyslog-ng-3.6.0alpha0.so
(gdb) bt
#0 0xb7f49de0 in g_static_mutex_get_mutex_impl@plt ()

from /home/micek/syslog-ng-collector/checkout-dir/syslog-ng-ose-3.6/lib/.libs/libsyslog-ng-3.6.0alpha0.so
#1 0xb7f8a695 in log_template_append_format_with_context (self=self@entry=0x805db70, messages=messages@entry=0xbffff0f4,

num_messages=num_messages@entry=1, opts=0x80567ec, opts@entry=0x0, tz=tz@entry=0, seq_num=seq_num@entry=999, 
context_id=context_id@entry=0x804dabb "test-context-id", result=result@entry=0x806e2d0) at lib/template/templates.c:1405

#2 0xb7f8ab23 in log_template_append_format (self=self@entry=0x805db70, lm=lm@entry=0x805a690, opts=opts@entry=0x0,

tz=tz@entry=0, seq_num=seq_num@entry=999, context_id=context_id@entry=0x804dabb "test-context-id", 
result=result@entry=0x806e2d0) at lib/template/templates.c:1430

#3 0xb7f8ab8c in log_template_format (self=self@entry=0x805db70, lm=lm@entry=0x805a690, opts=opts@entry=0x0, tz=tz@entry=0,

seq_num=seq_num@entry=999, context_id=context_id@entry=0x804dabb "test-context-id", result=result@entry=0x806e2d0)
at lib/template/templates.c:1437

#4 0x0804b1ea in assert_template_format_with_escaping (template=template@entry=0x804cc59 "$(substr $HOST 1 3)",

escaping=escaping@entry=0, expected=expected@entry=0x804cc55 "zor") at libtest/template_lib.c:109

#5 0x0804b26f in assert_template_format (template=template@entry=0x804cc59 "$(substr $HOST 1 3)",

expected=expected@entry=0x804cc55 "zor") at libtest/template_lib.c:94

#6 0x0804abb7 in test_str_funcs () at modules/basicfuncs/tests/test_basicfuncs.c:50
#7 0x0804a80a in main (argc=1, argv=0xbffff254) at modules/basicfuncs/tests/test_basicfuncs.c:113

last commit:
commit a7df8c7
Merge: 4581c3d 83265ad
Author: Gergely Nagy [email protected]
Date: Thu May 22 15:02:21 2014 +0200

Merge pull request #128 from algernon/h/value-pairs/warning-fixes

value-pairs: Silence a few compiler warnings

uname:
Linux micek-HP-Mini-110-4100 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:31:42 UTC 2014 i686 i686 i686 GNU/Linux

lsb_release:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty

In afsocket, there's modules/afsocket/transport-unix-socket.c and modules/afsocket/unix-credentials.*. Together, they allow syslog-ng to retrieve the UNIX credentials passed over on a UNIX socket. So far, this has been ported to and tested on Linux, ported but not tested on FreeBSD.

But it may or may not compile, or work, on other platforms. Testing and porting that needs to be done before the 3.6.1 release.

If you are using "base" or "selected_macros", the date will contain the value of R_DATE macro instead of the S_DATE.
If you are using "everything", the DATE is fine.

config:
source s_file_57d097b7282846beb3b14d2d8d4f08a6 {
file("/var/testdb_working_dir/9509b21c-e6f9-42f8-b2a8-8cc34d26e46a.txt");
};
destination d_file_8bf0d1a88a6a43f38965186bd67f1584 {
file("/var/testdb_working_dir/97773739-62b8-46c6-9a9e-4e57924e059e.txt"
template("$(format_json --scope selected_macros)"
));
};
log {
source(s_file_57d097b7282846beb3b14d2d8d4f08a6);
destination(d_file_8bf0d1a88a6a43f38965186bd67f1584);

flags(flow-control);
};

Consider the following config:

#
@version: 3.5
#
source s_file{
    file("/dev/stdin" flags(no-parse)) ;
};
#
parser p_myparser_pdb {
  db_parser(
    file("/tmp/patterndb.xml")
  );
};
#
destination d_file {
   file("/dev/stdout" template("${answer} syslog-ng\n") );
};

#
log {
  source(s_file)  ;
  parser(p_myparser_pdb);
  destination(d_file);
};

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2014-06-16'>
  <ruleset name='myname' id='123456789'>
    <rules>
      <rule id='123' class='myclass' provider='bcn'>
        <patterns>
            <pattern>@ESTRING:greeting: @@ANYSTRING:who@</pattern>
        </patterns>
        <actions>
          <action condition='"${greeting}" eq "hello"'>
            <message>
              <values>
                <value  name="answer">hi</value>
              </values>
            </message>
          </action>
          <action condition='"${greeting}" eq "goodbye"'>
            <message>
              <values>
                <value  name="answer">see you later</value>
              </values>
            </message>
          </action>
        </actions>
      </rule>
    </rules>
  </ruleset>
</patterndb>

This is not working as expected, as both conditions always apply:

⚡ syslog-ng -Fv
syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'
hello bob
hi syslog-ng
see you later syslog-ng
 syslog-ng
goodbye sam
hi syslog-ng
see you later syslog-ng
 syslog-ng
hasta_la_vista dude
hi syslog-ng
see you later syslog-ng
 syslog-ng

I tried to use the following template function, but syslog-ng didn't accept it:

template("$(format-json  --scope rfc5424  --exclude FILE_NAME --pair .SDATA=$(format-json --scope sdata))\n")

The error message was:

Error parsing affile, Error compiling template, error=$ in /home/balabit/install/syslog-ng-3.5/etc/syslog-ng.conf at line 50, column 18:

        template("$(format-json  --scope rfc5424  --exclude FILE_NAME .SDATA=$(format-json --scope sdata))\n")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng

I used syslog-ng OSE 3.5.

Using the following config syslog-ng does not warning for the database name longer than 64 char.

destination d_mongodb {
mongodb(
servers("localhost:28018")
database("11111111111111111111111111111111111111111111111111111111111111111")
safe_mode(yes)
);
};

syslog-ng console-log:
Initializing MongoDB destination; address='localhost', port='28018', database='11111111111111111111111111111111111111111111111111111111111111111', collection='messages'
Worker thread started; driver='d_mongodb#0'
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.5.3'

console log after sending message to syslog-ng:
Syslog connection accepted; fd='14', client='AF_INET(127.0.0.1:58979)', local='AF_INET(127.0.0.1:1112)'
Incoming log entry; line='<38>1 2014-02-27T16:00:43+02:00 localhost prg00000 1234 - - \xef\xbb\xbfseq: 0000000000, thread: 0000, runid: 1393513243, stamp: 2014-02-27T16:00:43 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPAD\x0a'
EOF occurred while reading; fd='14'
Syslog connection closed; fd='14', client='AF_INET(127.0.0.1:58979)', local='AF_INET(127.0.0.1:1112)'
Closing log transport fd; fd='14'

MongoDB limits and thresholds: http://docs.mongodb.org/manual/reference/limits/

syslog-ng version:
syslog-ng 3.5.3
Installer-Version: 3.5.3

Could you please backport the following patch for the next 3.5 maintenace release?
75a4fd0

Thank you in advance!

destination config here:

amqp(
vhost("/")
host("rabbitmq-0.internal.example.com")
port(5672)
exchange("raw_logs")
exchange_type("fanout")
routing_key("")
body("")
persistent(yes)
value-pairs(
scope("selected-macros" "nv-pairs" "sdata")
)
);

Backtrace here:

Core was generated by `syslog-ng -dev -F --enable-core'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f1b1a65099f in ?? () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) backtrace
#0 0x00007f1b1a65099f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f1b171b4bd2 in sasl_response (args=0x7f1b1b27da78, method=AMQP_SASL_METHOD_PLAIN,

pool=0x7f1b10002040) at ../../../../../modules/afamqp/rabbitmq-c/librabbitmq/amqp_socket.c:149

#2 amqp_login_inner (vl=0x7f1b1b27da78, sasl_method=AMQP_SASL_METHOD_PLAIN, heartbeat=0, frame_max=131072,

channel_max=0, state=0x7f1b10002000)
at ../../../../../modules/afamqp/rabbitmq-c/librabbitmq/amqp_socket.c:432

#3 amqp_login (state=0x7f1b10002000, vhost=0x152d9b0 "/", channel_max=, frame_max=131072,

heartbeat=0, sasl_method=AMQP_SASL_METHOD_PLAIN)
at ../../../../../modules/afamqp/rabbitmq-c/librabbitmq/amqp_socket.c:517

#4 0x00007f1b171acb72 in afamqp_dd_connect (self=0x15235a0, reconnect=)

at ../../modules/afamqp/afamqp.c:334

#5 0x00007f1b1ae18be8 in log_threaded_dest_driver_worker_thread_main (arg=0x15235a0)

at ../../lib/logthrdestdrv.c:57

#6 0x00007f1b1ae1e22a in worker_thread_func (st=0x1514e20) at ../../lib/misc.c:586
#7 0x00007f1b1ab5c9b5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#8 0x00007f1b1a8dde9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007f1b1a60a3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

When reloading the configuration, there are quite a few spots where we're leaking memory. This hurts not only when stress testing with lots of reloads, but when you leave a syslog-ng running with a big and complex configuration for months, with daily reloads, it shows.

A few spots I identified:

• template_content_inner calls log_template_compile() with a fresh GError, but that variable is never freed as far as I see (and valgrind confirms)
• I'm seeing LogTemplate leaks from the same area, but I didn't go after it yet.
• source_afinet_tcp_params appears to leak the AFInetSourceDriver instance.
• A static mutex appears to be leaked from LogThreadedDestDrivers log_threaded_dest_driver_worker_thread_main() via log_queue_check_items()
• A static mutex also appears to be leaked via main_loop_call()

There are possibly other leaks, many drivers do not unref templates and other settings in their setter functions.

To reproduce these, I'm running syslog-ng -F with a custom, but simple config under valgrind, and on another terminal:

i=1; while true; do printf "%03d: " $i; sbin/syslog-ng-ctl reload; i=$(expr $i + 1); done

After a short while, I stop the reloads and syslog-ng, and check the valgrind log. Look for leaks where the "N bytes in X blocks" part has an X close to the number of reloads made - those are the leaks that happen during reload.

libtool: link: gcc -std=gnu99 -shared -o .libs/libsyslog-ng-3.4.8.so .libs/libsyslog_ng_la-afinter.o .libs/libsyslog_ng_la-alarms.o .libs/libsyslog_ng_la-apphook.o .libs/libsyslog_ng_la-block-ref-parser.o .libs/libsyslog_ng_la-cfg.o .libs/libsyslog_ng_la-cfg-args.o .libs/libsyslog_ng_la-cfg-lexer.o .libs/libsyslog_ng_la-cfg-lexer-subst.o .libs/libsyslog_ng_la-cfg-parser.o .libs/libsyslog_ng_la-cfg-tree.o .libs/libsyslog_ng_la-children.o .libs/libsyslog_ng_la-compat.o .libs/libsyslog_ng_la-control.o .libs/libsyslog_ng_la-dnscache.o .libs/libsyslog_ng_la-driver.o .libs/libsyslog_ng_la-file-perms.o .libs/libsyslog_ng_la-filter.o .libs/libsyslog_ng_la-filter-expr-parser.o .libs/libsyslog_ng_la-globals.o .libs/libsyslog_ng_la-gprocess.o .libs/libsyslog_ng_la-gsockaddr.o .libs/libsyslog_ng_la-gsocket.o .libs/libsyslog_ng_la-logmatcher.o .libs/libsyslog_ng_la-logmpx.o .libs/libsyslog_ng_la-logmsg.o .libs/libsyslog_ng_la-logparser.o .libs/libsyslog_ng_la-logpipe.o .libs/libsyslog_ng_la-logproto-client.o .libs/libsyslog_ng_la-logproto-server.o .libs/libsyslog_ng_la-logproto-buffered-server.o .libs/libsyslog_ng_la-logproto-dgram-server.o .libs/libsyslog_ng_la-logproto-framed-client.o .libs/libsyslog_ng_la-logproto-framed-server.o .libs/libsyslog_ng_la-logproto-text-client.o .libs/libsyslog_ng_la-logproto-text-server.o .libs/libsyslog_ng_la-logproto-record-server.o .libs/libsyslog_ng_la-logproto-builtins.o .libs/libsyslog_ng_la-logqueue.o .libs/libsyslog_ng_la-logqueue-fifo.o .libs/libsyslog_ng_la-logreader.o .libs/libsyslog_ng_la-logrewrite.o .libs/libsyslog_ng_la-logsource.o .libs/libsyslog_ng_la-logstamp.o .libs/libsyslog_ng_la-logtransport.o .libs/libsyslog_ng_la-logwriter.o .libs/libsyslog_ng_la-mainloop.o .libs/libsyslog_ng_la-memtrace.o .libs/libsyslog_ng_la-messages.o .libs/libsyslog_ng_la-misc.o .libs/libsyslog_ng_la-ml-batched-timer.o .libs/libsyslog_ng_la-msg-format.o .libs/libsyslog_ng_la-nvtable.o .libs/libsyslog_ng_la-parser-expr-parser.o .libs/libsyslog_ng_la-persist-state.o .libs/libsyslog_ng_la-plugin.o .libs/libsyslog_ng_la-pragma-parser.o .libs/libsyslog_ng_la-rewrite-expr-parser.o .libs/libsyslog_ng_la-scratch-buffers.o .libs/libsyslog_ng_la-serialize.o .libs/libsyslog_ng_la-stats.o .libs/libsyslog_ng_la-str-format.o .libs/libsyslog_ng_la-syslog-names.o .libs/libsyslog_ng_la-tags.o .libs/libsyslog_ng_la-templates.o .libs/libsyslog_ng_la-timeutils.o .libs/libsyslog_ng_la-utils.o .libs/libsyslog_ng_la-value-pairs.o .libs/libsyslog_ng_la-vptransform.o .libs/libsyslog_ng_la-cfg-lex.o .libs/libsyslog_ng_la-cfg-grammar.o .libs/libsyslog_ng_la-filter-expr-grammar.o .libs/libsyslog_ng_la-block-ref-grammar.o .libs/libsyslog_ng_la-pragma-grammar.o .libs/libsyslog_ng_la-parser-expr-grammar.o .libs/libsyslog_ng_la-rewrite-expr-grammar.o -Wl,-blibpath:/usr/local/lib:/opt/freeware/lib:/opt/soe/local/syslog-ng-3.4.8/eventlog/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix5.3.0.0/4.8.2:/opt/freeware/lib/gcc/powerpc-ibm-aix5.3.0.0/4.8.2/../../..:/usr/lib:/lib -lrt -lnsl -L/usr/local/lib -lgmodule-2.0 -lgthread-2.0 -lglib-2.0 /usr/local/lib/libintl.a -L/opt/freeware/lib -liconv -L/opt/soe/local/syslog-ng-3.4.8/eventlog/lib -levtlog -ldl -L../lib/ivykis/src /home/syslog-ng-3.4.8/lib/ivykis/src/.libs/libivykis.a -lc -Wl,-bnoentry -O2 -pthread -Wl,-brtl -pthread -Wl,--whole-archive -Wl,--no-whole-archive -pthread -Wl,-bE:.libs/libsyslog-ng.exp -Wl,-bernotok
ld: 0706-012 The -- flag is not recognized.
ld: 0706-012 The -w flag is not recognized.
ld: 0706-012 The -h flag is not recognized.
ld: 0706-012 The -- flag is not recognized.
ld: 0706-027 The -n flag is ignored.
collect2: error: ld returned 255 exit status
Makefile:789: recipe for target 'libsyslog-ng.la' failed
gmake[4]: *** [libsyslog-ng.la] Error 1
gmake[4]: Leaving directory '/home/syslog-ng-3.4.8/lib'
Makefile:1517: recipe for target 'all-recursive' failed
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory '/home/syslog-ng-3.4.8/lib'
Makefile:679: recipe for target 'all' failed
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory '/home/syslog-ng-3.4.8/lib'
Makefile:514: recipe for target 'all-recursive' failed
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory '/home/syslog-ng-3.4.8'
Makefile:418: recipe for target 'all' failed
gmake: *** [all] Error 2

It would be tremendously useful to be able to specify string delimiters for csv-parsers

delimiters(":\n")

Obviously this would break multiple delimiters, not sure how to make this backwards compatible.

In my understanding, pdbtool should merge the versions of individual files with the following logic:

• fail when there is a version mismatch (or use highest, if backwards compatible)
• use the provided version if all files have the same version

See: https://github.com/balabit/syslog-ng/blob/master/modules/dbparser/pdbtool/pdbtool.c#L314-L315

the stored counter of statistics should only increase when the message is stored in the queue (so syslog-ng could not send the message).
Now in case of mongodb, it is increased every time when a message is sent

Assuming I have this configuration:

@version: 3.5
source s_foo { tcp(port(12345)); };
log { source(s_foo); destination { file("/dev/stdout", template("SRC: $SOURCE\n")); }; };

If I send a log message, $SOURCE will not be set, and stats for the center counter will not be incremented, either. This is because in afsocket_sd_init_instance(), we explicitly set the queue method to NULL, which will then fall back to log_pipe_forward_msg(), which does not set SOURCE, nor does it increment the center counter.

My first idea would be to remove that line, and have log_src_driver_queue_method() do its thing. But I am unsure about the implications.

@bazsi: Any insight?

(Thanks @faxm0dem for the report!)

As requested by @algernon

==14100== 4,088,118 bytes in 534,893 blocks are definitely lost in loss record 3,711 of 3,711
==14100==    at 0x4C279EE: malloc (vg_replace_malloc.c:270)
==14100==    by 0x5B83DC4: g_malloc (in /lib64/libglib-2.0.so.0.2600.1)
==14100==    by 0x5B9B6CD: g_strdup (in /lib64/libglib-2.0.so.0.2600.1)
==14100==    by 0x50B4486: vp_walker_name_split (value-pairs.c:666)
==14100==    by 0x50B451A: value_pairs_walker (value-pairs.c:684)
==14100==    by 0x50B3801: vp_foreach_helper (value-pairs.c:333)
==14100==    by 0x5BA8385: g_tree_foreach (in /lib64/libglib-2.0.so.0.2600.1)
==14100==    by 0x50B3B1F: value_pairs_foreach_sorted (value-pairs.c:391)
==14100==    by 0x50B46B2: value_pairs_walk (value-pairs.c:730)
==14100==    by 0x967C6D4: tf_json_append (format-json.c:256)
==14100==    by 0x967C763: tf_json_call (format-json.c:273)
==14100==    by 0x50D981F: log_template_append_format_with_context (templates.c:1386)
==14100== 
==14100== LEAK SUMMARY:
==14100==    definitely lost: 4,264,972 bytes in 562,898 blocks
==14100==    indirectly lost: 582,079 bytes in 19,459 blocks
==14100==      possibly lost: 103 bytes in 13 blocks
==14100==    still reachable: 237,300 bytes in 3,667 blocks
==14100==         suppressed: 0 bytes in 0 blocks
==14100== 
==14100== ERROR SUMMARY: 1282 errors from 26 contexts (suppressed: 6 from 6)
==14100== 
==14100== 58 errors in context 1 of 26:
==14100== Source and destination overlap in memcpy(0xc684854, 0xc68485d, 13)
==14100==    at 0x4C29B8E: memcpy (mc_replace_strmem.c:882)
==14100==    by 0x50A78F3: nv_table_add_value (nvtable.c:431)
==14100==    by 0x50A7730: nv_table_make_direct (nvtable.c:369)
==14100==    by 0x50A7FF1: nv_table_foreach_entry (nvtable.c:618)
==14100==    by 0x50A780B: nv_table_add_value (nvtable.c:405)
==14100==    by 0x50969D2: log_msg_set_value (logmsg.c:507)
==14100==    by 0x765E385: pdb_message_apply (patterndb.c:306)
==14100==    by 0x765EE84: pdb_rule_run_actions (patterndb.c:602)
==14100==    by 0x7661094: pattern_db_expire_entry (patterndb.c:1454)
==14100==    by 0x7662444: timer_wheel_set_time (timerwheel.c:320)
==14100==    by 0x766127E: pattern_db_set_time (patterndb.c:1514)
==14100==    by 0x76619A1: pattern_db_process (patterndb.c:1673)
==14100== 
==14100== 
==14100== 1200 errors in context 2 of 26:
==14100== Source and destination overlap in memcpy(0x7fefffe00, 0x7fefffe00, 10)
==14100==    at 0x4C29B8E: memcpy (mc_replace_strmem.c:882)
==14100==    by 0x50A56BA: resolve_sockaddr (misc.c:270)
==14100==    by 0x509CC89: log_source_mangle_hostname (logsource.c:130)
==14100==    by 0x509D0A7: log_source_queue (logsource.c:216)
==14100==    by 0x509AF4C: log_pipe_queue (logpipe.h:320)
==14100==    by 0x509B885: log_reader_handle_line (logreader.c:311)
==14100==    by 0x509B9B4: log_reader_fetch_log (logreader.c:369)
==14100==    by 0x509B174: log_reader_work_perform (logreader.c:83)
==14100==    by 0x509B466: log_reader_io_process_input (logreader.c:188)
==14100==    by 0x50ABE6F: poll_events_invoke_callback (poll-events.c:29)
==14100==    by 0x667D876: ??? (in /lib64/libivykis.so.0.3.5)
==14100==    by 0x667E683: iv_main (in /lib64/libivykis.so.0.3.5)
==14100== 
--14100-- 
--14100-- used_suppression:      4 U1004-ARM-_dl_relocate_object
--14100-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==14100== 
==14100== ERROR SUMMARY: 1282 errors from 26 contexts (suppressed: 6 from 6)

Though fixed in the SQL destination, a lot of other drivers will restart sequence numbering on reload. This should be corrected. (See 6edb634 for the SQL fix, which should be ported to other drivers too.)

Perhaps this should be done in LogThreadedDestDriver?

Right now we detect whether to enable linux capability support by checking for sys/capabilities.h only. We also check for -lcap, but if we can't find it, we just ignore it. We should disable linux capabilities if we can't find libcap.

This affects every branch of syslog-ng.

It's not my problem actually, but reported by multiple users in GetSatisfaction:

http://communities.balabit.com/balabit/topics/syslog_ng_version_3_4_host_closes_tcp_syslog_connection_with_message_incoming_frame_larger_than_log_msg_size

Downgrading to 3.2 resolved the issue for one of the reporters, but that's an EoL version...