syslog-ng / syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, message queues, databases (SQL and NoSQL alike), and more.

Quickstart

The simplest configuration accepts system logs from /dev/log (from applications or forwarded by systemd) and writes everything to a single file:

@version: 4.7
@include "scl.conf"

log {
	source { system(); };
	destination { file("/var/log/syslog"); };
};

This one additionally processes logs from the network (TCP/514 by default):

@version: 4.7
@include "scl.conf"

log {
	source {
		system();
		network();
	};
	destination { file("/var/log/syslog"); };
};

This config is designed for structured/application logging, using local submission via JSON, and outputting in key=value format:

@version: 4.7
@include "scl.conf"

log {
	source { system(); };
	destination { file("/var/log/app.log" template("$(format-welf --subkeys .cim.)\n")); };
};

To submit a structured log using logger, you might run:

$ logger '@cim: {"name1":"value1", "name2":"value2"}'

In which case the resulting message will be:

name1=value1 name2=value2

For a brief introduction to configuring the syslog-ng application, see the quickstart guide.

Features

• Receive and send RFC3164 and RFC5424 style syslog messages
• Receive and send JSON formatted messages
• Work with any kind of unstructured data
• Classify and structure logs using built-in parsers (csv-parser(), db-parser(), kv-parser(), etc.)
• Normalize, crunch, and process logs as they flow through the system
• Hand over logs for further processing using files, message queues (like AMQP), or databases (like PostgreSQL or MongoDB)
• Forward logs to big data tools (like Elasticsearch, Apache Kafka, or Apache Hadoop)

Performance

• syslog-ng provides performance levels comparable to a large cluster when running on a single node
• In the simplest use case, it scales up to 600-800k messages per second
• But classification, parsing, and filtering still produce several tens of thousands of messages per second

Community

• syslog-ng is developed by a community of volunteers, the best way to contact us is via our github project page project, our gitter channel or our mailing list.
• syslog-ng is integrated into almost all Linux distributions and BSDs, it is also incorporated into a number of products, see our powered by syslog-ng page for more details.

Sponsors

• Axoflow is the company of Balazs Scheidler, the original creator and main developer of syslog-ng, and the creators of the Kubernetes Logging Operator. Currently Axoflow is the most active contributor of syslog-ng, and offers commercial support, professional services, and related products.
• Balabit is the original commercial sponsor of the syslog-ng project, and was acquired by One Identity in 2018. One Identity offers a commercial edition for syslog-ng, called the syslog-ng Premium Edition.

Feedback

We are really interested to see who uses our software, so if you do use it and you like what you see, please tell us about it. A star on github or an email saying thanks means a lot already, but telling us about your use case, your experience, and things to improve would be much appreciated.

Just send an email to feedback (at) syslog-ng.org.

Feedback Powers Open Source.

Installation from source

Releases and precompiled tarballs are available on GitHub.

To compile from source, the easiest is to use dbld, a docker based, self-hosted compile/build/release infrastructure within the source tree. See dbld/README.md for more information.

For the brave souls who want to compile syslog-ng from scratch, the usual drill applies:

$ ./configure && make && make install

The extra effort in contrast with the dbld based build is the need to fetch and install all build dependencies of syslog-ng (of which there are a few).

If you don't have a configure script (because of cloning from git, for example), run ./autogen.sh to generate it.

Some of the functionality of syslog-ng is compiled only if the required development libraries are present. The configure script displays a summary of enabled features at the end of its run. For details, see the syslog-ng compiling instructions.

Installation from binaries

Binaries are available in various Linux distributions and contributors maintain packages of the latest and greatest syslog-ng version for various OSes.

Debian/Ubuntu

Simply invoke the following command as root:

# apt install syslog-ng

The latest versions of syslog-ng are available for a wide range of Debian and Ubuntu releases from our APT repository.

The packages and the APT repository are provided "as is" without warranty of any kind, on a best-effort level.

Supported distributions

syslog-ng packages are released for the following distribution versions (x86-64):

Distro version	sources.list component name
Ubuntu 24.04	ubuntu-noble
Ubuntu 23.10	ubuntu-mantic
Ubuntu 23.04	ubuntu-lunar
Ubuntu 22.04	ubuntu-jammy
Ubuntu 20.04	ubuntu-focal
Debian 12	debian-bookworm
Debian 11	debian-bullseye
Debian Unstable	debian-sid
Debian Testing	debian-testing

Adding the APT repository

1. Download and install the release signing key:

   wget -qO - https://ose-repo.syslog-ng.com/apt/syslog-ng-ose-pub.asc | sudo apt-key add -
   

2. Add the repository containing the latest build of syslog-ng to the APT sources. For example, stable releases on Ubuntu 22.04:

   echo "deb https://ose-repo.syslog-ng.com/apt/ stable ubuntu-noble" | sudo tee -a /etc/apt/sources.list.d/syslog-ng-ose.list
   

3. Run apt update

Nightly builds

Nightly packages are built and released from the git master branch everyday.

Use nightly instead of stable in step 2 to use the nightly APT repository. E.g.:

echo "deb https://ose-repo.syslog-ng.com/apt/ nightly ubuntu-noble" | sudo tee -a /etc/apt/sources.list.d/syslog-ng-ose.list

Nightly builds can be used for testing purposes (obtaining new features and bugfixes) at the risk of breakage.

Arch Linux

# pacman -S syslog-ng

Fedora

syslog-ng is available as a Fedora package that you can install using dnf:

dnf install syslog-ng

You can download packages for the latest versions from here.

For instructions on how to install syslog-ng on RPM distributions, see the blog post Installing latest syslog-ng on RHEL and other RPM distributions.

If you wish to install the latest RPM package that comes from a recent commit in Git for testing purposes, read the blog post, RPM packages from syslog-ng Git HEAD.

macOS

# brew install syslog-ng

Others

Binaries for other platforms are listed on the official third party page.

Installation from Docker image

Binaries are also available as a Docker image. To find out more, check out the blog post, Your central log server in Docker.

There are alternatives to the upstream provided, bare syslog-ng image, such as the AxoSyslog image for running syslog-ng in Kubernetes.

• AxoSyslog, a cloud native distribution for syslog-ng announcement
• AxoSyslog log collection for Kubernetes
• AxoSyslog Documentation
• GitHub

Documentation

For the latest, markdown based version, see the syslog-ng documentation center. The official documentation of the earlier versions of syslog-ng Open Source Edition provided by One Identity is available here.

An alternative, markdown based, improved, community maintained version of the documentation is available as AxoSyslog Core documentation. Source code

Contributing

If you would like to contribute to syslog-ng, to fix a bug or create a new module, the syslog-ng pages helps you take the first steps to working with the code base.

syslog-ng's Issues

libmongo-client minimum version is too small

The configure.ac script has a libmongo-client >= 0.1.6 minimum requirement for the library, while the code relies on features and symbols added in >= 0.1.8. The minimum version should be bumped.

mongodb: Support for throttle()?

According to 470beca, throttle() should be supported by the MongoDB destination. However, due to an earlier patch (7c484af), we ignore that and print a warning.

We need to revisit whether throttling works now, and if so, remove the warning and the disabling code.

Add configure option to enable all modules

It will be good if I can use ./configure --enable-all-modules instead of:
./configure --enable-geoip --enable-systemd --enable-redis --enable-smtp --enable-stomp --enable-amqp --enable-json --enable-mongodb --enable-pacct --enable-sql --enable-spoof-source --enable-ipv6 --enable-ssl

patterndb parser: undocumented feature

It seems it is possible to modify a message using actions without creating new ones by omitting the message element.
Is this an undocumented feature, or a bug?

#
@version: 3.5
#
source s_file{
    file("/dev/stdin" flags(no-parse)) ;
};
#
parser p_myparser_pdb {
  db_parser(
    file("/tmp/patterndb.xml")
  );
};
#
destination d_file {
   file("/dev/stdout" template("MESSAGE=``${MESSAGE}`` answer=``${answer}``\n") );
};

#
log {
  source(s_file)  ;
  parser(p_myparser_pdb);
  destination(d_file);
};

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2014-06-16'>
  <ruleset name='myname' id='123456789'>
    <rules>
      <rule id='123' class='myclass' provider='bcn'>
        <patterns>
            <pattern>@ESTRING:greeting: @@ANYSTRING:who@</pattern>
        </patterns>
        <actions>
          <action trigger='match'>
            <values>
              <value  name="answer">hi</value>
            </values>
          </action>
        </actions>
      </rule>
    </rules>
  </ruleset>
</patterndb>

This is somehow functional, but doesn't validate the xsd.

⚡ syslog-ng -Fv
syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'
hello mr oizo
MESSAGE=`hello mr oizo` answer=`hi`
this_aint_gonna_match
MESSAGE=`this_aint_gonna_match` answer=``

IMHO this is a useful feature, though maybe not very comprehensive in its "current" form. Maybe this would be more clear:

<action>
  <rewrite>
    ...
  </rewrite>
</action>

Threaded destination vs Worker threads

Suppose we have the following config:

@version: 3.6
source s_net { tcp(port(12345) max-connections(100)); };
destination d_mongo { mongodb(database("sng") collection("stress")); };
log { source(s_net); destination(d_mongo); flags(flow-control); };

If we start stressing this with loggen, like this: loggen -r 10000 -I 100 -i 127.0.0.1 12345 --active-connections=10, it will crash and burn:

ERROR:../lib/logqueue-fifo.c:220:log_queue_fifo_push_tail: assertion failed: (thread_id < 0 || log_queue_max_threads > thread_id)

The reason this happens is because both IO workers and the extra thread used by LogThreadedDestDriver count towards the maximum number of threads allowed. But we don't take the extra thread into consideration when starting worker threads.

I see multiple solutions here:

1. Migrate LogThreadedDestDriver to pthreads or plain ivykis, skipping the entire worker thread infrastructure
2. Whenever creating a thread from LogThreadedDestDriver, increase log_queue_max_threads
3. Create a separate API for these extra threads, and run separate bookkeeping for them, that's not based on thread_id
4. Have a soft and a hard limit on threads: IO workers would be limited to log_queue_max_threads, but other threads would only have the (already existing) 64 thread limit. This is basically a simplified version of the previous option.

@bazsi: I'd love to have some input on this: which option to go for, or if you have any other suggestions on how to proceed.

(Problem spotted by and reported internally by @mitzkia)

Syslog-ng make install should install ivykis headers when using internal ivykis.

Because without this, incubator couldn't compile when compiling syslog-ng from source with internal ivykis.

Enhancement: csv-parser multichar delimiter

It would be tremendously useful to be able to specify string delimiters for csv-parsers

delimiters(":\n")

Obviously this would break multiple delimiters, not sure how to make this backwards compatible.

format_json: wrong date when base/selected_macros is used

If you are using "base" or "selected_macros", the date will contain the value of R_DATE macro instead of the S_DATE.
If you are using "everything", the DATE is fine.

config:
source s_file_57d097b7282846beb3b14d2d8d4f08a6 {
file("/var/testdb_working_dir/9509b21c-e6f9-42f8-b2a8-8cc34d26e46a.txt");
};
destination d_file_8bf0d1a88a6a43f38965186bd67f1584 {
file("/var/testdb_working_dir/97773739-62b8-46c6-9a9e-4e57924e059e.txt"
template("$(format_json --scope selected_macros)"
));
};
log {
source(s_file_57d097b7282846beb3b14d2d8d4f08a6);
destination(d_file_8bf0d1a88a6a43f38965186bd67f1584);

flags(flow-control);
};

afsocket: unix credentials on legacy unix platforms

In afsocket, there's modules/afsocket/transport-unix-socket.c and modules/afsocket/unix-credentials.*. Together, they allow syslog-ng to retrieve the UNIX credentials passed over on a UNIX socket. So far, this has been ported to and tested on Linux, ported but not tested on FreeBSD.

But it may or may not compile, or work, on other platforms. Testing and porting that needs to be done before the 3.6.1 release.

Backport log_macro_expand bugfix for 3.5

Could you please backport the following patch for the next 3.5 maintenace release?
75a4fd0

Thank you in advance!

afsocket source does not set SOURCE

Assuming I have this configuration:

@version: 3.5
source s_foo { tcp(port(12345)); };
log { source(s_foo); destination { file("/dev/stdout", template("SRC: $SOURCE\n")); }; };

If I send a log message, $SOURCE will not be set, and stats for the center counter will not be incremented, either. This is because in afsocket_sd_init_instance(), we explicitly set the queue method to NULL, which will then fall back to log_pipe_forward_msg(), which does not set SOURCE, nor does it increment the center counter.

My first idea would be to remove that line, and have log_src_driver_queue_method() do its thing. But I am unsure about the implications.

@bazsi: Any insight?

(Thanks @faxm0dem for the report!)

implement multi-line-timeout

well, it seems that syslog-ng does not detect any files in this mode and/or at least not displaying any incoming message.
To reproduce the issue: create the source file with this content:

root@thor-t410:/home/pzolee/zwa/projects/syslog-ng-ose-3.6/install# cat /var/testdb_working_dir/073358cb-d1ab-415a-a2a3-a874a500b080.txt
Mar 18 12:07:04 testhost localprg PREFIX msg_with_prefix_without_garbage first line
msg_with_prefix_without_garbage second line
msg_with_prefix_without_garbage third line

then start syslog-ng with the config below.
The result:

[2014-06-18T09:43:28.842848] Module loaded and initialized successfully; module='syslogformat'
[2014-06-18T09:43:28.843001] Running application hooks; hook='1'
[2014-06-18T09:43:28.843019] Running application hooks; hook='3'
[2014-06-18T09:43:28.843054] syslog-ng starting up; version='3.6.0alpha0'

Thats all, syslog-ng does not recognize the file.

Expected result: syslog-ng processes the file then creates the proper multi line message when the timeout is reached:
"A message is considered complete if no new lines arrive to the message for 10 seconds, even if no line matching the multi-line-garbage() option is received."

config

@version: 3.6
options {
    threaded(yes);
};
source s_file_97358a54e6924af4826d9ab6ccb074fb {
    file("/var/testdb_working_dir/073358cb-d1ab-415a-a2a3-a874a500b080.txt" keep_hostname(yes)
 multi-line-mode(regexp)
 multi_line_prefix("PREFIX")
);
};
destination d_file_3f633b65908846c085561437c13d2ff8 {
    file("/var/testdb_working_dir/d333cb26-8a64-4bc1-8089-b04c1ace8a20.txt");
};
log {
source(s_file_97358a54e6924af4826d9ab6ccb074fb);
destination(d_file_3f633b65908846c085561437c13d2ff8);

flags(flow-control);
};

Embedded template functions don't work

I tried to use the following template function, but syslog-ng didn't accept it:

template("$(format-json  --scope rfc5424  --exclude FILE_NAME --pair .SDATA=$(format-json --scope sdata))\n")

The error message was:

Error parsing affile, Error compiling template, error=$ in /home/balabit/install/syslog-ng-3.5/etc/syslog-ng.conf at line 50, column 18:

        template("$(format-json  --scope rfc5424  --exclude FILE_NAME .SDATA=$(format-json --scope sdata))\n")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng

I used syslog-ng OSE 3.5.

file_destination is not reopened after error

The issue:
When syslog-ng starts, the given file destination exists, but this is a directory thus syslog-ng cannot write it when a message incomes.
Than I remove the directory, but syslog-ng never tries to reopen the destination and to write the message again. After reloading syslog-ng, this message is lost.

output:
[2014-06-17T10:40:45.483910] syslog-ng starting up; version='3.6.0alpha0'
[2014-06-17T10:40:46.485061] Follow mode file still does not exist; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:47.486176] Follow mode file still does not exist; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:48.430951] EOF on control channel, closing connection;
[2014-06-17T10:40:48.487127] Follow mode file still does not exist; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:49.488325] Follow-mode file source moved, tracking of the new file is started; filename='/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt'
[2014-06-17T10:40:49.488961] Incoming log entry; line='<38>Feb 25 14:09:07 testhost testapp: test message'
[2014-06-17T10:40:49.535173] Requesting flow control; location='/home/pzolee/zwa/projects/syslo'
[2014-06-17T10:40:49.535294] Initializing destination file writer; template='/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d', filename='/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d'
[2014-06-17T10:40:49.535386] Error opening file for writing; filename='/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d', error='Is a directory (21)'
that's all, no more debug message, syslog-ng never tries to reopen the destination again.

config:
options {
keep_hostname(yes);
time_reopen(1);
threaded(yes);
};
source s_file_d99c49dff6c3455ab7ade6a135c31bd8 {
file("/var/testdb_working_dir/245c9ba0-578f-4b56-a0aa-4453b4905c0c.txt");
};
destination d_file_7e7572d8447143d192718a19c213b28f {
file("/var/testdb_working_dir/f0812566-be02-4931-968a-6ed2bf5efe1d");
};
log {
source(s_file_d99c49dff6c3455ab7ade6a135c31bd8);
destination(d_file_7e7572d8447143d192718a19c213b28f);

flags(flow-control);
};

system(): FreeBSD jail support

On FreeBSD, when in jail, /dev/klog does not exist, and should not be included in system(). We should detect if we're running in a jail, and not include that source in that case.

To detect the jail, sysctl security.jail.jailed returns 1 if in jail, 0 if not. Need to figure out how to do that from C, though.

The zone file parser does not support Tzif3 files

As the title says: the zone file parser does not support tzif3 format files, which is produced by zic >= 2013. This causes test case failures on at least Fedora.

The updated manpage says:

For version-3-format time zone files, the POSIX-TZ-style string may use
two minor extensions to the POSIX TZ format, as described in
newtzset(3).  First, the hours part of its transition times may be
signed and range from -167 through 167 instead of the POSIX-required
unsigned values from 0 through 24.  Second, DST is in effect all year
if it starts January 1 at 00:00 and ends December 31 at 24:00 plus the
difference between daylight saving and standard time.

Originally reported by Peter Czanik [email protected]

patterndb parser: PROGRAM and MSG should be configurable

Currently the patterndb parser enforces the use of the following macros:

• $PROGRAM for the ruleset patterns
• $MESSAGE for the rule patterns

The latter is not entirely true, as it seems to be possible according to the source to use template() in a db_parser block. It is however not officially possible, from reading the docs.

Nevertheless I have the strong feeling this is an unnecessary limitation that could be easily lifted without hacking the config (copy whatever is in $custom_message into $MESSAGE and same for *PROGRAM). This is particularily true when using e.g. JSON formatted input.

parser p_new_pdb {
  file("/var/lib/syslog-ng/patterndb/default.xml")
  message_template("$my_message")
  program_template("$my_program")
};

Time to update some defaults?

A recent discussion here in the syslog-ng room at work, the topic of ancient defaults came up, and we were wondering if we could tweak a few defaults for 3.6, to cater for the expectations of a wide number of installations:

• Up flush-lines() from 1 to maybe 100. This has a noticable performance impact in a good way, at the expense of possibly losing more messages. I would rather have a bit more performance by default, instead of trying very hard to not loose a single message and crawling (compared to others, with default configs).
• Switch to threaded(yes) by default.

There may be a couple of other options that may worth revisiting the default for, this two came up this morning.

Memory leaks during reload

When reloading the configuration, there are quite a few spots where we're leaking memory. This hurts not only when stress testing with lots of reloads, but when you leave a syslog-ng running with a big and complex configuration for months, with daily reloads, it shows.

A few spots I identified:

• template_content_inner calls log_template_compile() with a fresh GError, but that variable is never freed as far as I see (and valgrind confirms)
• I'm seeing LogTemplate leaks from the same area, but I didn't go after it yet.
• source_afinet_tcp_params appears to leak the AFInetSourceDriver instance.
• A static mutex appears to be leaked from LogThreadedDestDrivers log_threaded_dest_driver_worker_thread_main() via log_queue_check_items()
• A static mutex also appears to be leaked via main_loop_call()

There are possibly other leaks, many drivers do not unref templates and other settings in their setter functions.

To reproduce these, I'm running syslog-ng -F with a custom, but simple config under valgrind, and on another terminal:

i=1; while true; do printf "%03d: " $i; sbin/syslog-ng-ctl reload; i=$(expr $i + 1); done

After a short while, I stop the reloads and syslog-ng, and check the valgrind log. Look for leaks where the "N bytes in X blocks" part has an X close to the number of reloads made - those are the leaks that happen during reload.

Facility Filter Ranges Don't Work

I thought I must have just messed up some syntax here, but nothing I do seems to work. The issue is with my attempts to specify filter facility function ranges - here's the error message (from "syslog-ng -s"):

"Error parsing filter expression, syntax error, unexpected LL_DOTDOT, expecting ')' in /etc/syslog-ng.conf at line 69, column 35:

filter f_namain { facility(local0 .. local7) and program(namain); };

..."

Filter function specifications like "facility(kern, authpriv, cron)" and "level(warn .. emerg)" appear to work fine.

Enhancement: PRI template function

When not using syslog protocol it's not possible without some hackish template function to extract the priority and facility and/or their numeric versions from PRI. It would be nice to make that template function available, for instance for people who send json formatted logs, e.g.

{"PRI":42, "PROGRAM":"kernel","MESSAGE":"I/O error"}

That being said, same goes for legacy_msghdr: a template function to extract program and pid would come in handy.

Sequence numbering restarts on reload

Though fixed in the SQL destination, a lot of other drivers will restart sequence numbering on reload. This should be corrected. (See 6edb634 for the SQL fix, which should be ported to other drivers too.)

Perhaps this should be done in LogThreadedDestDriver?

amqp module causes syslog-ng (3.4 and 3.5) to segfault

destination config here:

amqp(
vhost("/")
host("rabbitmq-0.internal.example.com")
port(5672)
exchange("raw_logs")
exchange_type("fanout")
routing_key("")
body("")
persistent(yes)
value-pairs(
scope("selected-macros" "nv-pairs" "sdata")
)
);

Backtrace here:

Core was generated by `syslog-ng -dev -F --enable-core'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f1b1a65099f in ?? () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) backtrace
#0 0x00007f1b1a65099f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f1b171b4bd2 in sasl_response (args=0x7f1b1b27da78, method=AMQP_SASL_METHOD_PLAIN,

pool=0x7f1b10002040) at ../../../../../modules/afamqp/rabbitmq-c/librabbitmq/amqp_socket.c:149

#2 amqp_login_inner (vl=0x7f1b1b27da78, sasl_method=AMQP_SASL_METHOD_PLAIN, heartbeat=0, frame_max=131072,

channel_max=0, state=0x7f1b10002000)
at ../../../../../modules/afamqp/rabbitmq-c/librabbitmq/amqp_socket.c:432

#3 amqp_login (state=0x7f1b10002000, vhost=0x152d9b0 "/", channel_max=, frame_max=131072,

heartbeat=0, sasl_method=AMQP_SASL_METHOD_PLAIN)
at ../../../../../modules/afamqp/rabbitmq-c/librabbitmq/amqp_socket.c:517

#4 0x00007f1b171acb72 in afamqp_dd_connect (self=0x15235a0, reconnect=)

at ../../modules/afamqp/afamqp.c:334

#5 0x00007f1b1ae18be8 in log_threaded_dest_driver_worker_thread_main (arg=0x15235a0)

at ../../lib/logthrdestdrv.c:57

#6 0x00007f1b1ae1e22a in worker_thread_func (st=0x1514e20) at ../../lib/misc.c:586
#7 0x00007f1b1ab5c9b5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#8 0x00007f1b1a8dde9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007f1b1a60a3fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

latest on Macos: compile issues

Hello,

I'm trying to update syslog-ng for macports on Macos (10.9.4)
A patch is referenced in old bugzilla https://bugzilla.balabit.com/show_bug.cgi?id=155
so I fit it to current configure.ac in 3.5.5 but I fall on another issue

  CC       tests/loggen/loggen.o
tests/loggen/loggen.c:911:80: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
    (double) sum_count * USEC_PER_SEC / diff_usec, sum_count, sum_time.tv_sec, sum_time.tv_usec / 1000, raw_message_length,
                                                                               ^~~~~~~~~~~~~~~~~~~~~~~
  CC       syslog-ng/main.o
1 warning generated.
  CC       syslog-ng/syslog-ng-ctl.o
  AR       libtest/libsyslog-ng-test.a
syslog-ng/syslog-ng-ctl.c:98:60: warning: comparison of unsigned expression < 0 is always false [-Wtautological-compare]
      if ((len = read(control_socket, buff, BUFF_LEN - 1)) < 0)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~
  CCLD     lib/libsyslog-ng.la
1 warning generated.
  CCLD     tests/loggen/loggen
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
duplicate symbol _last_parser in:
    lib/.libs/lib_libsyslog_ng_la-cfg-grammar.o
    lib/parser/.libs/lib_libsyslog_ng_la-parser-expr-grammar.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:3682: recipe for target 'lib/libsyslog-ng.la' failed

outcome also presented here Homebrew/legacy-homebrew#22030

outside of *.y there are

extern LogParser *last_parser;

in modules/dbparser and modules/json.
var is used but not declared in modules/syslogformat and modules/csvparser so it could be that
it's only present as extern in one header lib/cfg-grammar.h

A possible explanation:
http://samwho.co.uk/blog/2013/12/08/duplicate-symbol-what/

Misdetecting linux capabilities on FreeBSD

Right now we detect whether to enable linux capability support by checking for sys/capabilities.h only. We also check for -lcap, but if we can't find it, we just ignore it. We should disable linux capabilities if we can't find libcap.

This affects every branch of syslog-ng.

syslog-ng v3.4.8 issue on solaris sparc 8

Attempting to compile syslog-ng v3.4.8 from source on solaris sparc 8 fails with

logsource.c: In function ‘log_source_msg_ack’:
logsource.c:85: error: ‘CLOCK_MONOTONIC’ undeclared (first use in this function)
logsource.c:85: error: (Each undeclared identifier is reported only once
logsource.c:85: error: for each function it appears in.)

Because of mongodb restriction database names longer than 64 char. not created

Using the following config syslog-ng does not warning for the database name longer than 64 char.

destination d_mongodb {
mongodb(
servers("localhost:28018")
database("11111111111111111111111111111111111111111111111111111111111111111")
safe_mode(yes)
);
};

syslog-ng console-log:
Initializing MongoDB destination; address='localhost', port='28018', database='11111111111111111111111111111111111111111111111111111111111111111', collection='messages'
Worker thread started; driver='d_mongodb#0'
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.5.3'

console log after sending message to syslog-ng:
Syslog connection accepted; fd='14', client='AF_INET(127.0.0.1:58979)', local='AF_INET(127.0.0.1:1112)'
Incoming log entry; line='<38>1 2014-02-27T16:00:43+02:00 localhost prg00000 1234 - - \xef\xbb\xbfseq: 0000000000, thread: 0000, runid: 1393513243, stamp: 2014-02-27T16:00:43 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPAD\x0a'
EOF occurred while reading; fd='14'
Syslog connection closed; fd='14', client='AF_INET(127.0.0.1:58979)', local='AF_INET(127.0.0.1:1112)'
Closing log transport fd; fd='14'

MongoDB limits and thresholds: http://docs.mongodb.org/manual/reference/limits/

syslog-ng version:
syslog-ng 3.5.3
Installer-Version: 3.5.3

Kernel log message time drift

There is a strange problem in syslog-ng that the kernel timestamp drifts over time. I now have an uptime of 5d 13h and the kernel log timestamp has drifted by about 11 minutes and gets greater as time passes. If I remove syslog-ng and install rsyslog or busybox-klogd/syslogd then the timestamp is correct.

This only happens with kernel log messages and not other normal messages.

pdbtool merge generates XML with version 3

In my understanding, pdbtool should merge the versions of individual files with the following logic:

• fail when there is a version mismatch (or use highest, if backwards compatible)
• use the provided version if all files have the same version

See: https://github.com/balabit/syslog-ng/blob/master/modules/dbparser/pdbtool/pdbtool.c#L314-L315

statistics: stored should mean queued

the stored counter of statistics should only increase when the message is stored in the queue (so syslog-ng could not send the message).
Now in case of mongodb, it is increased every time when a message is sent

Incoming frame larger than log_msg_size()

It's not my problem actually, but reported by multiple users in GetSatisfaction:

http://communities.balabit.com/balabit/topics/syslog_ng_version_3_4_host_closes_tcp_syslog_connection_with_message_incoming_frame_larger_than_log_msg_size

Downgrading to 3.2 resolved the issue for one of the reporters, but that's an EoL version...

Support mongodb connection over SSL

I cannot find any information about how to connect to a MongoDB database using SSL. If the MongoDB server has the sslOnNormalPorts configuration enabled, clients are required to connecto to it via SSL. I don't see any option in the mongodb destination to enable such a flag. Does it autodetect it? Or it's not supported at all?

value-pairs key() vs special macros

value-pairs(key("PRI")) does not do what one would expect, because PRI is not stored in the nvtable, but is a special macro. value-pairs should be clever enough to take those into account too.

memory leak in 3.5.5 vp/json

As requested by @algernon

==14100== 4,088,118 bytes in 534,893 blocks are definitely lost in loss record 3,711 of 3,711
==14100==    at 0x4C279EE: malloc (vg_replace_malloc.c:270)
==14100==    by 0x5B83DC4: g_malloc (in /lib64/libglib-2.0.so.0.2600.1)
==14100==    by 0x5B9B6CD: g_strdup (in /lib64/libglib-2.0.so.0.2600.1)
==14100==    by 0x50B4486: vp_walker_name_split (value-pairs.c:666)
==14100==    by 0x50B451A: value_pairs_walker (value-pairs.c:684)
==14100==    by 0x50B3801: vp_foreach_helper (value-pairs.c:333)
==14100==    by 0x5BA8385: g_tree_foreach (in /lib64/libglib-2.0.so.0.2600.1)
==14100==    by 0x50B3B1F: value_pairs_foreach_sorted (value-pairs.c:391)
==14100==    by 0x50B46B2: value_pairs_walk (value-pairs.c:730)
==14100==    by 0x967C6D4: tf_json_append (format-json.c:256)
==14100==    by 0x967C763: tf_json_call (format-json.c:273)
==14100==    by 0x50D981F: log_template_append_format_with_context (templates.c:1386)
==14100== 
==14100== LEAK SUMMARY:
==14100==    definitely lost: 4,264,972 bytes in 562,898 blocks
==14100==    indirectly lost: 582,079 bytes in 19,459 blocks
==14100==      possibly lost: 103 bytes in 13 blocks
==14100==    still reachable: 237,300 bytes in 3,667 blocks
==14100==         suppressed: 0 bytes in 0 blocks
==14100== 
==14100== ERROR SUMMARY: 1282 errors from 26 contexts (suppressed: 6 from 6)
==14100== 
==14100== 58 errors in context 1 of 26:
==14100== Source and destination overlap in memcpy(0xc684854, 0xc68485d, 13)
==14100==    at 0x4C29B8E: memcpy (mc_replace_strmem.c:882)
==14100==    by 0x50A78F3: nv_table_add_value (nvtable.c:431)
==14100==    by 0x50A7730: nv_table_make_direct (nvtable.c:369)
==14100==    by 0x50A7FF1: nv_table_foreach_entry (nvtable.c:618)
==14100==    by 0x50A780B: nv_table_add_value (nvtable.c:405)
==14100==    by 0x50969D2: log_msg_set_value (logmsg.c:507)
==14100==    by 0x765E385: pdb_message_apply (patterndb.c:306)
==14100==    by 0x765EE84: pdb_rule_run_actions (patterndb.c:602)
==14100==    by 0x7661094: pattern_db_expire_entry (patterndb.c:1454)
==14100==    by 0x7662444: timer_wheel_set_time (timerwheel.c:320)
==14100==    by 0x766127E: pattern_db_set_time (patterndb.c:1514)
==14100==    by 0x76619A1: pattern_db_process (patterndb.c:1673)
==14100== 
==14100== 
==14100== 1200 errors in context 2 of 26:
==14100== Source and destination overlap in memcpy(0x7fefffe00, 0x7fefffe00, 10)
==14100==    at 0x4C29B8E: memcpy (mc_replace_strmem.c:882)
==14100==    by 0x50A56BA: resolve_sockaddr (misc.c:270)
==14100==    by 0x509CC89: log_source_mangle_hostname (logsource.c:130)
==14100==    by 0x509D0A7: log_source_queue (logsource.c:216)
==14100==    by 0x509AF4C: log_pipe_queue (logpipe.h:320)
==14100==    by 0x509B885: log_reader_handle_line (logreader.c:311)
==14100==    by 0x509B9B4: log_reader_fetch_log (logreader.c:369)
==14100==    by 0x509B174: log_reader_work_perform (logreader.c:83)
==14100==    by 0x509B466: log_reader_io_process_input (logreader.c:188)
==14100==    by 0x50ABE6F: poll_events_invoke_callback (poll-events.c:29)
==14100==    by 0x667D876: ??? (in /lib64/libivykis.so.0.3.5)
==14100==    by 0x667E683: iv_main (in /lib64/libivykis.so.0.3.5)
==14100== 
--14100-- 
--14100-- used_suppression:      4 U1004-ARM-_dl_relocate_object
--14100-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==14100== 
==14100== ERROR SUMMARY: 1282 errors from 26 contexts (suppressed: 6 from 6)

Enhancement: separate actions from rules in patterndb

I feel a strong need to separate business logic (actions) from implementation logic (patterns). The reason is simple: patterns can and should be shared with the community. Actions however are very business specific, and thus should not be fused together with patterns, which would make sharing difficult. We need a way to have actions and patterns in different files.

I see two possibilities, both of which are somewhat backwards-compatible:

• Add new XML tag sets, e.g.:

<patterndb ...>
  <actions>
    <action rule_id='ruleid_to_apply_action_to'>
      ...
    </action>
  </actions>
</patterndb>

• Use existing tags but merge multiple rule tags with same id

The first possibility seems saner to me, as it would probably be easier to implement.

patterndb parser: bug in condition

Consider the following config:

#
@version: 3.5
#
source s_file{
    file("/dev/stdin" flags(no-parse)) ;
};
#
parser p_myparser_pdb {
  db_parser(
    file("/tmp/patterndb.xml")
  );
};
#
destination d_file {
   file("/dev/stdout" template("${answer} syslog-ng\n") );
};

#
log {
  source(s_file)  ;
  parser(p_myparser_pdb);
  destination(d_file);
};

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2014-06-16'>
  <ruleset name='myname' id='123456789'>
    <rules>
      <rule id='123' class='myclass' provider='bcn'>
        <patterns>
            <pattern>@ESTRING:greeting: @@ANYSTRING:who@</pattern>
        </patterns>
        <actions>
          <action condition='"${greeting}" eq "hello"'>
            <message>
              <values>
                <value  name="answer">hi</value>
              </values>
            </message>
          </action>
          <action condition='"${greeting}" eq "goodbye"'>
            <message>
              <values>
                <value  name="answer">see you later</value>
              </values>
            </message>
          </action>
        </actions>
      </rule>
    </rules>
  </ruleset>
</patterndb>

This is not working as expected, as both conditions always apply:

⚡ syslog-ng -Fv
syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'
hello bob
hi syslog-ng
see you later syslog-ng
 syslog-ng
goodbye sam
hi syslog-ng
see you later syslog-ng
 syslog-ng
hasta_la_vista dude
hi syslog-ng
see you later syslog-ng
 syslog-ng

system(): A tool to expand system() only

Since the system() source is implemented in C, it is kind of tiresome to figure out what it expands to on a given system. We should ship a tool that does that for us, and prints the expanded part only.

A shell script can do this just fine.

Urgent: syslog-ng v3.4.8 issue on AIX 5.3

libtool: link: gcc -std=gnu99 -shared -o .libs/libsyslog-ng-3.4.8.so .libs/libsyslog_ng_la-afinter.o .libs/libsyslog_ng_la-alarms.o .libs/libsyslog_ng_la-apphook.o .libs/libsyslog_ng_la-block-ref-parser.o .libs/libsyslog_ng_la-cfg.o .libs/libsyslog_ng_la-cfg-args.o .libs/libsyslog_ng_la-cfg-lexer.o .libs/libsyslog_ng_la-cfg-lexer-subst.o .libs/libsyslog_ng_la-cfg-parser.o .libs/libsyslog_ng_la-cfg-tree.o .libs/libsyslog_ng_la-children.o .libs/libsyslog_ng_la-compat.o .libs/libsyslog_ng_la-control.o .libs/libsyslog_ng_la-dnscache.o .libs/libsyslog_ng_la-driver.o .libs/libsyslog_ng_la-file-perms.o .libs/libsyslog_ng_la-filter.o .libs/libsyslog_ng_la-filter-expr-parser.o .libs/libsyslog_ng_la-globals.o .libs/libsyslog_ng_la-gprocess.o .libs/libsyslog_ng_la-gsockaddr.o .libs/libsyslog_ng_la-gsocket.o .libs/libsyslog_ng_la-logmatcher.o .libs/libsyslog_ng_la-logmpx.o .libs/libsyslog_ng_la-logmsg.o .libs/libsyslog_ng_la-logparser.o .libs/libsyslog_ng_la-logpipe.o .libs/libsyslog_ng_la-logproto-client.o .libs/libsyslog_ng_la-logproto-server.o .libs/libsyslog_ng_la-logproto-buffered-server.o .libs/libsyslog_ng_la-logproto-dgram-server.o .libs/libsyslog_ng_la-logproto-framed-client.o .libs/libsyslog_ng_la-logproto-framed-server.o .libs/libsyslog_ng_la-logproto-text-client.o .libs/libsyslog_ng_la-logproto-text-server.o .libs/libsyslog_ng_la-logproto-record-server.o .libs/libsyslog_ng_la-logproto-builtins.o .libs/libsyslog_ng_la-logqueue.o .libs/libsyslog_ng_la-logqueue-fifo.o .libs/libsyslog_ng_la-logreader.o .libs/libsyslog_ng_la-logrewrite.o .libs/libsyslog_ng_la-logsource.o .libs/libsyslog_ng_la-logstamp.o .libs/libsyslog_ng_la-logtransport.o .libs/libsyslog_ng_la-logwriter.o .libs/libsyslog_ng_la-mainloop.o .libs/libsyslog_ng_la-memtrace.o .libs/libsyslog_ng_la-messages.o .libs/libsyslog_ng_la-misc.o .libs/libsyslog_ng_la-ml-batched-timer.o .libs/libsyslog_ng_la-msg-format.o .libs/libsyslog_ng_la-nvtable.o .libs/libsyslog_ng_la-parser-expr-parser.o .libs/libsyslog_ng_la-persist-state.o .libs/libsyslog_ng_la-plugin.o .libs/libsyslog_ng_la-pragma-parser.o .libs/libsyslog_ng_la-rewrite-expr-parser.o .libs/libsyslog_ng_la-scratch-buffers.o .libs/libsyslog_ng_la-serialize.o .libs/libsyslog_ng_la-stats.o .libs/libsyslog_ng_la-str-format.o .libs/libsyslog_ng_la-syslog-names.o .libs/libsyslog_ng_la-tags.o .libs/libsyslog_ng_la-templates.o .libs/libsyslog_ng_la-timeutils.o .libs/libsyslog_ng_la-utils.o .libs/libsyslog_ng_la-value-pairs.o .libs/libsyslog_ng_la-vptransform.o .libs/libsyslog_ng_la-cfg-lex.o .libs/libsyslog_ng_la-cfg-grammar.o .libs/libsyslog_ng_la-filter-expr-grammar.o .libs/libsyslog_ng_la-block-ref-grammar.o .libs/libsyslog_ng_la-pragma-grammar.o .libs/libsyslog_ng_la-parser-expr-grammar.o .libs/libsyslog_ng_la-rewrite-expr-grammar.o -Wl,-blibpath:/usr/local/lib:/opt/freeware/lib:/opt/soe/local/syslog-ng-3.4.8/eventlog/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix5.3.0.0/4.8.2:/opt/freeware/lib/gcc/powerpc-ibm-aix5.3.0.0/4.8.2/../../..:/usr/lib:/lib -lrt -lnsl -L/usr/local/lib -lgmodule-2.0 -lgthread-2.0 -lglib-2.0 /usr/local/lib/libintl.a -L/opt/freeware/lib -liconv -L/opt/soe/local/syslog-ng-3.4.8/eventlog/lib -levtlog -ldl -L../lib/ivykis/src /home/syslog-ng-3.4.8/lib/ivykis/src/.libs/libivykis.a -lc -Wl,-bnoentry -O2 -pthread -Wl,-brtl -pthread -Wl,--whole-archive -Wl,--no-whole-archive -pthread -Wl,-bE:.libs/libsyslog-ng.exp -Wl,-bernotok
ld: 0706-012 The -- flag is not recognized.
ld: 0706-012 The -w flag is not recognized.
ld: 0706-012 The -h flag is not recognized.
ld: 0706-012 The -- flag is not recognized.
ld: 0706-027 The -n flag is ignored.
collect2: error: ld returned 255 exit status
Makefile:789: recipe for target 'libsyslog-ng.la' failed
gmake[4]: *** [libsyslog-ng.la] Error 1
gmake[4]: Leaving directory '/home/syslog-ng-3.4.8/lib'
Makefile:1517: recipe for target 'all-recursive' failed
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory '/home/syslog-ng-3.4.8/lib'
Makefile:679: recipe for target 'all' failed
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory '/home/syslog-ng-3.4.8/lib'
Makefile:514: recipe for target 'all-recursive' failed
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory '/home/syslog-ng-3.4.8'
Makefile:418: recipe for target 'all' failed
gmake: *** [all] Error 2

mongodb: fallback not works if the first server is unreachable

In the following config the first server is unreachable, the second is up. syslog-ng does not fallback to the second server.

destination d_mongodb {
mongodb(
servers("localhost:27017", "localhost:28018")
safe_mode(yes)
);
};

console log:
Initializing MongoDB destination; address='localhost', port='27017', database='syslog', collection='messages'
Worker thread started; driver='d_mongodb#0'
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.5.3'
Error connecting to MongoDB;
Syslog connection accepted; fd='13', client='AF_INET(127.0.0.1:59023)', local='AF_INET(127.0.0.1:1112)'
Incoming log entry; line='<38>1 2014-02-27T16:11:21+02:00 localhost prg00000 1234 - - \xef\xbb\xbfseq: 0000000000, thread: 0000, runid: 1393513881, stamp: 2014-02-27T16:11:21 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPAD\x0a'
EOF occurred while reading; fd='13'
Syslog connection closed; fd='13', client='AF_INET(127.0.0.1:59023)', local='AF_INET(127.0.0.1:1112)'
Closing log transport fd; fd='13'
Error connecting to MongoDB;
Network error while inserting into MongoDB; time_reopen='60'

Patterndb packaging and link to webpage

We should make debian packages from the https://github.com/balabit/syslog-ng-patterndb
repo, to be able to distribute it easily. We should also make a link to this repo from the syslog-ng.org site.

Further plans: We should also consolidate them: some files has .pdb extensions, some
of them has .xml. And they are separated in different files, so it is hard to load them,
or manage them.

pdbtool merge converts greater-than character

Simply put, when merging two pattern databases, the > symbol is converted to &gt in the output file. This is meant to match literally in my log line (from a Cisco ASA log).

Make obsolete the mongodb host() and port() options

syslog-ng OSE admin guide writes about these options are obsolete.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guides/en/syslog-ng-ose-v3.5-guide-admin/html-single/index.html#reference-destination-mongodb

There is also missing a config warning for that options.

Grammar does not work with bison 3+

There are multiple issues when trying to compile the grammar with bison 3:

• The configure script flags 3.0 as too old (I have an easy fix for this)
• YYID appears to be gone
• bison throws a ton of warnings, we previously grepped through (we need -Wno-other in AM_YFLAGS now)

For the moment, I reverted my Debian unstable and Ubuntu Trusty build environments to bison 2.7, but that won't do in the long run.

test_basicfuncs cause SIGSEGV on ubuntu trusty

relevant output for make check:
./test-driver: line 107: 8752 Segmentation fault (core dumped) "$@" > $log_file 2>&1
FAIL: modules/basicfuncs/tests/test_basicfuncs

bt for test_basicfuncs's core:
Program received signal SIGSEGV, Segmentation fault.
0xb7f49de0 in g_static_mutex_get_mutex_impl@plt ()
from /home/micek/syslog-ng-collector/checkout-dir/syslog-ng-ose-3.6/lib/.libs/libsyslog-ng-3.6.0alpha0.so
(gdb) bt
#0 0xb7f49de0 in g_static_mutex_get_mutex_impl@plt ()

from /home/micek/syslog-ng-collector/checkout-dir/syslog-ng-ose-3.6/lib/.libs/libsyslog-ng-3.6.0alpha0.so
#1 0xb7f8a695 in log_template_append_format_with_context (self=self@entry=0x805db70, messages=messages@entry=0xbffff0f4,

num_messages=num_messages@entry=1, opts=0x80567ec, opts@entry=0x0, tz=tz@entry=0, seq_num=seq_num@entry=999, 
context_id=context_id@entry=0x804dabb "test-context-id", result=result@entry=0x806e2d0) at lib/template/templates.c:1405

#2 0xb7f8ab23 in log_template_append_format (self=self@entry=0x805db70, lm=lm@entry=0x805a690, opts=opts@entry=0x0,

tz=tz@entry=0, seq_num=seq_num@entry=999, context_id=context_id@entry=0x804dabb "test-context-id", 
result=result@entry=0x806e2d0) at lib/template/templates.c:1430

#3 0xb7f8ab8c in log_template_format (self=self@entry=0x805db70, lm=lm@entry=0x805a690, opts=opts@entry=0x0, tz=tz@entry=0,

seq_num=seq_num@entry=999, context_id=context_id@entry=0x804dabb "test-context-id", result=result@entry=0x806e2d0)
at lib/template/templates.c:1437

#4 0x0804b1ea in assert_template_format_with_escaping (template=template@entry=0x804cc59 "$(substr $HOST 1 3)",

escaping=escaping@entry=0, expected=expected@entry=0x804cc55 "zor") at libtest/template_lib.c:109

#5 0x0804b26f in assert_template_format (template=template@entry=0x804cc59 "$(substr $HOST 1 3)",

expected=expected@entry=0x804cc55 "zor") at libtest/template_lib.c:94

#6 0x0804abb7 in test_str_funcs () at modules/basicfuncs/tests/test_basicfuncs.c:50
#7 0x0804a80a in main (argc=1, argv=0xbffff254) at modules/basicfuncs/tests/test_basicfuncs.c:113

last commit:
commit a7df8c7
Merge: 4581c3d 83265ad
Author: Gergely Nagy [email protected]
Date: Thu May 22 15:02:21 2014 +0200

Merge pull request #128 from algernon/h/value-pairs/warning-fixes

value-pairs: Silence a few compiler warnings

uname:
Linux micek-HP-Mini-110-4100 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:31:42 UTC 2014 i686 i686 i686 GNU/Linux

lsb_release:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty

docbug: log_fetch_limit cannot be global

The OSE 3.5 documentation claims in Figure 8.3. Managing log messages in syslog-ng: "The log_fetch_limit() parameter can be set as a global option, or for every source individually.".

The syslog-ng daemon (3.5.3) doesn't agree:

Using a global log-fetch-limit() option was removed, please use a per-source log-fetch-limit();
Using a global log-iw-size() option was removed, please use a per-source log-iw-size();

Parser overwriting .classifier.class when matching over multiple parsers

Viktor replies to my original query (below):
"If a log message does not match any pattern for a parser, syslog-ng db-parser sets its .classifier.class to "unknown" regardless of the field's previous state. So if it matched on a previous parser, the next parser will overwrite it if it doesn't match on that. I think it's a bug rather than a feature, so could you please open an issue for that on github?"

I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases):

log {
filter(f_auth);
parser("login");
parser("sshd");
parser("su");
parser("sudo");
log {
filter(f_class_system);
...
};
};

The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db other than the last referenced parser() db. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') does not match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter does match.

--with-libmongo-client option does not have auto choice.

And it's default parameter is internal(aka git submodule), which is not fortunate if you are a distro packager. (mostly you want to use the system libmongo-client library).

3.6 fails to compile on FreeBSD 10

  CC       modules/afsocket/modules_afsocket_libafsocket_notls_la-transport-unix-socket.lo
../modules/afsocket/transport-unix-socket.c:145:55: warning: declaration of 'struct ucred' will not be visible outside of this function [-Wvisibility]
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:147:40: error: incomplete definition of type 'struct ucred'
  _add_nv_pair_int(aux, ".unix.pid", uc->pid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:145:55: note: forward declaration of 'struct ucred'
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:148:40: error: incomplete definition of type 'struct ucred'
  _add_nv_pair_int(aux, ".unix.uid", uc->uid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:145:55: note: forward declaration of 'struct ucred'
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:149:40: error: incomplete definition of type 'struct ucred'
  _add_nv_pair_int(aux, ".unix.gid", uc->gid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:145:55: note: forward declaration of 'struct ucred'
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                      ^
../modules/afsocket/transport-unix-socket.c:188:64: error: use of undeclared identifier 'SCM_CREDENTIALS'
      if (cmsg->cmsg_level == SOL_SOCKET && cmsg->cmsg_type == SCM_CREDENTIALS) 
                                                               ^
../modules/afsocket/transport-unix-socket.c:192:40: error: incomplete definition of type 'struct ucred'
          _feed_aux_from_procfs(aux, uc->pid);
                                     ~~^
../modules/afsocket/transport-unix-socket.c:190:18: note: forward declaration of 'struct ucred'
          struct ucred *uc = (struct ucred *) CMSG_DATA(cmsg);
                 ^
../modules/afsocket/transport-unix-socket.c:193:37: warning: incompatible pointer types passing 'struct ucred *' to parameter of type 'struct ucred *'
      [-Wincompatible-pointer-types]
          _feed_aux_from_ucred(aux, uc);
                                    ^~
../modules/afsocket/transport-unix-socket.c:145:62: note: passing argument to parameter 'uc' here
_feed_aux_from_ucred(LogTransportAuxData *aux, struct ucred *uc)
                                                             ^
../modules/afsocket/transport-unix-socket.c:248:30: error: use of undeclared identifier 'SO_PASSCRED'
  setsockopt(fd, SOL_SOCKET, SO_PASSCRED, &one, sizeof(one));
                             ^
2 warnings and 6 errors generated.

In short, ucred is a linux-ism, FreeBSD has cmsgcred with similar contents.

Timezones not supported in format-json macro?

syslog-ng 3.5.4.1 (freebsd)

config:
@Version: 3.5
log {
source { file("/dev/stdin" time_zone('PST8PDT')); };
destination { file("/dev/stdout" time_zone('UTC') template("$DATE $(format-json date=$DATE isodate=$ISODATE)\n")); };
};

$ syslog-ng -vFf ./syslogng-json-date.conf
Jun 24 00:00:01.295 host proc: message
Jun 24 07:00:01 {"isodate":"2014-06-24T00:00:01-07:00","date":"Jun 24 00:00:01"}

pdbtool merge: htmlentities break functionality

I have a pdb file which triggers an action on a given condition:

<action condition='"${appacct.difference_dur}" &gt;= "604800"'>

pdbtool merge transforms this snippet into:

<action condition='&quot;${appacct.difference_dur}&quot; &gt;= &quot;604800&quot;'>

Problem is, the condition now matches all events, even those where ${appacct.difference_dur} < 604800.