Parser overwriting .classifier.class when matching over multiple parsers about syslog-ng

[bazsi]

Well, the original usecase of patterndb is that you merge them into a single file.

The unknown class was implemented not too long ago to support another usecase. Could be made an option but I dont really like too many knobs.

Is merging the databases with pdbtool merge an option for you?

[ColdForest]

Sure, this is possible. I probably would have gravitated to this initially if the multi-parser use case was documented more specifically somewhere, or if I'd come across a suitably equivalent example. However, I wasn't able to find this and ended up experimenting with various configurations. The 'junction' solution also appears to work fine for me and there are some (second-order) benefits to this (i.e., doesn't require the additional pdbtool merge step, allows for isolation of rules in independent and smaller files, etc.).

[faxm0dem]

@ColdForest we still keep the individual pdb files in /etc/syslog-ng/patterndb/, and pdbtool merge is being called automatically for us whenever there is changes in any of those files. Moreover, keeping all rules in a merged file also makes it much easier to actually test messages, and understand which rule they will trigger using pdbtool test or pdbtool match.

[ColdForest]

Thanks for your thoughts. If merging is the preferred workflow then I'm happy to do this (but I will just say again that the junction solution also works for me). A couple questions though: 1) how do you implement the "automated" merge (based on changes in the patterndb files), and 2) the OSE - v3.5 - admin guide has several references to "see the 'The syslog-ng Administrator Guide' for details" (which document is this? is it different than the OSE admin guide? the link resolves to nothing useful, just a list of documents like the OSE admin guide, which is a bit self-referential ;))?

[bazsi]

There's update-patterndb script, that merges patterns. Syslog-ng then recognizes the modification and reloads patterns automatically. You might use inotify-tools to watch for file changes and trigger an update-patterndb call.

There's only one admin guide so these are probably broken links. Just drop an email to the docs team at [email protected]

[ColdForest]

Thanks @bazsi.

BTW, the documentation does also provide this statement regarding log statement syntax in the manual page for syslog-ng.conf:

log {
source(s1); source(s2); ...
optional_element(filter1|parser1|rewrite1);...
optional_element(filter2|parser2|rewrite2);...
destination(d1); destination(d2); ...
flags(flag1[, flag2...]);
};

This does give one the impression that multiple parser() references can exist within a single-level log statement block.

[faxm0dem]

@ColdForest we use a soonish-to-be-released puppet module to handle the automatic merge, test, and reload. I can give you a beta-version if you're interested.

[ColdForest]

@faxm0dem Cool, thanks for the offer. Let me play around a bit first to see where I land here (my current junction-based solution provides at least run-time correctness so any further work I do to [auto-]merge pattern files will be a secondary activity). Cheers.

[bazsi]

You can use multiple parsers, as there are other kind of parsers too

See for instance csv-parser()

[ihrwein]

@ColdForest: IMHO you can find faxmodem's Puppet module here: https://github.com/ccin2p3/puppet-patterndb

Can we close this issue?

[ColdForest]

Thanks for checking back: I'm fine with you guys closing this if you think this has been addressed (the original reason for opening this ticket was by request from Viktor, who thought the originally observed behaviour was a bug - not sure if this "bug" has actually been "fixed"?).

[faxm0dem]

I think it is not fixed, and we need to decide if it will be fixed.
We have the following workaround in production:

filter f_pdb_classifier_unknown {
    tags(".classifier.unknown");
};
...
log{
  ...
  parser(p_pdb);
  junction {
        channel {
            filter(f_pdb_classifier_unknown);
            parser(p_pdb_fallback);
        };
        channel {
            flags(fallback);
        };
    };
};
...

[bazsi]

I am not entirely sure the best course of action here, .classifier.class was set to unknown so it becomes possible to check whether db-parser matched. If it would leave the .class intact, it would mean that you wouldn't have that information easily.

On the other hand, combining db-parser() into chains seems also to be a reasonable thing to do, especially if you use the template() option to specify the input for db-parser(). (e.g. extract some things in the first invocation and then process it again, perhaps with a different set of correllation rules).

I would consider how our plans about splitting db-parser into matching and correllation would look like and then decide this one as well. does that sound reasonable?

Thanks

[faxm0dem]

Shiny!

[ColdForest]

Sounds good to me too!

[ihrwein]

@bazsi You made some progress in the splitting process. Did you think about chained dbparsers in the meantime?

[bazsi]

Well, once we merge groupingby() we probably won't be using separate db-parser instances just for the sake of different correllation rules.

However, we might stack extraction into multiple steps,applying db-parser to different parts of the message (with different template ())

Do I can imagine changing the class if its unset or unknown by default. And add an option to clear the class before doing the db-parser processing.

This way the default would-be more intuitive but still support the more intricate usecases.

What do you think?

[faxm0dem]

What about appending the class to the previous one (using a coma separated list (or array once that will be available)) unless the previous one was unknown?

[bazsi]

Well, also doable, I was thinking about some kind of arrays, but we don't have a clear idea about that.

Hmm, TAGS does the same though.

[ihrwein]

If we used chained dbparsers we create a coupling between the "length of the chain" and the level of parsing. I mean, if we have to parse a message 3 times, that requires 3 chained dbparsers. The configuration of dbparser would be highly coupled with syslog-ng's config.

What if dbparser inject the generated messages before itself? That's only one instance, but we can support arbitrary levels of parsing. We may have to work around the possibility of loops, though.

I think supporting more than one context id is a good idea, but we have to do it in a clever way.

[furiel]

We were looking into this ticket, and decided to close this issue with documentation update. In the meantime, ifelse and app-parser became available, that makes easy to resolve the problem when subsequent db-parsers overwrite .classifier.class.

For example in case only a few db-parsers need to be used, one can use if:

@version: 3.20
log {
  source { stdin(); };
  parser { db-parser(file("sshd.xml")); };
  if (match("^unknown$" value(".classifier.class"))) {
    parser { db-parser(file("openvpn.xml")); };
  };
  destination { file("/dev/stdout" template("$PROGRAM ${.classifier.class}\n")); };
};

Or in case there are many, one could use app-parser:

@version: 3.20

application sshd[custom-patterndb] {
    parser { db-parser(file("sshd.xml") drop-unmatched(yes)); };
};

application openvpn[custom-patterndb] {
    parser { db-parser(file("openvpn.xml") drop-unmatched(yes)); };
};

log {
  source { stdin(); };
  parser { app-parser(topic(custom-patterndb)); };
  destination { file("/dev/stdout" template("$PROGRAM ${.classifier.class}\n")); };
};

Other related files:

openvpn.xml

<patterndb version='5'><ruleset name='openvpn' id='12345678'>
<pattern>openvpn</pattern>
<rules>
  <rule id='12347598' class="openvpn">
     <patterns><pattern>openvpn</pattern></patterns>
  </rule>
</rules>
</ruleset>
</patterndb>

sshd.xml

<patterndb version='5'><ruleset name='sshd' id='12345678'>
<pattern>sshd</pattern>
<rules>
  <rule id='12347598' class="sshd">
     <patterns><pattern>sshd</pattern></patterns>
  </rule>
</rules>
</ruleset>
</patterndb>

To test:

echo "sshd: sshd\nopenvpn: openvpn" | ./syslog-ng -Fe -f ../etc/patterndb.conf

results in:

syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'
[2019-03-20T15:44:55.984714] Log pattern database reloaded; file='sshd.xml', version='5', pub_date='(null)'
[2019-03-20T15:44:55.984802] Log pattern database reloaded; file='openvpn.xml', version='5', pub_date='(null)'
[2019-03-20T15:44:55.985074] syslog-ng starting up; version='3.20.1'
[2019-03-20T15:44:55.985492] syslog-ng shutting down; version='3.20.1'
sshd sshd
openvpn openvpn

If any participants see a use case for db-parser chaining, please give it a fresh start in a new ticket, collecting the use case, example messages etc.