Improving logrus performance about logrus

[sirupsen]

Hi @pjvds thanks a lot for your input!

This is the actually the first time I've heard of people who've been running into serious performance issues with the logger in production. Performance hasn't been a hardcore goal (e.g. in some cases we've favoured nice API and usability over performance, although obviously harvesting any performance benefits we could) because the time spent in logrus in the applications I have has been negligible, likely because I log much less than you do. However, if you're seeing problems, let's discuss how we can best solve them.

How much are you logging?

Can you track down more specifically which codepath is slow (you're mentioning the low-level calls which is great)? It might be a hook or a specific part of the formatter.

I have a hunch it has to do with the formatter, e.g. it's doing a sort by default which is notoriously slow (but hasn't been a big deal in my cases).

[pi3r]

@pjvds do you have a branch with your current changes? I'm just curious :)

[StabbyCutyou]

I would be interested in seeing the proposed differences to the surface API this might entail. I love how easy to use and extend logrus messages are, but with an app that has very tight performance constraints, I would additionally love to know that I'm not losing time in my logging framework of choice.

[placeybordeaux]

As a consumer I am willing to refactor code to gain a performance increase assuming the numbers you are quoting are realistic. However logrus is imported by 702 packages. Breaking changes are pretty rough, any way to offer a new version or different API for the faster stuff?

[cep21]

I as well have seem performance issues w/ logrus related to the garbage creation. I solved this by removing the WithField() calls for debug logs by using the following function.

// Debug to logger a key/value pair and message.  Intended to save the mem alloc that WithField creates
func Debug(l *logrus.Logger, key string, val interface{}, msg string) {
    if l.Level >= logrus.DebugLevel {
        l.WithField(key, val).Debug(msg)
    }
}

[sirupsen]

@cep21 would being able to pass a function that's lazily evaluated to WithField{,s} work for you? Or is it simply because you're doing so many map allocations?

[cep21]

For me, on this issue, it was the large number of allocs and the GC pressure I was trying to reduce. l.WithField(key, val).Debug(msg) requires a few allocs even when I'm not at debug level.

The only I can see to get around this is to reverse the order.

  l.Debug(msg).WithField(key, val).Fire()

With this order of operations, l.Debug(msg) could return a no-op singleton and make any future calls no-op non memory allocs. With the order logrus currently has, l.WithField(key, val) requires an alloc because it doesn't know that Debug is coming next.

[imkira]

Good to find more people discussing this!

@cep21 Actually I am submitting a PR to fix the first issue you talked about.
Regarding the logger.WithField / WithFields problem, I am also proposing a similar approach to yours.
Please check #154

I am basically proposing either

• inverted logic: log.Info(...).WithFields(...)
• like your Fire but trying to be semver compatible: log.WithLevel(Info).WithFields(...).Log(...)

[pmoust]

Rel: PERF: use buffer pool in json formatter #755

[alexstov]

Hi, WithFields over-allocates memory even when it doesn't have to. In this particular example (see attached) the log level is set to 'warn', but a call to Trace allocated ~32Mb. Has anybody found a reliable workaround for this issue? I haven't profiled the cep21 solution above. Does it work? Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.