logrotate support needed about puppet-elasticsearch

[laurikari]

I know this issue is long closed, but just ran into this three years later. We're using the official Elasticsearch deb packages.

Due to a flurry of errors being logged, one of the servers in our cluster had generated several gigabytes of logs, nearly filling the disk. Now, 3 months later, the disk actually did run out of space, resulting in a failure of the node. Fortunately, the cluster survived, and the node recovered after freeing up space.

My question is, where should I take this up? Who maintains the dpkg packages? The default behaviour should be that log files get rotated. A package which doesn't do that is basically broken.

[electrical]

Hi, shouldn't this be the job of the package to provide such logrorate configs?

[blalor]

Perhaps, but the es deb doesn't. It could be made optional. Also, out of the box, es creates cluster-name-specific log files that are logrotate-hostile (date in rotated filename).

I'm intending to submit a PR for this; just wanted to make sure nobody else was working on it.

Brian Lalor
[email protected]

On Jun 13, 2013, at 6:39 AM, Richard Pijnenburg [email protected] wrote:

Hi, shouldn't this be the job of the package to provide such logrorate configs?

—
Reply to this email directly or view it on GitHub.

[electrical]

@blalor ES is doing its own log rotation per day if its configured in the logging.xml file.
Perhaps worth looking into there and make changes in there?

[blalor]

It actually uses logging.yml, and the default will fill up the filesystem. Here's what I'm currently using in my Puppet module that configures ES (please forgive the dirty copy/paste and dirtier inline file contents; this was a quick hack):

    ## @todo put into elasticsearch module
    file {'/etc/logrotate.d/elasticsearch':
        content => "/var/log/elasticsearch/*.log {
    daily
    missingok
    rotate 5
    compress
    notifempty
    copytruncate
}",
    }

    ## @todo put into elasticsearch module
    file {'/etc/elasticsearch/logging.yml':
        notify  => Service['elasticsearch'],
        content => "rootLogger: INFO, file
logger:
  # log action execution errors for easier debugging
  action: DEBUG
  # reduce the logging for aws, too much is logged under the default INFO
  com.amazonaws: WARN

  index.search.slowlog: TRACE, index_search_slow_log_file
  index.indexing.slowlog: TRACE, index_indexing_slow_log_file

additivity:
  index.search.slowlog: false
  index.indexing.slowlog: false

appender:
  file:
    type: file
    file: \${path.logs}/\${cluster.name}.log
    layout:
      type: pattern
      conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"

  index_search_slow_log_file:
    type: file
    file: \${path.logs}/\${cluster.name}_index_search_slowlog.log
    layout:
      type: pattern
      conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"

  index_indexing_slow_log_file:
    type: file
    file: \${path.logs}/\${cluster.name}_index_indexing_slowlog.log
    layout:
      type: pattern
      conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"
"
    }

This replaces all the file appenders with non-rotational ones, and they're then cleaned with logrotate. I'd like to see this better integrated with the ES puppet module.

[electrical]

Ahh, the problem is that the old logfiles don't get deleted. but it should created logfiles based on the date right?

[blalor]

Yes, the default configuration generates a logfile for each day. elasticsearch.log.2013-07-26, for example. They're never swept. A cron job to delete the old logs could work, too, but I prefer to have a consistent log filename.

[electrical]

Would a solution be to provide with a cronjob that removes old logfiles based on the default logging settings ( file per day ) ?
that should be easy to provide and should solve the most common issue.

[blalor]

On Aug 13, 2013, at 8:21 AM, Richard Pijnenburg [email protected] wrote:

Would a solution be to provide with a cronjob that removes old logfiles based on the default logging settings ( file per day ) ?
that should be easy to provide and should solve the most common issue.

I don't think that'd be any simpler; someone will still have to manage the shell script responsible for doing the pruning.

[electrical]

@blalor very true. but if i would provide a shellscript with the puppet module :-)

[blalor]

Given how easy it is to configure ES to create logfiles sanely, I feel like it would be more natural to use the existing OS support for rotation. Configuring one file and dropping another into a directory vs managing a shell script, a cron job, and making the schedule configurable.

[electrical]

Good point.
Perhaps an idea to create a define or class to manage the logging stuff for ES then?
Any idea's?

[jamescarr]

I'd say leave that out and keep it simple. I think people should choose to use the logrotate module to rotate their logs to let them decide how to configure it.

[blalor]

I think providing puppet configuration for logging.yml would be a good starting point.

[electrical]

I'll put the logging.yml on my todo list.
Logrotate it self i'll leave up to the user to configure with their own preferred module/settings.

[tylerjl]

Sorry about the rotation issues, @laurikari. Like electrical mentioned, the module does have logging.yml management in it now, although rotation in the sense of a file for logrotate.d isn't.

The upstream elasticsearch repo is the place to discuss packaging (look under the :Packaging label for related issues). Based upon what I've read to get some context, the log4j version shipped with ES can't rotate/delete old log files. If you want to push for some logrotate files to be shipped with dev/rpm packages, that'd be a place to file issues.

If the need is intense enough, we could probably start handling rotation files by dropping a config into logrotate.d, though I'm not familiar with how the log file timestamps may change between versions of ES.

[laurikari]

Thanks @tylerjl! I'll take this upstream.