Comments (17)
I know this issue is long closed, but just ran into this three years later. We're using the official Elasticsearch deb packages.
Due to a flurry of errors being logged, one of the servers in our cluster had generated several gigabytes of logs, nearly filling the disk. Now, 3 months later, the disk actually did run out of space, resulting in a failure of the node. Fortunately, the cluster survived, and the node recovered after freeing up space.
My question is, where should I take this up? Who maintains the dpkg packages? The default behaviour should be that log files get rotated. A package which doesn't do that is basically broken.
from puppet-elasticsearch.
Hi, shouldn't this be the job of the package to provide such logrorate configs?
from puppet-elasticsearch.
Perhaps, but the es deb doesn't. It could be made optional. Also, out of the box, es creates cluster-name-specific log files that are logrotate-hostile (date in rotated filename).
I'm intending to submit a PR for this; just wanted to make sure nobody else was working on it.
Brian Lalor
[email protected]
On Jun 13, 2013, at 6:39 AM, Richard Pijnenburg [email protected] wrote:
Hi, shouldn't this be the job of the package to provide such logrorate configs?
—
Reply to this email directly or view it on GitHub.
from puppet-elasticsearch.
@blalor ES is doing its own log rotation per day if its configured in the logging.xml file.
Perhaps worth looking into there and make changes in there?
from puppet-elasticsearch.
It actually uses logging.yml, and the default will fill up the filesystem. Here's what I'm currently using in my Puppet module that configures ES (please forgive the dirty copy/paste and dirtier inline file contents; this was a quick hack):
## @todo put into elasticsearch module
file {'/etc/logrotate.d/elasticsearch':
content => "/var/log/elasticsearch/*.log {
daily
missingok
rotate 5
compress
notifempty
copytruncate
}",
}
## @todo put into elasticsearch module
file {'/etc/elasticsearch/logging.yml':
notify => Service['elasticsearch'],
content => "rootLogger: INFO, file
logger:
# log action execution errors for easier debugging
action: DEBUG
# reduce the logging for aws, too much is logged under the default INFO
com.amazonaws: WARN
index.search.slowlog: TRACE, index_search_slow_log_file
index.indexing.slowlog: TRACE, index_indexing_slow_log_file
additivity:
index.search.slowlog: false
index.indexing.slowlog: false
appender:
file:
type: file
file: \${path.logs}/\${cluster.name}.log
layout:
type: pattern
conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"
index_search_slow_log_file:
type: file
file: \${path.logs}/\${cluster.name}_index_search_slowlog.log
layout:
type: pattern
conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"
index_indexing_slow_log_file:
type: file
file: \${path.logs}/\${cluster.name}_index_indexing_slowlog.log
layout:
type: pattern
conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"
"
}
This replaces all the file appenders with non-rotational ones, and they're then cleaned with logrotate. I'd like to see this better integrated with the ES puppet module.
from puppet-elasticsearch.
Ahh, the problem is that the old logfiles don't get deleted. but it should created logfiles based on the date right?
from puppet-elasticsearch.
Yes, the default configuration generates a logfile for each day. elasticsearch.log.2013-07-26
, for example. They're never swept. A cron job to delete the old logs could work, too, but I prefer to have a consistent log filename.
from puppet-elasticsearch.
Would a solution be to provide with a cronjob that removes old logfiles based on the default logging settings ( file per day ) ?
that should be easy to provide and should solve the most common issue.
from puppet-elasticsearch.
On Aug 13, 2013, at 8:21 AM, Richard Pijnenburg [email protected] wrote:
Would a solution be to provide with a cronjob that removes old logfiles based on the default logging settings ( file per day ) ?
that should be easy to provide and should solve the most common issue.
I don't think that'd be any simpler; someone will still have to manage the shell script responsible for doing the pruning.
from puppet-elasticsearch.
@blalor very true. but if i would provide a shellscript with the puppet module :-)
from puppet-elasticsearch.
Given how easy it is to configure ES to create logfiles sanely, I feel like it would be more natural to use the existing OS support for rotation. Configuring one file and dropping another into a directory vs managing a shell script, a cron job, and making the schedule configurable.
from puppet-elasticsearch.
Good point.
Perhaps an idea to create a define or class to manage the logging stuff for ES then?
Any idea's?
from puppet-elasticsearch.
I'd say leave that out and keep it simple. I think people should choose to use the logrotate module to rotate their logs to let them decide how to configure it.
from puppet-elasticsearch.
I think providing puppet configuration for logging.yml would be a good starting point.
from puppet-elasticsearch.
I'll put the logging.yml on my todo list.
Logrotate it self i'll leave up to the user to configure with their own preferred module/settings.
from puppet-elasticsearch.
Sorry about the rotation issues, @laurikari. Like electrical mentioned, the module does have logging.yml management in it now, although rotation in the sense of a file for logrotate.d
isn't.
The upstream elasticsearch repo is the place to discuss packaging (look under the :Packaging
label for related issues). Based upon what I've read to get some context, the log4j version shipped with ES can't rotate/delete old log files. If you want to push for some logrotate files to be shipped with dev/rpm packages, that'd be a place to file issues.
If the need is intense enough, we could probably start handling rotation files by dropping a config into logrotate.d
, though I'm not familiar with how the log file timestamps may change between versions of ES.
from puppet-elasticsearch.
Thanks @tylerjl! I'll take this upstream.
from puppet-elasticsearch.
Related Issues (20)
- elasticsearch_keystore can't modify frozen String
- elasticsearch_keystore resource fails due to legacy instance code HOT 3
- Node facts not generated when Elasticsearch in cluster mode
- Could not evaluate: undefined method `join' for :name:Symbol HOT 2
- elasticsearch_plugin/ruby.rb wrong path to call plugin binary HOT 2
- ssl=true and version is not defined, gives an error HOT 1
- Setting an Elastic Version should be optional HOT 1
- New release, please HOT 1
- OpenSearch support ? HOT 1
- Elasticsearch API responded with: no handler found for uri [/_xpack/license?acknowledge=true] and method [PUT]
- File jvm.options in root is unmanaged/may be missing
- why was log4j2.properties.erb configuration commented ? HOT 1
- warnings when used with stlib 9. HOT 1
- fix lib directory permissions without changing the mode of files in the directory
- puppet-elastic_stack version HOT 1
- Use Elasticsearch 7+ as default for configuration HOT 1
- mediawiki needs 6.8.23 and I needed to change this
- Issues on the elasticsearch_component_template.rb and elasticsearch_index_template.rb HOT 2
- Problem with puppet CA and elasticsearch.keystore
- `bin/plugin` is used insead of `bin/elasticsearch-plugin` on the first run
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from puppet-elasticsearch.