lusis / chef-logstash Goto Github PK
View Code? Open in Web Editor NEWSemi-official Logstash cookbook
License: Other
Semi-official Logstash cookbook
License: Other
I muddled through getting this to install, coming across multiple issues (#129, #130, #131) and got the recipe to actually complete. Sadly, it produced a totally wrong apache file:
NameVirtualHost *:4500
<VirtualHost *:4500>
ServerName 10.0.2.15
DocumentRoot ""
<Directory "">
DirectoryIndex index.php
</Directory>
</VirtualHost>
I got the ServerAlias coming from hostname and put the document root in, but when I got to directory index I just gave up. It needs passenger or something, but this...
I dont mean to sound like a douche, but who pushed this? Why is it so far from being vaguely right?
probably via --grok-patterns-path PATH
in LOGSTASH_OPTS
The block of code that creates the patterns in logstash::agent notified "service[logstash_server]" instead of "service[logstash_agent]"
Please consider supporting the elasticsearch_http output directive in addition to just elasticsearch, as elasticsearch_http is less version specific.
Does the 0.7.0 branch still depend on RabbitMQ?
https://github.com/lusis/chef-logstash/blob/0.7.0/metadata.rb#L13
I thought Logstash 1.2.x used Redis instead.
By default the embedded elastic search will look for it's data directory under the current directory when logstash_server is started. The current init script does not change the CWD, so on startup the embedded elastic search looks for it's data directly under the root directory "/data/elasticsearch" and so it fails to find or create the directory.
One solution is to set the CWD appropriately in the init script. Or we could set the es.path.home or es.path.data variables on the command line: https://logstash.jira.com/browse/LOGSTASH-125
The former is the simplest and doesn't appear to break anything for agent or server inits.
Was "all rights reserved" intentional? Perhaps update this to be Apache 2.0?
https://github.com/lusis/chef-logstash/blob/master/recipes/pyshipper.rb#L7
Can we please add example of logstash_server, logstash_client, and elasticsearch_server roles into readme file?
Currently you have only ruby code which is not even a role, and yet you expect to have those 3 roles in roles/ directory.
This is a node template which is specified in readme
name "logstash_server"
description "Attributes and run_lists specific to FAO's logstash instance"
default_attributes(
:logstash => {
:server => {
:enable_embedded_es => false,
:inputs => [
:amqp => {
:type => "all",
:host => "127.0.0.1",
:exchange => "rawlogs",
:name => "rawlogs_consumer"
}
]}})
we need some example of roles if you expect them.. not nodes, would be nice to have three of them. logstash_client.json, logstash_server.json. elasticsearch_server.json (or else kibana won't work out-of-the-box)
I am a total n00b to logstash.
queries like @source_host:"foo1.example.org" work but @source_host:"foo*.example.org" return no results
@source_host:foo* w/out quotes does work
@source_host:foo*.example.org does not
I assume this is an elasticsearch issue and may be out of scope of this cookbook
kibana's config.php has a disable_fullscan property, but it is currently set to false
index_epoch = get_index_epoch(unprefixed_index_name)
The get_index_epoch() method takes a separator argument so that it can determine the epoch of the index ... however, you don't pass that argument. If you use a different separator value like I do, the script runs cleanly but fails to get the epoch. This means that it never deletes an index, and doesn't indicate that there was a failure either.
The line should look like this:
index_epoch = get_index_epoch(unprefixed_index_name, separator=separator)
The 1.2.x format has switched to a new conditional format that surrounds the plugin directives rather then embedded in them.
http://logstash.net/docs/1.2.2/configuration
i observe httpd restarts on every chef run, I believe this has to do w/ the apache definition used
Or document how to use nginx instead. I personally do not use apache, but would like to use this cookbook in favor of my own nonsense :)
* file[/var/log/logstash/logstash_beaver.log] action touch
- update utime on file /var/log/logstash/logstash_beaver.log
* file[/var/run/logstash/logstash_beaver.pid] action touch
- update utime on file /var/run/logstash/logstash_beaver.pid
* python_pip[beaver==22] action install
- install package python_pip[beaver==22] version latest
* template[/opt/logstash/beaver/etc/beaver.conf] action create (up to date)
* service[logstash_beaver] action enable
================================================================================
Error executing action `enable` on resource 'service[logstash_beaver]'
================================================================================
Chef::Exceptions::Exec
----------------------
/sbin/insserv -r -f logstash_beaver returned 1, expected 0
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/logstash/recipes/beaver.rb
207: service "logstash_beaver" do
208: supports :restart => true, :reload => false, :status => true
209: action [:enable, :start]
210: end
211:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/logstash/recipes/beaver.rb:207:in `from_file'
service("logstash_beaver") do
action [:enable, :start]
supports {:restart=>true, :reload=>false, :status=>true}
retries 0
retry_delay 2
service_name "logstash_beaver"
pattern "logstash_beaver"
startup_type :automatic
cookbook_name "logstash"
recipe_name "beaver"
end
root@holms:/home/holms# ls /etc/init.d/ | grep beaver
root@holms:/home/holms#
As you can see service not found. I'm using debian 7.2, any ideas/?
It seems that start_command is missing:
start
on resource 'service[logstash_agent]'58: end
59: service "logstash_agent" do
60: provider Chef::Provider::Service::Upstart
61: action [ :enable, :start ]
62: end
63: else
service("logstash_agent") do
provider Chef::Provider::Service::Upstart
action [:enable, :start]
supports {:restart=>false, :reload=>false, :status=>false}
retries 0
retry_delay 2
service_name "logstash_agent"
pattern "logstash_agent"
startup_type :automatic
cookbook_name "logstash"
recipe_name "agent"
end
Cheers,
Armin
This is obvious when you think about it but the ability to define, say, multiple mutate filters or multiple tcp inputs is impossible because of duplicate key names.
I'm not sure the best way to handle this. Something that was designed to make things data driven actually only supports pretty basic use cases - one of each type of "thing".
The conditional logic in the output section of agent.conf.erb does not allow a user to provide overriden output settings if logstash_server_ip is not set.
search() results are not order-stable, so the first object returned will change randomly, causing unnecessary log stash server restarts. Sorting the results prior to picking the 'first' object resolves this. This only occurs when a search() returns more than one result.
This seems to be a greater issue under Chef 11 than Chef 10.
I have have been tasked with adding some exec commands to our logstash config. The command uses awk, so if I try running something like:
command => "cat /proc/loadavg | awk '{print $2}'"
However, I am not sure how to add this to my role, as it takes the above and turns it into:
'command' => 'cat /proc/loadavg | awk '{print $2}''
Is there a way to turn the outside "'" to quotes?
Thank you for your help.
Geoff
Hi, with version 0.5.5 I get an error when installing.
OS: Ubuntu 12.04
Looks like kibana's initialization script is missing some shell related defs.
The first error is that source
is not defined.
I checked and the default shell for the user kibana
is sh
so I replaced source
with .
and it passed, but then the next error comes which is can't file ~/.bash_profile
.
So while it makes sense that it can't find the .bash_profile when the user's shell is sh - still I don't know how to continue from here, because the next command bundle
requires some defs in bash_profile...
Anyway, I hope my short diagnosis helps. Here's the log.
Eventually I used v0.5.1 of the cookbook (which I recalled worked for me in the past) and it worked.
[2013-02-25T07:02:56+00:00] INFO: Processing bash[bundle install] action run (logstash::kibana line 103)
[2013-02-25T07:02:56+00:00] INFO: Processing service[kibana] action enable (logstash::kibana line 110)
[2013-02-25T07:02:57+00:00] INFO: Processing service[kibana] action start (logstash::kibana line 110)
================================================================================
Error executing action `start` on resource 'service[kibana]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '127'
---- Begin output of /etc/init.d/kibana start ----
#!/usr/bin/env bash
STDOUT:
STDERR: sh: 1: source: not found
---- End output of /etc/init.d/kibana start ----
Ran /etc/init.d/kibana start returned 127
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/logstash/recipes/kibana.rb
110: service "kibana" do
111: supports :status => true, :restart => true
112: action [:enable, :start]
113: subscribes :restart, [ "link[#{kibana_home}]", "template[#{kibana_home}/KibanaConfig.rb]", "template[#{kibana_home}/kibana-daemon.rb]" ]
114: end
115:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/logstash/recipes/kibana.rb:110:in `from_file'
service("kibana") do
action [:enable, :start]
supports {:status=>true, :restart=>true}
retries 0
retry_delay 2
service_name "kibana"
enabled true
pattern "kibana"
startup_type :automatic
cookbook_name "logstash"
recipe_name "kibana"
end
[2013-02-25T07:02:57+00:00] ERROR: Running exception handlers
[2013-02-25T07:02:57+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[2013-02-25T07:02:57+00:00] ERROR: Exception handlers complete
[2013-02-25T07:02:57+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-02-25T07:02:57+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: service[kibana] (logstash::kibana line 110) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '127'
---- Begin output of /etc/init.d/kibana start ----
STDOUT:
STDERR: sh: 1: source: not found
---- End output of /etc/init.d/kibana start ----
Ran /etc/init.d/kibana start returned 127
Gist of full output: https://gist.github.com/stevepereira/6288036
Error executing action `enable` on resource 'service[logstash_beaver]'
Chef::Exceptions::Exec
----------------------
/usr/sbin/update-rc.d logstash_beaver defaults returned 1, expected 0
---- Begin output of /usr/sbin/update-rc.d logstash_beaver defaults ----
STDOUT: STDERR: update-rc.d: /etc/init.d/logstash_beaver: file does not exist
---- End output of /usr/sbin/update-rc.d logstash_beaver defaults ----
Resource Declaration:
---------------------
# In /etc/chef/chef-solo-1/cookbooks/logstash/recipes/beaver.rb
207: service "logstash_beaver" do
208: supports :restart => true, :reload => false, :status => true
209: action [:enable, :start]
210: end
211:
Compiled Resource:
------------------
# Declared in /etc/chef/chef-solo-1/cookbooks/logstash/recipes/beaver.rb:207:in `from_file'
service("logstash_beaver") do
action [:enable, :start]
supports {:restart=>true, :reload=>false, :status=>true}
retries 0
retry_delay 2
service_name "logstash_beaver"
pattern "logstash_beaver"
startup_type :automatic
cookbook_name :logstash
recipe_name "beaver"
end
The version of this cookbook published at community.opscode.com is very outdated, that version still freez at 0.3.0 while the most recent version in GitHub is 0.5.1
I think at this point it's time to remove the legacy php interface and go strictly with the Ruby one. Thoughts?
Would it be possible to allow setting the node name of the elasticsearch node? When It joined my cluster it freaked me out until I understood what it was doing as part of my cluster, but would like to be able to set a name for the sake of my future self and anyone else who looks at our ES cluster.
I know I could do this by overriding the config temple, but this seems like something useful for others, and inline with the level of configuration this library already offers.
I could submit a pull request if there is a chance for this to be merged, but be warned I and NOT a ruby dev, and am fairly green with Chef still.
sv-logstash-agent-run.erb references logstash.conf and not shipper.conf like the upstart script does
The source cookbook adds a lot of dependencies that aren't typically used. Would it be possible to split it out to a different cookbook for those people who compile from source?
Report by @bretthoerner that the OOB experience on 10.04 is broken:
I'll stand up a 10.04 VM and troubleshoot.
Sorry to bother you again, but as you can see /sbin/start logstash_agent is failed.
And I've tried revert snapshot multiple times.
The only thing which I have is /etc/init/logstash_agent.conf, and it's actually ok.
[2013-11-21T15:48:11+00:00] INFO: Forking chef instance to converge...
[2013-11-21T15:48:11+00:00] INFO: *** Chef 11.8.0 ***
[2013-11-21T15:48:11+00:00] INFO: Chef-client pid: 3689
[2013-11-21T15:48:11+00:00] INFO: Run List is [role[logstash_client]]
[2013-11-21T15:48:11+00:00] INFO: Run List expands to [logstash::agent]
[2013-11-21T15:48:11+00:00] INFO: Starting Chef Run for logstash-client
[2013-11-21T15:48:11+00:00] INFO: Running start handlers
[2013-11-21T15:48:11+00:00] INFO: Start handlers complete.
[2013-11-21T15:48:11+00:00] INFO: HTTP Request Returned 404 Object Not Found:
[2013-11-21T15:48:12+00:00] INFO: Loading cookbooks [ant, apt, ark, aws, build-essential, chef_handler, dmg, erlang, git, java, logrotate, logstash, python, rabbitmq, runit, windows, yum, yumrepo]
[2013-11-21T15:48:12+00:00] INFO: Processing package[openjdk-6-jdk] action install (java::openjdk line 35)
[2013-11-21T15:48:13+00:00] INFO: Processing package[openjdk-6-jre-headless] action install (java::openjdk line 35)
[2013-11-21T15:48:13+00:00] INFO: Processing bash[update-java-alternatives] action run (java::openjdk line 39)
[2013-11-21T15:48:13+00:00] INFO: bash[update-java-alternatives] ran successfully
[2013-11-21T15:48:13+00:00] INFO: Processing ruby_block[set-env-java-home] action run (java::set_java_home line 19)
[2013-11-21T15:48:13+00:00] INFO: ruby_block[set-env-java-home] called
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/etc/profile.d] action create (java::set_java_home line 26)
[2013-11-21T15:48:13+00:00] INFO: Processing file[/etc/profile.d/jdk.sh] action create (java::set_java_home line 30)
[2013-11-21T15:48:13+00:00] INFO: Processing package[sun-java6-jdk] action purge (java::default line 25)
[2013-11-21T15:48:13+00:00] INFO: Processing package[sun-java6-bin] action purge (java::default line 25)
[2013-11-21T15:48:13+00:00] INFO: Processing package[sun-java6-jre] action purge (java::default line 25)
[2013-11-21T15:48:13+00:00] INFO: Processing group[logstash] action create (logstash::default line 9)
[2013-11-21T15:48:13+00:00] INFO: Processing user[logstash] action create (logstash::default line 13)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash] action create (logstash::default line 23)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent] action create (logstash::agent line 60)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/bin] action create (logstash::agent line 68)
[2013-11-21T15:48:13+00:00] INFO: Processing link[/var/lib/logstash/bin] action create (logstash::agent line 75)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/etc] action create (logstash::agent line 68)
[2013-11-21T15:48:13+00:00] INFO: Processing link[/var/lib/logstash/etc] action create (logstash::agent line 75)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/lib] action create (logstash::agent line 68)
[2013-11-21T15:48:13+00:00] INFO: Processing link[/var/lib/logstash/lib] action create (logstash::agent line 75)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/tmp] action create (logstash::agent line 68)
[2013-11-21T15:48:13+00:00] INFO: Processing link[/var/lib/logstash/tmp] action create (logstash::agent line 75)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/log] action create (logstash::agent line 68)
[2013-11-21T15:48:13+00:00] INFO: Processing link[/var/lib/logstash/log] action create (logstash::agent line 75)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/etc/conf.d] action create (logstash::agent line 80)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/opt/logstash/agent/etc/patterns] action create (logstash::agent line 87)
[2013-11-21T15:48:13+00:00] INFO: Processing remote_file[/opt/logstash/agent/lib/logstash-1.2.2.jar] action create_if_missing (logstash::agent line 107)
[2013-11-21T15:48:13+00:00] INFO: Processing link[/opt/logstash/agent/lib/logstash.jar] action create (logstash::agent line 116)
[2013-11-21T15:48:13+00:00] INFO: Processing template[/opt/logstash/agent/etc/conf.d/logstash.conf] action create (logstash::agent line 131)
[2013-11-21T15:48:13+00:00] INFO: Processing directory[/var/log/logstash] action create (logstash::agent line 160)
[2013-11-21T15:48:13+00:00] INFO: Processing template[/etc/init/logstash_agent.conf] action create (logstash::agent line 173)
[2013-11-21T15:48:13+00:00] INFO: Processing service[logstash_agent] action enable (logstash::agent line 179)
[2013-11-21T15:48:13+00:00] INFO: Processing service[logstash_agent] action start (logstash::agent line 179)
�[0m
================================================================================�[0m
�[31mError executing action `start` on resource 'service[logstash_agent]'�[0m
================================================================================�[0m
�[0m
Chef::Exceptions::Exec�[0m
----------------------�[0m
/sbin/start logstash_agent returned 1, expected 0�[0m
�[0m
Resource Declaration:�[0m
---------------------�[0m
# In /var/chef/cache/cookbooks/logstash/recipes/agent.rb
179: service "logstash_agent" do
180: provider Chef::Provider::Service::Upstart
181: action [ :enable, :start ]
182: end
183: else
�[0m
�[0m
Compiled Resource:�[0m
------------------�[0m
# Declared in /var/chef/cache/cookbooks/logstash/recipes/agent.rb:179:in `from_file'
service("logstash_agent") do
provider Chef::Provider::Service::Upstart
action [:enable, :start]
supports {:restart=>false, :reload=>false, :status=>false}
retries 0
retry_delay 2
service_name "logstash_agent"
enabled true
pattern "logstash_agent"
startup_type :automatic
cookbook_name "logstash"
recipe_name "agent"
end
�[0m
�[0m
[2013-11-21T15:48:13+00:00] INFO: Running queued delayed notifications before re-raising exception
[2013-11-21T15:48:13+00:00] ERROR: Running exception handlers
[2013-11-21T15:48:13+00:00] ERROR: Exception handlers complete
[2013-11-21T15:48:13+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-11-21T15:48:13+00:00] ERROR: service[logstash_agent] (logstash::agent line 179) had an error: Chef::Exceptions::Exec: /sbin/start logstash_agent returned 1, expected 0
[2013-11-21T15:48:13+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Here's a role:
ubuntu@chef:~/lab/roles$ cat logstash_client.json
{
"name": "logstash_client",
"json_class": "Chef::Role",
"run_list": [
"recipe[logstash::agent]"
],
"description": "",
"chef_type": "role",
"override_attributes": {
}
}
ubuntu@chef:~/lab/roles$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.3 LTS
Release: 12.04
Codename: precise
When [:logstash][:kibana][:basedir] gets set it gets the value set it gets the value set in attibutes/default. If you set a default or override for [:logstash][:basedir] in a role it will work for the server and agent, but not for kibana, which will go under /opt/logstash
Would it be a good addition to extend the cookbook to support logstash using AWS discovery to find an elasticearch node with the AWS plugin. What would be the best way to do it? Perhaps add an elasticsearch.yml in the working directory of logstash and then download the plugin to somewhere especially for logstash's ES client?
When I do chef-client -o "recipe[logstash::agent]" -l info
, I get
HTTP Request Returned 412 Precondition Failed:
{
"message"=>"Unable to satisfy constraints on package yum due to solution constraint (logstash >= 0.0.0). Solution constraints that may result in a constraint on yum: [(logstash = 0.6.1) -> (git = 2.5.2) -> (runit = 1.1.4) -> (yum >= 0.0.0)], [(logstash = 0.6.1) -> (git = 2.5.2) -> (yum >= 0.0.0)], [(logstash = 0.6.1) -> (rbenv = 1.6.5) -> (git = 2.5.2) -> (runit = 1.1.4) -> (yum >= 0.0.0)], [(logstash = 0.6.1) -> (rbenv = 1.6.5) -> (git = 2.5.2) -> (yum >= 0.0.0)], [(logstash = 0.6.1) -> (runit = 1.1.4) -> (yum >= 0.0.0)], [(logstash = 0.6.1) -> (python = 1.4.0) -> (yum >= 0.0.0)], [(logstash = 0.6.1) -> (rabbitmq = 2.3.0) -> (erlang = 1.3.2) -> (yum >= 0.5.0)], [(logstash = 0.6.1) -> (yumrepo = 2.0.0) -> (yum >= 2.3.0)]",
"unsatisfiable_run_list_item"=>"(logstash >= 0.0.0)",
"non_existent_cookbooks"=>[],
"most_constrained_cookbooks"=>[
"yum = 2.2.2 -> []"
]
}
Consider adding a recipe that optionally installs tokyocabinet / grok to support the grok filter.
While the default port is set at 80, the kibana user is unprivileged and cannot claim that port. This causes the init script to fail on RedHat systems when using the defaults.
The make file for logstash uses system ruby, but ruby is not installed to the system by default when running -d chef-full. So the install fails.
Mixlib::ShellOut::ShellCommandFailed: execute[build-logstash](logstash::source line 28) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of make clean && make VERSION=36a3b3d55bca6c9a8caf4705fcb64972d3c6af14 ----
STDOUT: => Cleaning up
make[1]: Entering directory /opt/logstash/source/lib/logstash/config' ragel -R grammar.rl make[1]: Leaving directory
/opt/logstash/source/lib/logstash/config'
STDERR: make[1]: ragel: Command not found
make[1]: *** [grammar.rb] Error 127
make: *** [lib/logstash/config/grammar.rb] Error 2
---- End output of make clean && make VERSION=36a3b3d55bca6c9a8caf4705fcb64972d3c6af14 ----
Vagrant up fails with the following mini-test handler error.
NOTE: I had to upgrade the Chef client to v11 to this far.
[2013-06-26T04:34:49+02:00] DEBUG: gem_package[minitest-chef-handler] using gem '/opt/vagrant_ruby/bin/gem'
[2013-06-26T04:34:49+02:00] DEBUG: gem_package[minitest-chef-handler] no installed version found for minitest-chef-handler (>= 0)
[2013-06-26T04:34:50+02:00] DEBUG: found gem minitest-chef-handler version 1.0.1 for platform ruby from http://rubygems.org/
[2013-06-26T04:35:05+02:00] DEBUG: gem_package[minitest-chef-handler] resetting gem environment to default
[2013-06-26T04:35:05+02:00] DEBUG: filtered backtrace of compile error: /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file'
[2013-06-26T04:35:05+02:00] DEBUG: filtered backtrace of compile error: /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file'
[2013-06-26T04:35:05+02:00] DEBUG: backtrace entry for compile error: '/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file''
[2013-06-26T04:35:05+02:00] DEBUG: Line number of compile error: '14'
================================================================================
Recipe Compile Error in /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb
================================================================================
LoadError
---------
cannot load such file -- minitest-chef-handler
Cookbook Trace:
---------------
/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file'
Relevant File Content:
----------------------
/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:
7: gem_package "minitest-chef-handler" do
8: action :nothing
9: end.run_action(:install)
10:
11: Gem.clear_paths
12: # Ensure minitest gem is utilized
13: gem "minitest"
14>> require "minitest-chef-handler"
15:
16: recipes = node['recipes']
17: if recipes.empty? and Chef::Config[:solo]
18: #If you have roles listed in your run list they are NOT expanded
19: recipes = node.run_list.map {|item| item.name if item.type == :recipe }
20: end
21:
22: # Directory to store cookbook tests
23: directory "minitest test location" do
[2013-06-26T04:35:06+02:00] DEBUG: Re-raising exception: LoadError - cannot load such file -- minitest-chef-handler
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:in `instance_eval'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:in `from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/cookbook_version.rb:346:in `load_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:151:in `load_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:139:in `block in compile_recipes'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:in `compile_recipes'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:74:in `compile'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:86:in `load'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:224:in `setup_run_context'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:467:in `do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:239:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-solo:25:in `<top (required)>'
/usr/bin/chef-solo:23:in `load'
/usr/bin/chef-solo:23:in `<main>'
[2013-06-26T04:35:06+02:00] ERROR: Running exception handlers
[2013-06-26T04:35:06+02:00] ERROR: Exception handlers complete
[2013-06-26T04:35:06+02:00] FATAL: Stacktrace dumped to /etc/vagrant-chef/chef-stacktrace.out
[2013-06-26T04:35:06+02:00] DEBUG: LoadError: cannot load such file -- minitest-chef-handler
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:in `instance_eval'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:in `from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/cookbook_version.rb:346:in `load_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:151:in `load_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:139:in `block in compile_recipes'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:in `compile_recipes'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:74:in `compile'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:86:in `load'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:224:in `setup_run_context'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:467:in `do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:239:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-solo:25:in `<top (required)>'
/usr/bin/chef-solo:23:in `load'
/usr/bin/chef-solo:23:in `<main>'
[2013-06-26T04:35:06+02:00] FATAL: LoadError: cannot load such file -- minitest-chef-handler
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.
The pattern field in a grok filter can be repeated in valid logstash configuration, but since it is currently treated as a k/v this is not possible.
Sample:
grok {
patterns_dir => '/opt/logstash/server/etc/patterns'
pattern => '%{TOMCAT2LOCALHOST}'
pattern => 'INFO: %{GREEDYDATA:infomessage}'
pattern => '%{GREEDYDATA:continuation}'
type => 'tomcat-error'
}
After applying the logstash::client recipe, my rhel 6.2 boxes think they are ubuntu
node['normal_attrs']['platform?'] returns 'ubuntu'
removing that same recipe causes it to go back to redhat
I don't see anything in the logstash cookbook itself that could cause this, so i am digging into the dependencies:
apache2 php build-essential git python java ant
investigating
Using 0.6.1 exactly as is I get the following error when I include the kibana recipe:
https://gist.github.com/philsturgeon/5859679
If I move the user definitions to above the rbenv includes it seems to work fine. E.g:
kibana_base = node['logstash']['kibana']['basedir']
kibana_home = node['logstash']['kibana']['home']
kibana_log_dir = node['logstash']['kibana']['log_dir']
kibana_pid_dir = node['logstash']['kibana']['pid_dir']
user "kibana" do
supports :manage_home => true
home "/home/kibana"
shell "/bin/bash"
end
node.override[:rbenv][:group_users] = [ "kibana" ]
include_recipe "rbenv::default"
include_recipe "rbenv::ruby_build"
I'm not going to PR this one as its pretty vague and I've obviously just hacked this to make it work, but I wanted to let you know.
Hi - we have Debian boxes so it looks like we're out of luck with your repo.
Would you say that's a fair assessment?
Do you know of a cookbook that does provide chef-logstash for Debian boxes?
Thanks,
Martin.
I'd like to pull Kibana out of this cookbook alltogether for several reasons:
Logstash jar will have kibana emdedded in the next release
Kind of a SRP thing
There are now two kibana cookbooks:
I like the idea of a cookbook "owning" its dependencies but Kibana was always in there because the default webui in logstash was so depressing.
Comments? @philsturgeon @bryanwb @ChrisLundquist
I'm trying to generate a Logstash server configuration that looks something like this:
filter {
multiline {
'pattern' => '^[^0-9]'
'what' => 'previous'
}
grok {
'pattern' => '%{DATE:log_date} %{TIME:log_time},%{NUMBER:log_time_millis} [%{DATA:log_context}] %{LOGLEVEL:log_level}\s+%{JAVACLASS:java_class} - %{GREEDYDATA:log_message}'
'tags' => ['grok_java']
}
mutate {
'replace' => [ "@timestamp", "%{log_date}T%{log_time}.%{log_time_millis}Z" ]
'replace' => [ "@message", "%{log_message}" ]
}
mutate {
'remove' => ['log_message', 'log_date', 'log_time', 'log_time_millis']
}
}
From what I see in LogstashConf this isn't possible because the filter array is sorted by hash name which does not allow multiple plugin instances and also orders filters alphabetically instead of in declaration order. Is there another way to do this or should I work on modifying logstash_conf.erb to support this?
I've got multiple duplicates of this message in logs.
This occurs when I restart beaver service:
service logstash_beaver restart
# tail -f /var/log/logstash/logstash_beaver.log
beaver: error: argument -t/--transport: expected one argument
usage: beaver [-h] [-c CONFIG] [-d] [-D] [-f FILES [FILES ...]]
[-F {json,msgpack,string,raw}] [-H HOSTNAME] [-m {bind,connect}]
[-l OUTPUT] [-p PATH] [-P PID]
[-t {rabbitmq,redis,stdout,zmq,udp}] [-v] [--fqdn]
Hi,
the chef community site lists 0.5.9 as the latest cookbook version, however here on github it is 0.6.1. Can the community version be updated as well?
http://community.opscode.com/cookbooks/logstash
Kind regards,
Maik
The Redisio cookbok (for example) has a nice build pattern where install and enable are separate recipes, which I like because it allows me more control over the server behavior.
I'd love to see the same ability added here.
Thanks!
so I can tell what's different between what's over at http://community.opscode.com/cookbooks/logstash
I like to keep my nodes simple and uniform, so as a general rule they all run the Logstash agent.
That goes for the node that runs the server, too. This allows me to write the configuration with no special cases.
This cookbook mostly works with this setup, except that both agent and server try to create symlinks in the same directories, overwriting each other on every Chef run.
Would you consider making the symlinks optional?
Also, the logrotate configuration file has the same name for both—the fix here is trivial, I'm going to just send a pull request.
Using chef-logstash
version 0.5.6
I get the following error:
NoMethodError: undefined method `notifies' for "link[/opt/logstash/kibana/current]":String
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/resource.rb:313:in `subscribes'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/resource.rb:312:in `each'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/resource.rb:312:in `subscribes'
/tmp/vagrant-chef-1/chef-solo-1/cookbooks/logstash/recipes/kibana.rb:128:in `from_file'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/mixin/recipe_definition_dsl_core.rb:74:in `instance_eval'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/mixin/recipe_definition_dsl_core.rb:74:in `method_missing'
/tmp/vagrant-chef-1/chef-solo-1/cookbooks/logstash/recipes/kibana.rb:125:in `from_file'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/cookbook_version.rb:578:in `load_recipe'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/mixin/language_include_recipe.rb:46:in `load_recipe'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/mixin/language_include_recipe.rb:33:in `include_recipe'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/mixin/language_include_recipe.rb:27:in `each'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/mixin/language_include_recipe.rb:27:in `include_recipe'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:72:in `load'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:69:in `each'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:69:in `load'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:199:in `setup_run_context'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:162:in `run'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/solo.rb:207:in `run_application'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/solo.rb:195:in `loop'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/solo.rb:195:in `run_application'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application.rb:70:in `run'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.12.0/bin/chef-solo:25
/opt/vagrant_ruby/bin/chef-solo:19:in `load'
Platform: Debian 7.1
Logstash Cookbook: 0.5.9
Chef Client: 11.4.4
Whenever I add the logstash::agent
recipe to a new node, the Chef run fails with the following error (segment of output):
- enable service runit_service[logstash_agent]
================================================================================
Error executing action `enable` on resource 'runit_service[logstash_agent]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /usr/bin/sv restart /etc/service/logstash_agent ----
STDOUT: timeout: down: /etc/service/logstash_agent: 0s, normally up, want up
STDERR:
---- End output of /usr/bin/sv restart /etc/service/logstash_agent ----
Ran /usr/bin/sv restart /etc/service/logstash_agent returned 1
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/runit/libraries/provider_runit_service.rb:179:in `restart_service'
/var/chef/cache/cookbooks/runit/libraries/provider_runit_service.rb:95:in `action_enable'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/logstash/recipes/agent.rb
110: runit_service "logstash_agent"
111: end
I was able to find out that the command failed because, while the tree structure for the library has been laid out, the actual logstash jarfile is missing.
opsdude@XXXX:~$ sudo /etc/service/logstash_agent/run
Unable to access jarfile /opt/logstash/agent/lib/logstash.jar
opsdude@XXXX:~$ tree /opt/logstash/agent/
/opt/logstash/agent/
├── bin
├── etc
│ ├── conf.d
│ └── patterns
├── lib
├── log
└── tmp
7 directories, 0 files
The remote_file
and template
resource responsible for installing the agent jarfile and config, respectively, are defined after the runit service declaration.
Does it not make sense to have these resources defined beforehand?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.