vkhatri / chef-filebeat Goto Github PK

View Code? Open in Web Editor NEW

32.0 6.0 83.0 335 KB

Chef Cookbook to Manage Elastic Filebeat https://supermarket.chef.io/cookbooks/filebeat

License: Apache License 2.0

Ruby 94.14% HTML 2.61% Shell 3.25%

chef-cookbook filebeat logging beats

chef-filebeat's People

Contributors

Stargazers

Watchers

chef-filebeat's Issues

HWRP does not restart filebeat service

When a change is flagged by the HWRP the filebeat service doesn't restart. To verify this I recorded the pids before a change run:

vagrant@default-ubuntu-1404:~$ ps awwux | grep [b]eat
root      2055  0.0  0.0   9392     0 ?        S    Jun30   0:00 /usr/bin/filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root      2062  0.0  1.5 261132  5884 ?        Sl   Jun30   0:19 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

Ran Chef changing action on the provider to :delete

-----> Starting Kitchen (v1.5.0)
-----> Converging <default-ubuntu-1404>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 4.2.1...
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
-----> Chef Omnibus installation detected (install only if missing)
       Transferring files to <default-ubuntu-1404>
       Starting Chef Client, version 12.11.18
       resolving cookbooks for run list: ["apt", "filebeat::default"]
       Synchronizing Cookbooks:
         - apt (4.0.1)         
         - yum (3.11.0)        
         - ms_dotnet (2.6.2)   
         - chef_handler (1.4.0)
         - filebeat (0.3.5)    
         - windows (1.44.1)    
         - compat_resource (12.10.6)
         - powershell (3.3.2)  
       Installing Cookbook Gems:
       Compiling Cookbooks...  
       [2016-07-01T15:54:15+00:00] WARN: Chef::Provider::AptRepository already exists!  Cannot create deprecation class for LWRP provider apt_repository from cookbook apt
       [2016-07-01T15:54:15+00:00] WARN: AptRepository already exists!  Deprecation class overwrites Custom resource apt_repository from cookbook apt
       [2016-07-01T15:54:15+00:00] WARN: Please use the powershell_script resource in Chef Client 11.6+
       Converging 18 resources 
       Recipe: apt::default    
         * file[/var/lib/apt/periodic/update-success-stamp] action nothing (skipped due to action :nothing)
         * apt_update[periodic] action periodic (up to date)
         * execute[apt-get update] action nothing (skipped due to action :nothing)
         * execute[apt-get autoremove] action nothing (skipped due to action :nothing)
         * execute[apt-get autoclean] action nothing (skipped due to action :nothing)
         * directory[/var/cache/local] action create (up to date)
         * directory[/var/cache/local/preseeding] action create (up to date)
         * template[/etc/apt/apt.conf.d/10recommends] action create (up to date)
         * apt_package[apt-transport-https] action install (up to date)
       Recipe: filebeat::install_package
         * apt_repository[beats] action add
           * remote_file[/tmp/kitchen/cache/GPG-KEY-elasticsearch] action create (up to date)
           * execute[apt-key add /tmp/kitchen/cache/GPG-KEY-elasticsearch] action run (skipped due to not_if)
           * execute[apt-cache gencaches] action nothing (skipped due to action :nothing)
           * apt_update[beats] action nothing (skipped due to action :nothing)
           * file[/etc/apt/sources.list.d/beats.list] action create (up to date)
            (up to date)       
         * apt_package[filebeat] action install (up to date)
       Recipe: filebeat::config
         * directory[/etc/filebeat/conf.d] action create (up to date)
         * file[/etc/filebeat/filebeat.yml] action create (up to date)
         * file[prospector-system_logs] action create (up to date)
         * powershell[install filebeat as service] action run (skipped due to only_if)
         * ruby_block[delay filebeat service start] action run
           - execute the ruby block delay filebeat service start
         * service[filebeat] action enable (up to date)
         * service[filebeat] action nothing (skipped due to action :nothing)
       Recipe: filebeat::default
         * filebeat_prospector[messages] action delete
         Recipe: <Dynamically Defined Resource>
           * file[prospector_messages] action delete
             - delete file /etc/filebeat/conf.d/prospector-messages.yml

       Recipe: filebeat::config
         * service[filebeat] action start (up to date)

       Running handlers:       
       Running handlers complete
       Chef Client finished, 3/26 resources updated in 07 seconds
       Finished converging <default-ubuntu-1404> (0m18.37s).
-----> Kitchen is finished. (0m18.84s)

Note that the file was deleted correctly but the service action was :start and marked as up to date. Per provider_prospector.rb#L64 I would expect the action to be restart.

Check the pids again. Note that they are the same signifying no restart took place:

vagrant@default-ubuntu-1404:~$ ps awwux | grep [b]eat
root      2055  0.0  0.0   9392     0 ?        S    Jun30   0:00 /usr/bin/filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root      2062  0.0  1.5 261132  5884 ?        Sl   Jun30   0:19 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

As I test I manually restarted filebeat and the pids do change, it is not a HUP.

vagrant@default-ubuntu-1404:~$ ps awwux | grep [b]eat
root      2055  0.0  0.0   9392     0 ?        S    Jun30   0:00 /usr/bin/filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root      2062  0.0  1.5 261132  5884 ?        Sl   Jun30   0:19 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
vagrant@default-ubuntu-1404:~$ sudo /etc/init.d/filebeat restart
 * Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat
   ...done.
vagrant@default-ubuntu-1404:~$ ps awwux | grep [b]eat
root      4450  0.0  0.0   9392   180 pts/0    S    15:54   0:00 /usr/bin/filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
root      4451  1.5  2.7 294968 10364 pts/0    Sl   15:54   0:00 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

I expect this is a wider bug than just the HWRP as the code in the HWRP looks fine, but I've only confirmed it via the provider.

node['filebeat']['prospectors'] resources clean up

when a prospector is deleted from node['filebeat']['prospectors'], it does not clean up .yml file for filebeat_prospector resource.

add specs for LWRP filebeat_prospector

TLS configuration for logstash output?

What would be the best way to add TLS authentication to the logstash output config section?

fix rubocop error

$ bundle exec rake lint
Starting Foodcritic linting...
Completed!
Running RuboCop...
Inspecting 12 files
....C.......
Offenses:
libraries/provider_prospector.rb:53:19: C: Don't use parentheses around a literal.
        t.content(({ 'filebeat' => { 'prospectors' => [content] } }).to_yaml)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12 files inspected, 1 offense detected
RuboCop failed!

Chef version 12.9.37 introduced (slightly) incompatible native apt_repository resource

The new apt_repository resource adds a default value for "distribution" which of course breaks the filebeat repo config because it doesn't need one. You can work around this by doing:

apt_repository 'your_repo' do
  distribution ''
end

See the issue I filed with Chef about it: chef/chef#4828
And the commit that introduced the problem: chef/chef@79548ac

filebeat is not restarted on upgrade

Hi, if upgrade filebeat with filebeat::install_package it is not restarted and change doesn't have an effect.
It happend that I regenerated filebeat configs first and only after it upgraded filebeat. Now I can restart filebeat only manually (as there is no recipe for it) or by changing filebeat config.

fix kitchen test

Question - How do you automate uploading the template?

The filebeat documentation instructs the user to upload an index template to elasticsearch

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html#filebeat-template

curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json

It doesn't appear that the cookbook supports doing that automatically. Is this a manual step?

I'm deploying elasticsearch into production behind a firewall where I don't have direct access to upload a template manually.

Maybe the cookbook could automate this with http_request.
https://docs.chef.io/resource_http_request.html

file beat Alpha (5.0)

I need to push my logs to kafka and for that I might need filebeat 5.0, though it is in alpha. But I am not sure about the repo and other settings and I am relying heavy on readymade cookbooks! Is there any cook available? Help is greatly appreciated.

missing attribute node['filebeat']['windows']['version_string']

Also, "filebeat-#{node['filebeat']['version']}-windows" can be used instead of forming attribute node['filebeat']['windows']['version_string'].

Can not add multiple "fields" via LWRP filebeat_prospector

Hello,

I can't add more than one fields in hash.
If write:

 fields = { "type" => "foo", "system" => "bar" }

there is no field section in config file at all.

If write

  fields 'type' => "foo"

everything is OK.

  fields 'type' => "foo" 
  fields 'system' => "bar"

in prospector config will be last value "bar"

ruby 2.1.8p440
Chef: 12.9.41

Thanks

LWRP filebeat_prospector Problem with backslash in regex by using multiline

I want to use the following regex within LWRP filebeat_prospector
['multiline']['pattern'] = '^\d{4}/\d+/\d+ \d+:\d+:\d+ ['
['multiline']['negate'] = true
['multiline']['match'] = 'after'

But I get on the server:
multiline: !ruby/hash:Chef::Node::ImmutableMash
pattern: "^\d{4}/\d+/\d+ \d+:\d+:\d+ ["
negate: true
match: after

and the error
syslog:May 10 12:30:33 ip-10-18-4-20 /usr/bin/filebeat[28838]: log.go:135: Stop Harvesting. Unexpected encoding line reader error: error parsing regexp: invalid escape sequence: \d

powershell failed to run due to space inside the path

Hi,
I'm running this on Windows 8 and powershell failed to run due to my filebeat base dir include a space in the path. (C:\Program Files)
There is a fix in Chef , see here :
chef/chef#4036
But if you can't upgrade your chef, here is a workaround
I have added a single quote in config.rb

code "'#{node['filebeat']['windows']['base_dir']}/filebeat-#{node['filebeat']['version']}-windows/install-service-filebeat.ps1'"

Plans for LWRP / Custom Resources

Great cookbook. Because the configuration is done through node attributes, this has the potential to create really big role/policy files. Think if you want to manage 20 different files, the json on the node/role attributes would become very large.

Are there any plans to add LWRP or custom resources so that users could create wrapper cookbooks?

The elasticsearch and logstash cookbooks are great examples of HWRP and LWRP cookbooks.

Here is the documentation for custom resources which replace LWRP and HWRP however they only work with chef 12.5 or newer

Unable to set action on resource provider

I'm attempting to use the action resource with cookbook 0.3.4 and it is failing:

filebeat_prospector 'thing'
  ...
  action :delete
end

    ================================================================================
    Error executing action `delete` on resource 'filebeat_prospector[thing]'
    ================================================================================

    NoMethodError
    -------------
    file[prospector_thing] (dynamically defined) had an error: NoMethodError: undefined method `to_sym' for [:delete]:Array

    Cookbook Trace:
    ---------------
    /var/chef/cache/cookbooks/filebeat/libraries/provider_prospector.rb:65:in `prospector_file'
    /var/chef/cache/cookbooks/filebeat/libraries/provider_prospector.rb:24:in `action_delete'

add yum_repository resource attribute metadata_expire

filebeat on windows

When I look at the filebeat service via "Services" I see it starting with -c pointing to the the installation dir.
However the config file will be copied to /etc/filebeat just like on Linux. So where should it be?

bump filebeat version to 1.0.0-rc1

Error executing action `start` on resource 'service[filebeat]'

Hi vkhatri,

I am getting the following error while I tried to test this cookbook on Vagrant with Chef-solo. I am using 'bento/ubuntu-14.04' box on Vagrant.

Error executing action start on resource 'service[filebeat]'

Even it fails to restart. So far I didn't mess with the file /etc/filebeat/filebeat.yml. I am new to this, I would glad if you assist me on this.

Version bump for latest changes?

@vkhatri I was wondering if you'll have a chance soon to do a version bump to 0.3.2 (or 0.4.0) to reflect the changes this past week and to upload it to the chef Supermarket? That would be great if that could be done soon, which would allow me to continue using your cookbook instead of forking my own copy.

Thank you!

use resource powershell instead of powershell_script

@coffeepac FYI, as per cookbook wiki, powershell_script resource is default resource powershell. It is available for Chef 11 and 12.

42>> powershell_script 'install filebeat as service' do
 43:    code "#{node['filebeat']['windows']['base_dir']}/#{node['filebeat']['windows']['version_string']}/install-service-filebeat.ps1"
 44:    only_if { node['platform'] == 'windows' }
 45:  end

Updating cookbook to use resource powershell. As, i do not have Windows platform to test it out and cookbook is broken at the moment, kindly submit PR if you find some issue with new changes.

https://github.com/chef-cookbooks/powershell#resourceprovider

add example for output configuration

Use multiple prospectors based on node attributes.

I am new to chef and used your baseline master recipe.It works like a gem. What I am trying to achieve next is to define prospectors for filebeats based on condition. If attribute 'a' is present then we have following a set of paths and if 'b' is present then we use different set of prospectors. My questions is two fold:

In your readme, you specify that we can add prospectors using node attributes (https://github.com/vkhatri/chef-filebeat#how-to-add-filebeat-output-via-node-attribute). Where do you define this file ? An example file or sample will be good.

I am using case conditions and define individual attributes as shown for each log types (Please refer following example). ?
case node['a']
when 'sys-log'
default['filebeat']['config']['filebeat']['prospectors']['paths'] = ["/var/log/messages",
"/var/log/syslog"]
default['filebeat']['config']['filebeat']['prospectors']['paths']['type'] = 'log'
default['filebeat']['config']['filebeat']['prospectors']['paths']['fields']['type'] = 'syslog'
when 'apache-log'
default['filebeat']['config']['filebeat']['prospectors']['paths'] = ["/var/log/localhost",
"/var/log/apache"]
default['filebeat']['config']['filebeat']['prospectors']['paths']['type'] = 'log'
default['filebeat']['config']['filebeat']['prospectors']['paths']['fields']['type'] = 'apache'
end

UPDATE: I was able to achieve the above use case using conditions but now the problem is that in my prospector-*.yml the formatting is missing char '-'. Any comments on how i can add using node attributes ?

Actual Output:
filebeat:
prospectors:
paths:
- C:\apps\logs\localhost_access_log*.txt
input_type: log
fields_under_root: true
fields:
logtype: access
document_type: access

Expected:
filebeat:
prospectors:
-
paths:
- C:\apps\logs\localhost_access_log*.txt
input_type: log
fields_under_root: true
fields:
logtype: access
document_type: access

Thanks

config_dir attribute not getting set

Further information is cookbook version 0.3.7, Chef 11.8.2.
Since the latest cookbook was pulled in, the config_dir attribute is not getting set properly and now /etc/filebeat/filebeat.yml ends up with a blank field for config directory.
Will post more if I can figure out further why this doesn't get set properly after the move to using attributes.rb.

enabled option for outputs was deprecated before Filebeat 1.0 release

Based on - https://github.com/elastic/filebeat/issues/317

I realized you may want to code in for backwards compatibility - but we should get removed from the README/docs. If it isn't in a branch already I am happy to fork and submit a PR to cleanup any docs references - and remove altogether if you're ok with that too.

remove dpkg install options in favor of repository install

'enabled' attributes are not properly respected by config.rb

The attributes default['filebeat']['config']['output']['elasticsearch']['enabled'] and default['filebeat']['config']['output']['logstash']['enabled'] result in sections which are rendered to the config file. In the case of elasticsearch, it attempts to load the plugin and if no hosts are defined fails to start.

The expected behavior is not to render any of the output sections in which the enabled value is set to false.

Error executing action `restart` on resource 'service[filebeat]'

Hi am facing issue when i use runit to start filebeat. After setting the node "default['filebeat']['service']['init_style'] = 'runit'", i get the error when use this cookbook at first

   Error executing action `restart` on resource 'service[filebeat]'
   ================================================================================

   Mixlib::ShellOut::ShellCommandFailed
   ------------------------------------
   Expected process to exit with [0], but received '1'
   ---- Begin output of /sbin/sv restart /etc/service/filebeat ----
   STDOUT: timeout: down: /etc/service/filebeat: 1s, normally up, want up
   STDERR: 
   ---- End output of /sbin/sv restart /etc/service/filebeat ----
   Ran /sbin/sv restart /etc/service/filebeat returned 1

But when i do the Kitchen converge the next time, it works fine.. Need some troubleshooting steps

Adding fields_under_root to prospectors

Hi, I am trying to add the fields_under_root: true field to my prospectors using roles. Here's an example role i created.

{
  "filebeat": {
    "prospectors": {
      "cisco": {
        "filebeat": {
          "prospectors": [
            {
              "paths": [
                "/var/log/172.x.x.x/*.log*",
              ],
              "document_type": "mylog",
              "fields": {
                "type": "log",
                "index_type": "cisco",
              },
              "fields_under_root": "true"
            }
          ]
        }
      },
  }
 }
}

I want the index_type field to be a top-level field so that i can use that as my index. But this role config generates the fields_under_ root field as fields_under_root: 'true' instead of fields_under_root: true
The quotes for the Boolean value is not supported my filebeat config and the fields go as fields.index_type. When i manually remove the quotes for "true" they go as top-level fields as index_type. I know its just Json to YAML conversion that adds the quotes but is there a way to fix this?

Edit: I am using the 0.4.2 cookbook version and filebeat 1.3 .1 on ubuntu 14.04 with LS 2.4.x and ES 2.4.x

Upgrading Filebeat in-place results in config collision

Seems the filebeat package wants to overwrite the config being created by the cookbook, resulting in a whole lotta pain:

* apt_package[filebeat] action install
  ================================================================================
  Error executing action `install` on resource 'apt_package[filebeat]'
  ================================================================================
  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Expected process to exit with [0], but received '100'
  ---- Begin output of apt-get -q -y install filebeat=1.1.2 ----
  STDOUT: Reading package lists...
  Building dependency tree...
  Reading state information...
  You might want to run 'apt-get -f install' to correct these:
  The following packages have unmet dependencies:
   linux-headers-3.13.0-83-generic : Depends: linux-headers-3.13.0-83 but it is not going to be installed
  STDERR: E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
  ---- End output of apt-get -q -y install filebeat=1.1.2 ----
  Ran apt-get -q -y install filebeat=1.1.2 returned 100
  Resource Declaration:
  ---------------------
  # In /var/chef/cache/cookbooks/filebeat/recipes/install_package.rb
   42: package 'filebeat' do
   43:   version node['platform_family'] == 'rhel' ? node['filebeat']['version'] + '-1' : node['filebeat']['version']
   44:   notifies :restart, 'service[filebeat]'
   45: end
  Compiled Resource:
  ------------------
  # Declared in /var/chef/cache/cookbooks/filebeat/recipes/install_package.rb:42:in `from_file'
  apt_package("filebeat") do
    action [:install]
    retries 0
    retry_delay 2
    default_guard_interpreter :default
    package_name "filebeat"
    version "1.1.2"
    declared_type :package
    cookbook_name "filebeat"
    recipe_name "install_package"
  end

The elasticsearch cookbook solves this by allowing an operator to pass in options, see lines 61-71 of the elasticsearch_install provider, and something similar here would be great.

Is there any setting to enable proxy url for the pulling the packages ?

release new version 0.2.8

How to escape quotes in LWRP's multiline patten?

I need to produce the following option in prospector (note the quotes):

multiline:
  pattern: '^<[0-9]+>'
  negate: true
  match: after

So I added the following option:

filebeat_prospector 'test' do
  ...
  multiline 'pattern' => "'^<[0-9]+>'", 'negate' => true, 'match' => 'after'
  ...
end

But after chef-client run the resulting option in filebeat prospector config file looks like this: pattern: ! '''^<[0-9]+>'''. I also tried to escape quotes likes this: pattern: '\'^<[0-9]+>'\' in LWRP, this produces the same result in config file.

[Ubuntu] Using distribution name in sources file causes 404

I think this might be a recent server/infra change by Elastic.

If the distribution name is used in the sources file the repository update fails with a 404.

This fails:
deb "https://packages.elastic.co/beats/apt" trusty stable main

This works:
deb "https://packages.elastic.co/beats/apt" stable main

Error:

Err https://packages.elastic.co trusty/stable amd64 Packages
  HttpError404
Err https://packages.elastic.co trusty/main amd64 Packages
  HttpError404
Ign https://packages.elastic.co trusty/main Translation-en_US
Ign https://packages.elastic.co trusty/main Translation-en
Ign https://packages.elastic.co trusty/stable Translation-en_US
Ign https://packages.elastic.co trusty/stable Translation-en
W: Failed to fetch https://packages.elastic.co/beats/apt/dists/trusty/stable/binary-amd64/Packages  HttpError404

W: Failed to fetch https://packages.elastic.co/beats/apt/dists/trusty/main/binary-amd64/Packages  HttpError404

E: Some index files failed to download. They have been ignored, or old ones used instead.

filebeat cookbook does not work with latest Powershell cookbook

It looks like the latest powershell cookbook release removed the powershell resource. As a result this cookbook fails with the below. A temporary fix is to lock the powershell version to the latest 3.X.X release.

NoMethodError
  -------------
  No resource or method named `powershell' for `Chef::Recipe "config"'

  Cookbook Trace:
  ---------------
    /var/chef/cookbooks/filebeat/recipes/config.rb:46:in `from_file'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:336:in `load_recipe'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:292:in `block in include_recipe'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:291:in `each'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:291:in `include_recipe'
    /var/chef/cookbooks/filebeat/recipes/default.rb:28:in `from_file'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:336:in `load_recipe'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:292:in `block in include_recipe'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:291:in `each'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:291:in `include_recipe'
    /var/chef/cookbooks/wrap_base/recipes/_logging.rb:104:in `from_file'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:336:in `load_recipe'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:292:in `block in include_recipe'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:291:in `each'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:291:in `include_recipe'
    /var/chef/cookbooks/wrap_base/recipes/default.rb:34:in `from_file'
    /var/chef/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:336:in `load_recipe'

  Relevant File Content:
  ----------------------
  /var/chef/cookbooks/filebeat/recipes/config.rb:

   39:    file "prospector-#{prospector}" do
   40:      path ::File.join(node['filebeat']['prospectors_dir'], "prospector-#{prospector}.yml")
   41:      content JSON.parse(configuration.to_json).to_yaml.lines.to_a[1..-1].join
   42:      notifies :restart, 'service[filebeat]' if node['filebeat']['notify_restart'] && !node['filebeat']['disable_service']
   43:    end
   44:  end
   45:
   46>> powershell 'install filebeat as service' do
   47:    code "& '#{node['filebeat']['windows']['base_dir']}/filebeat-#{node['filebeat']['version']}-windows/install-service-filebeat.ps1'"
   48:    only_if { node['platform'] == 'windows' }
   49:  end

add repository

Filebeat version attribute is not wrapper friendly

The package file name uses a derived attribute, as result the only way to set the version and have it respected is to use a precedence level higher than default, along with something like node.from_file(run_context.resolve_attribute(*parse_attribute_file_spec("filebeat::default")))

For this reason, derived attributes are not recommended.

See https://coderanger.net/derived-attributes/ for additional details.

Release cookbook version 0.3.3

Prospector multiline changed format

Filebeat 1.x used nested fields, while Filebeat 5.x switched to dot delimited fields. See https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#multiline

package not installed before 'service filebeat start'

getting the following chef errors when running filebeat::config

 * ruby_block[delay filebeat service start] action run[2016-01-25T22:37:39+00:00] INFO: Processing ruby_block[delay filebeat service start] action run (filebeat::config line 45)
[2016-01-25T22:37:39+00:00] INFO: ruby_block[delay filebeat service start] called

    - execute the ruby block delay filebeat service start
  * service[filebeat] action enable[2016-01-25T22:37:39+00:00] INFO: Processing service[filebeat] action enable (filebeat::config line 54)

    * service[filebeat]: unable to locate the init.d script!
    ================================================================================
    Error executing action `enable` on resource 'service[filebeat]'
    ================================================================================

    Chef::Exceptions::Service
    -------------------------
    service[filebeat]: unable to locate the init.d script!

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/filebeat/recipes/config.rb

     54: service 'filebeat' do
     55:   supports :status => true, :restart => true
     56:   action service_action
     57: end

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/filebeat/recipes/config.rb:54:in `from_file'

    service("filebeat") do
      action [:enable, :nothing]
      supports {:status=>true, :restart=>true}
      retries 0
      retry_delay 2
      guard_interpreter :default
      service_name "filebeat"
      pattern "filebeat"
      cookbook_name "filebeat"
      recipe_name "config"
    end

[2016-01-25T22:37:39+00:00] INFO: Running queued delayed notifications before re-raising exception
[2016-01-25T22:37:39+00:00] INFO: file[/etc/filebeat/filebeat.yml] sending restart action to service[filebeat] (delayed)
  * service[filebeat] action restart[2016-01-25T22:37:39+00:00] INFO: Processing service[filebeat] action restart (filebeat::config line 54)

    * service[filebeat]: unable to locate the init.d script!
    ================================================================================
    Error executing action `restart` on resource 'service[filebeat]'
    ================================================================================

    Chef::Exceptions::Service
    -------------------------
    service[filebeat]: unable to locate the init.d script!

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/filebeat/recipes/config.rb

     54: service 'filebeat' do
     55:   supports :status => true, :restart => true
     56:   action service_action
     57: end

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/filebeat/recipes/config.rb:54:in `from_file'

    service("filebeat") do
      action [:enable, :nothing]
      supports {:status=>true, :restart=>true}
      retries 0
      retry_delay 2
      guard_interpreter :default
      service_name "filebeat"
      pattern "filebeat"
      cookbook_name "filebeat"
      recipe_name "config"
    end

[2016-01-25T22:37:39+00:00] INFO: ruby_block[delay filebeat service start] sending start action to service[filebeat] (delayed)
  * service[filebeat] action start[2016-01-25T22:37:39+00:00] INFO: Processing service[filebeat] action start (filebeat::config line 54)

    * service[filebeat]: unable to locate the init.d script!
    ================================================================================
    Error executing action `start` on resource 'service[filebeat]'
    ================================================================================

    Chef::Exceptions::Service
    -------------------------
    service[filebeat]: unable to locate the init.d script!

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/filebeat/recipes/config.rb

     54: service 'filebeat' do
     55:   supports :status => true, :restart => true
     56:   action service_action
     57: end

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/filebeat/recipes/config.rb:54:in `from_file'

    service("filebeat") do
      action [:enable, :nothing]
      supports {:status=>true, :restart=>true}
      retries 0
      retry_delay 2
      guard_interpreter :default
      service_name "filebeat"
      pattern "filebeat"
      cookbook_name "filebeat"
      recipe_name "config"
    end

Cookbook does not support 32 bit platforms

The default behavior for the download URL is to assume 64 bit systems. This means that one needs to either define node['filebeat']['package_url'] for 32 bit systems exclude it from the run list on those systems.

https://github.com/vkhatri/chef-filebeat/blob/master/recipes/install.rb#L22-L23

The expected behavior is for the cookbook to properly identify the system architecture and download the correct package.

Adding Prospectors Using JSON not working.

I tried adding prospectors using node attributes in json format. I used the following with chef solo and this doesnot work. Can you assist and point me out what could be the problem here
{ "filebeat": { "prospectors": { "system_logs": { "filebeat": { "prospectors": [ { "paths": [ "/var/log/messages", "/var/log/syslog" ], "type": "log", "fields": { "type": "system_logs" } } ] } }, "secure_logs": { "filebeat": { "prospectors": [ { "paths": [ "/var/log/secure", "/var/log/auth.log" ], "type": "log", "fields": { "type": "secure_logs" } } ] } }, "apache_logs": { "filebeat": { "prospectors": [ { "paths": [ "/var/log/apache/*.log" ], "type": "log", "ignore_older": "24h", "scan_frequency": "15s", "harvester_buffer_size": 16384, "fields": { "type": "apache_logs" } } ] } } } }, "filebeat": { "config": { "output": { "logstash": { "enabled": true, "hosts": [ "127.0.0.1:5044" ] } } } }, "run_list": [ "recipe[filebeat::default]" ] }

NOTE: I used the above config both with and without the "default_attribute" wrapper and it doesnot work. Whereas, the logstash config passed in works.

yum package unexpectedly updating to filebeat v1.3.0

when i leave the version of the cookbook unpinned, do not alter any attributes for the version it installs v1.3.0 of the yum package and subsequent chef runs fail due to a candidate version mismatch. workarounds included pinning the cookbook version or pinning the filbeat version on our own end.

update to v1.0.1

fix chefspec warning

/Users/vkhatri/.rvm/rubies/ruby-2.2.2-rubies/bin/ruby -I/Users/vkhatri/.rvm/gems/ruby-2.2.2-rubies/gems/rspec-core-3.5.3/lib:/Users/vkhatri/.rvm/gems/ruby-2.2.2-rubies/gems/rspec-support-3.5.0/lib /Users/vkhatri/.rvm/gems/ruby-2.2.2-rubies/gems/rspec-core-3.5.3/exe/rspec --pattern spec/{,/*/}/*_spec.rb
[2016-09-09T16:25:29+05:30] WARN: Chef::Provider::AptRepository already exists! Cannot create deprecation class for LWRP provider apt_repository from cookbook apt
[2016-09-09T16:25:29+05:30] WARN: AptRepository already exists! Deprecation class overwrites Custom resource apt_repository from cookbook apt

vkhatri / chef-filebeat Goto Github PK

chef-filebeat's People

Contributors

Stargazers

Watchers

Forkers

chef-filebeat's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs