datadog / chef-datadog

Chef cookbook for Datadog Agent & Integrations

Datadog Chef Cookbook

The Datadog Chef recipes are used to deploy Datadog's components and configuration automatically. The cookbook includes support for:

• Datadog Agent v7.x (default)
• Datadog Agent v6.x
• Datadog Agent v5.x

Note: This page may discuss features that are not available for your selected version. Check the README of the git tag or gem version for your version's documentation.

Setup

Requirements

The Datadog Chef cookbook is compatible with chef-client >= 12.7. If you need support for Chef < 12.7, use a release 2.x of the cookbook. See the CHANGELOG for more info.

Platforms

The following platforms are supported:

• AlmaLinux (requires Chef 16 >= 16.10.8 or Chef >= 17.0.69)
• Amazon Linux
• CentOS
• Debian
• RedHat (RHEL 8 requires Chef >= 15)
• Rocky (requires Chef 16 >= 16.17.4 or Chef >= 17.1.35)
• Scientific Linux
• Ubuntu
• Windows
• SUSE (requires Chef >= 13.3)

Cookbooks

The following Opscode cookbooks are dependencies:

• apt
• chef_handler
• yum

Note: apt cookbook v7.1+ is needed to install the Agent on Debian 9+.

Chef

Chef 13 users: With Chef 13 and chef_handler 1.x, you may have trouble using the dd-handler recipe. The known workaround is to update your dependency to chef_handler >= 2.1.

Installation

1. Add the cookbook to your Chef server with Berkshelf or Knife:

   # Berksfile
   cookbook 'datadog', '~> 4.0'
   

   # Knife
   knife cookbook site install datadog

2. Set the Datadog-specific attributes in a role, environment, or another recipe:

   node.default['datadog']['api_key'] = "<YOUR_DD_API_KEY>"
   
   node.default['datadog']['application_key'] = "<YOUR_DD_APP_KEY>"
   

3. Upload the updated cookbook to your Chef server:

   berks upload
   # or
   knife cookbook upload datadog

4. After uploading, add the cookbook to your node's run_list or role:

   "run_list": [
     "recipe[datadog::dd-agent]"
   ]
   

5. Wait for the next scheduled chef-client run or trigger it manually.

Datadog attributes

The following methods are available for adding your Datadog API and application keys:

• As node attributes with an environment or role.
• As node attributes by declaring the keys in another cookbook at a higher precedence level.
• In the node run_state by setting node.run_state['datadog']['api_key'] in another cookbook preceding Datadog's recipes in the run_list. This approach does not store the credential in clear text on the Chef Server.

Note: When using the run state to store your API and application keys, set them at compile time before datadog::dd-handler in the run list.

Extra configuration

To add additional elements to the Agent configuration file (typically datadog.yaml) that are not directly available as attributes of the cookbook, use the node['datadog']['extra_config'] attribute. This is a hash attribute, which is marshaled into the configuration file accordingly.

Examples

The following code sets the field secret_backend_command in the configuration file datadog.yaml:

 default_attributes(
   'datadog' => {
     'extra_config' => {
       'secret_backend_command' => '/sbin/local-secrets'
     }
   }
 )

The secret_backend_command can also be set using:

default['datadog']['extra_config']['secret_backend_command'] = '/sbin/local-secrets'

For nested attributes, use object syntax. The following code sets the field logs_config in the configuration file datadog.yaml:

default['datadog']['extra_config']['logs_config'] = { 'use_port_443' => true }

AWS OpsWorks Chef deployment

Follow the steps below to deploy the Datadog Agent with Chef on AWS OpsWorks:

1. Add Chef custom JSON:

{"datadog":{"agent_major_version": 7, "api_key": "<API_KEY>", "application_key": "<APP_KEY>"}}

2. Include the recipe in the install-lifecycle recipe:

include_recipe '::dd-agent'

Integrations

Enable Agent integrations by including the recipe and configuration details in your role’s run-list and attributes. Note: You can use the datadog_monitor resource for enabling Agent integrations without a recipe.

Associate your recipes with the desired roles, for example role:chef-client should contain datadog::dd-handler and role:base should start the Agent with datadog::dd-agent. Below is an example role with the dd-handler, dd-agent, and mongo recipes:

name 'example'
description 'Example role using DataDog'

default_attributes(
  'datadog' => {
    'agent_major_version' => 7,
    'api_key' => '<YOUR_DD_API_KEY>',
    'application_key' => '<YOUR_DD_APP_KEY>',
    'mongo' => {
      'instances' => [
        {'host' => 'localhost', 'port' => '27017'}
      ]
    }
  }
)

run_list %w(
  recipe[datadog::dd-agent]
  recipe[datadog::dd-handler]
  recipe[datadog::mongo]
)

Note: data_bags are not used in this recipe because it is unlikely to have multiple API keys with only one application key.

Versions

By default, the current major version of this cookbook installs Agent v7. The following attributes are available to control the Agent version installed:

Parameter	Description
agent_major_version	Pin the major version of the Agent to 5, 6, or 7 (default).
agent_version	Pin a specific Agent version (recommended).
agent_package_action	(Linux only) Defaults to 'install' (recommended), 'upgrade' to get automatic Agent updates (not recommended, use the default and change the pinned agent_version to upgrade).
agent_flavor	(Linux only) Defaults to 'datadog-agent' to install the datadog-agent, can be set to 'datadog-iot-agent' to install the IOT agent.

See the sample attributes/default.rb for your cookbook version for all available attributes.

Upgrade

Some attribute names have changed from version 3.x to 4.x of the cookbook. Use this reference table to update your configuration:

Action	Cookbook 3.x	Cookbook 4.x
Install Agent 7.x	Not supported	'agent_major_version' => 7
Install Agent 6.x	'agent6' => true	'agent_major_version' => 6
Install Agent 5.x	'agent6' => false	'agent_major_version' => 5
Pin agent version	'agent_version' or 'agent6_version'	'agent_version' for all versions
Change package_action	'agent_package_action' or 'agent6_package_action'	'agent_package_action' for all versions
Change APT repo URL	'aptrepo' or 'agent6_aptrepo'	'aptrepo' for all versions
Change APT repo distribution	'aptrepo_dist' or 'agent6_aptrepo_dist'	'aptrepo_dist' for all versions
Change YUM repo	'yumrepo' or 'agent6_yumrepo'	'yumrepo' for all versions
Change SUSE repo	'yumrepo_suse' or 'agent6_yumrepo_suse'	'yumrepo_suse' for all versions

Use one of the following methods to upgrade from Agent v6 to v7:

• Set agent_major_version to 7, agent_package_action to install, and pin a specific v7 version as agent_version (recommended).
• Set agent_major_version to 7 and agent_package_action to upgrade.

The following example upgrades from Agent v6 to v7. The same applies if you are upgrading from Agent v5 to v6.

default_attributes(
  'datadog' => {
    'agent_major_version' => 7,
    'agent_version' => '7.25.1',
    'agent_package_action' => 'install',
  }
)

Downgrade

To downgrade the Agent version, set the 'agent_major_version', 'agent_version', and 'agent_allow_downgrade'.

The following example downgrades to Agent v6. The same applies if you are downgrading to Agent v5.

  default_attributes(
    'datadog' => {
      'agent_major_version' => 6,
      'agent_version' => '6.10.0',
      'agent_allow_downgrade' => true
    }
  )

Uninstall

To uninstall the Agent, remove the dd-agent recipe and add the remove-dd-agent recipe with no attributes.

Custom Agent repository

To use an Agent from a custom repository, you can set the aptrepo option.

By default, this option is equal to [signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] apt.datadoghq.com. If a custom value is set, another signed-by keyring can also be set [signed-by=custom-repo-keyring-path] custom-repo.

The example below uses the staging repository:

  default_attributes(
    'datadog' => {
      'aptrepo' => '[signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] apt.datad0g.com',
    }
  }

Recipes

Access the Datadog Chef recipes on GitHub.

Default

The default recipe is a placeholder.

Agent

The dd-agent recipe installs the Datadog Agent on the target system, sets your Datadog API key, and starts the service to report on local system metrics.

Note: Windows users upgrading the Agent from versions <= 5.10.1 to >= 5.12.0, set the windows_agent_use_exe attribute to true. For more details, see the dd-agent wiki.

Handler

The dd-handler recipe installs the chef-handler-datadog gem and invokes the handler at the end of a Chef run to report the details to the news feed.

DogStatsD

To install a language-specific library that interacts with DogStatsD:

• Ruby: dogstatsd-ruby recipe
• Python: Add a dependency on the poise-python cookbook to your custom/wrapper cookbook, and use the resource below. For more details, see the poise-python repository.

  python_package 'dogstatsd-python' # assumes python and pip are installed

Tracing

To install a language-specific library for application tracing (APM):

• Ruby: ddtrace-ruby recipe
• Python: Add a dependency on the poise-python cookbook to your custom/wrapper cookbook, and use the resource below. For more details, see the poise-python repository.

  python_package 'ddtrace' # assumes python and pip are installed

Integrations

There are many recipes to assist you with deploying Agent integration configuration files and dependencies.

System-probe

The system-probe recipe is automatically included by default. It writes the system-probe.yaml file. This behavior can be disabled by setting node['datadog']['system_probe']['manage_config'] to false.

To enable Network Performance Monitoring (NPM) in system-probe.yaml, set node['datadog']['system_probe']['network_enabled'] to true.

To enable Universal Service Monitoring (USM) in system-probe.yaml, set node['datadog']['system_probe']['service_monitoring_enabled'] to true.

Note for Windows users: NPM is supported on Windows with Agent v6.27+ and v7.27+. It ships as an optional component that is only installed if node['datadog']['system_probe']['network_enabled'] is set to true when the Agent is installed or upgraded. Because of this, existing installations might need to do an uninstall and reinstall of the Agent once to install the NPM component, unless the Agent is upgraded at the same time.

Resources

Integrations without recipes

Use the datadog_monitor resource for enabling Agent integrations without a recipe.

Actions

• :add: (default) Enables the integration by setting up the configuration file, adding the correct permissions to the file, and restarting the Agent.
• :remove: Disables an integration.

Syntax

datadog_monitor 'name' do
  init_config                       Hash # default value: {}
  instances                         Array # default value: []
  logs                              Array # default value: []
  use_integration_template          true, false # default value: false
  action                            Symbol # defaults to :add
end

Properties

Property	Description
'name'	The name of the Agent integration to configure and enable.
instances	The fields used to fill values under the instances section in the integration configuration file.
init_config	The fields used to fill values under the the init_config section in the integration configuration file.
logs	The fields used to fill values under the the logs section in the integration configuration file.
use_integration_template	Set to true (recommended) to use the default template, which writes the values of instances, init_config, and logs in the YAML under their respective keys. This defaults to false for backward compatibility, but may default to true in a future major version of the cookbook.

Example

This example enables the ElasticSearch integration by using the datadog_monitor resource. It provides the instance configuration (in this case: the URL to connect to ElasticSearch) and sets the use_integration_template flag to use the default configuration template. Also, it notifies the service[datadog-agent] resource to restart the Agent.

Note: The Agent installation must be above this recipe in the run list.

include_recipe '::dd-agent'

datadog_monitor 'elastic' do
  instances  [{'url' => 'http://localhost:9200'}]
  use_integration_template true
  notifies :restart, 'service[datadog-agent]' if node['datadog']['agent_start']
end

See the Datadog integration Chef recipes for additional examples.

Integration versions

To install a specific version of a Datadog integration, use the datadog_integration resource.

Actions

• :install: (default) Installs an integration with the specified version.
• :remove: Removes an integration.

Syntax

datadog_integration 'name' do
  version                      String         # version to install for :install action
  action                       Symbol         # defaults to :install
  third_party                  [true, false]  # defaults to :false
end

Properties

• 'name': The name of the Agent integration to install, for example: datadog-apache.
• version: The version of the integration to install (only required with the :install action).
• third_party: Set to false if installing a Datadog integration, true otherwise. Available for Datadog Agents version 6.21/7.21 and higher only.

Example

This example installs version 1.11.0 of the ElasticSearch integration by using the datadog_integration resource.

Note: The Agent installation must be above this recipe in the run list.

include_recipe '::dd-agent'

datadog_integration 'datadog-elastic' do
  version '1.11.0'
end

To get the available versions of the integrations, see the integration-specific CHANGELOG.md in the integrations-core repository.

Note: For Chef Windows users, the chef-client must have read access to the datadog.yaml file when the datadog-agent binary available on the node is used by this resource.

Development

Dockerized environment

To build a Docker environment with which to run kitchen tests, use the files under docker_test_env:

cd docker_test_env
docker build -t chef-datadog-test-env .

To run the container use:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock chef-datadog-test-env

Then attach a console to the container or use the VS Code remote-container feature to develop inside the container.

To run kitchen-docker tests from within the container:

# Note: Also set KITCHEN_DOCKER_HOSTNAME=host.docker.internal if on MacOS or Windows
# Run this under a login shell (otherwise `bundle` won't be found)
KITCHEN_LOCAL_YAML=kitchen.docker.yml bundle exec rake circle

chef-datadog's Issues

add `action` variable to package installation

Default should be :install, but can be toggled to :upgrade if the desire is to be up-to-date with the latest released Agent.

Add recipe for source install

dd-handler recipe incompatible with non-omnibus .deb chef-installs.

The datadog:dd-handler recipe will trigger a rubygems provided install of chef to be installed alongside the .deb installed chef.

In our case, this has silently upgraded us from chef 10.18.2 to 10.24.0.

With the non-omnibus chef packages1, the "chef_gem" resource installs gems system wide, with no difference to the standard "gem" resource.

Since chef was not installed via rubygems, and "chef-handler-datadog" depends on "chef", chef_gem "chef-handler-datadog" triggers the upgrade.

Ideally the datadog:dd-handler recipe should be able to install a .deb version of the handler. This would A) prevent this issue and B) allow us to keep to our policy of not installing any unpackaged software.

Replace execute with package provider

As seen here using the execute is probably not the best way to solve this, since the package provider does indeed provide an EasyInstall method.

network: passing multiple instances to network monitor increases counts

Currently, the method for passing a network config matches all others, however this adds to the instances array.
This should not map to the same, rather only a single entry can work for monitoring the network state of an instance.

a role:

default_attributes(
  'datadog' => {
    'network' => {
      'instances' => [
        {
          'collect_connection_state' => 'true',
          'excluded_interfaces' => ['lo', 'lo0']
        }
      ]
    }
  },
)

Along with the default attributes file network.rb will result in the following yaml:

init_config:

instances:
 - collect_connection_state: false
   excluded_interfaces:
     - lo
     - lo0
 - collect_connection_state: true
   excluded_interfaces:
     - lo
     - lo0

Please add "sock" to mysql template

Hey,

an important parameter is missing in the mysql Template templates/default/mysql.yaml.erb

It's called sock and needed to allow the datadog agent to connect to mysql through a local socket file.

Add tags to datadog.conf.erb, make them attributes

So that tags can be controlled directly from the chef role.

default kafka.yaml doesnt seem to work

Here's the yaml file created by the cookbook. It seems to be emitting system metrics but not jmx metrics.. Any suggestions"

instances:
  - host: localhost
    port: 9920

init_config:
  is_jmx: true

  # Metrics collected by this check. You should not have to modify this.
  conf:
    #
    # Aggregate cluster stats
    #
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesOutPerSec"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.net.bytes_out
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.net.bytes_in
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="BrokerTopicMetrics",name="AllTopicsMessagesInPerSec"'
      attribute:
        MeanRate:
          metric_type: gauge
          alias: kafka.messages_in

    #
    # Request timings
    #
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="BrokerTopicMetrics",name="AllTopicsFailedFetchRequestsPerSec"'
      attribute:
        MeanRate:
          metric_type: gauge
          alias: kafka.request.fetch.failed
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="BrokerTopicMetrics",name="AllTopicsFailedProduceRequestsPerSec"'
      attribute:
        MeanRate:
          metric_type: gauge
          alias: kafka.request.produce.failed
    - include:
      domain: '"kafka.network"'
      bean: '"kafka.network":type="RequestMetrics",name="Produce-TotalTimeMs"'
      attribute:
        Mean:
          metric_type: counter
          alias: kafka.request.produce.time.avg
        99thPercentile:
          metric_type: counter
          alias: kafka.request.produce.time.99percentile
    - include:
      domain: '"kafka.network"'
      bean: '"kafka.network":type="RequestMetrics",name="Fetch-TotalTimeMs"'
      attribute:
        Mean:
          metric_type: counter
          alias: kafka.request.fetch.time.avg
        99thPercentile:
          metric_type: counter
          alias: kafka.request.fetch.time.99percentile
    - include:
      domain: '"kafka.network"'
      bean: '"kafka.network":type="RequestMetrics",name="UpdateMetadata-TotalTimeMs"'
      attribute:
        Mean:
          metric_type: counter
          alias: kafka.request.update_metadata.time.avg
        99thPercentile:
          metric_type: counter
          alias: kafka.request.update_metadata.time.99percentile
    - include:
      domain: '"kafka.network"'
      bean: '"kafka.network":type="RequestMetrics",name="Metadata-TotalTimeMs"'
      attribute:
        Mean:
          metric_type: counter
          alias: kafka.request.metadata.time.avg
        99thPercentile:
          metric_type: counter
          alias: kafka.request.metadata.time.99percentile
    - include:
      domain: '"kafka.network"'
      bean: '"kafka.network":type="RequestMetrics",name="Offsets-TotalTimeMs"'
      attribute:
        Mean:
          metric_type: counter
          alias: kafka.request.offsets.time.avg
        99thPercentile:
          metric_type: counter
          alias: kafka.request.offsets.time.99percentile

    #
    # Replication stats
    #
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="ReplicaManager",name="ISRShrinksPerSec"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.replication.isr_shrinks
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="ReplicaManager",name="ISRExpandsPerSec"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.replication.isr_expands
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="ControllerStats",name="LeaderElectionRateAndTimeMs"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.replication.leader_elections
    - include:
      domain: '"kafka.server"'
      bean: '"kafka.server":type="ControllerStats",name="UncleanLeaderElectionsPerSec"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.replication.unclean_leader_elections

    #
    # Log flush stats
    #
    - include:
      domain: '"kafka.log"'
      bean: '"kafka.log":type="LogFlushStats",name="LogFlushRateAndTimeMs"'
      attribute:
        MeanRate:
          metric_type: counter
          alias: kafka.log.flush_rate

Agent install not working via chef-solo (apt-1.8.0)

Reported by 2 users

[2013-01-26T14:22:11+00:00] INFO: Processing apt_repository[datadog] action add (datadog::dd-agent line 24)

================================================================================
Error executing action `add` on resource 'apt_repository[datadog]'
================================================================================

RuntimeError
------------
The repository file to create is nil, cannot continue.

Cookbook Trace:
---------------
/srv/chef/file_store/cookbooks/apt/providers/repository.rb:123:in `class_from_file'

Resource Declaration:

---------------------
# In /srv/chef/file_store/cookbooks/datadog/recipes/dd-agent.rb

 23: 
 24:   apt_repository 'datadog' do
 25:     keyserver 'keyserver.ubuntu.com'
 26:     key 'C7A7DA52'
 27:     uri node['datadog']['aptrepo']
 28:     distribution "unstable"
 29:     components ["main"]
 30:     action :add
 31:   end
 32: 

Compiled Resource:
------------------
# Declared in /srv/chef/file_store/cookbooks/datadog/recipes/dd-agent.rb:24:in `from_file'

apt_repository("datadog") do
  keyserver "keyserver.ubuntu.com"
  cookbook_name "datadog"
  retry_delay 2
  components ["main"]
  uri "http://apt.datadoghq.com"
  retries 0
  distribution "unstable"
  key "C7A7DA52"
  recipe_name "dd-agent"
  action [:add]
end

[2013-01-26T14:22:11+00:00] DEBUG: Re-raising exception: RuntimeError - apt_repository[datadog] (datadog::dd-agent line 24) had an error: RuntimeError: The repository file to create is nil, cannot continue.
/srv/chef/file_store/cookbooks/apt/providers/repository.rb:123:in `class_from_file'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/provider.rb:207:in `instance_eval'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/provider.rb:207:in `action_add'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/provider.rb:119:in `send'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/provider.rb:119:in `run_action'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource.rb:593:in `run_action'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/runner.rb:49:in `run_action'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/runner.rb:81:in `converge'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/runner.rb:81:in `each'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/runner.rb:81:in `converge'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection.rb:94:in `execute_each_resource'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/resource_collection.rb:92:in `execute_each_resource'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/runner.rb:80:in `converge'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/client.rb:378:in `converge'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/client.rb:420:in `do_run'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/client.rb:176:in `run'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/application/client.rb:283:in `run_application'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/application/client.rb:270:in `loop'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/application/client.rb:270:in `run_application'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/../lib/chef/application.rb:70:in `run'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-10.14.2/bin/chef-client:26
  /opt/vagrant_ruby/bin/chef-client:19:in `load'
  /opt/vagrant_ruby/bin/chef-client:19

installing the agent but disabling it

For some hosts I'd like to be able to switch the agent on and off on a temporary basis.
But I'd still like to have the agent installed already.

Move cookbook to first-class repo

Steps that need to be done:

1. Move the code in the repo to the root of the project
2. Rename the current repo to chef-datadog
3. Create a new repo named Datadog/cookbooks add a README.md that points to the new one
4. Update any processes that use the repo to reflect the change

Steps 204 need to be done/confirmed by an Owner.

Datadog alerts can be controlled from chef

Contributed by @littonj

actions :create, :modify, :delete

attribute :name,          :kind_of => String, :name_attribute => true
attribute :query,         :kind_of => String, :required => true
attribute :message,       :kind_of => String, :required => true
attribute :silenced,      :equal_to => [true, false],    :default => false

attribute :api_key,       :kind_of => String, :required => true
attribute :application_key,       :kind_of => String, :required => true

default_action :create

def initialize(*args)
  super
  @action = :create
end

require 'rubygems'
require 'dogapi'

def get_alert_id(name)
  @dog=Dogapi::Client.new(new_resource.api_key, new_resource.application_key)

  existing_alerts=@dog.get_all_alerts()
  if (existing_alerts.nil? or existing_alerts[0] != "200") then
    Chef::Log.info("unable to retrieve current list of alerts from DataDog")
    return nil
  end
  existing_alerts[1]['alerts'].each do |a|
    if a['name'] == name then
      return a['id']
    end
  end
  return 0
end

action :create do
  alert_id=get_alert_id(new_resource.name)
  if (not alert_id.nil? and alert_id == 0) then
    @dog.alert(new_resource.query, {:name => new_resource.name, :message => new_resource.message, :silenced => new_resource.silenced})
  end
end

action :modify do
  alert_id=get_alert_id(new_resource.name)
  if (not alert_id.nil? and alert_id != 0) then
    @dog.update_alert(alert_id, new_resource.query, {:name => new_resource.name, :message => new_resource.message, :silenced => new_resource.silenced})
  end
end

action :delete do
  alert_id=get_alert_id(new_resource.name)
  if (not alert_id.nil? and alert_id != 0) then
    @dog.delete_alert(alert_id)
  end
end

Handler loading is broken with chef_handler 1.0.6

Since chef_handler left 1.0.2, chef-handler-datadog won't load properly anymore.

There's a few ways to potentially make it work:

1. Change the source line in the handler resource, like so:

   chef_handler "Datadog" do
     source ::File.join(Gem::Specification.find_by_name('chef-handler-datadog').gem_dir, "lib", "chef-handler-datadog.rb")
     ...
     action :nothing
   end.run_action(:enable)

2. Work of pushing some changes upstream to chef_handler, to allow for loading a handler in 3 different ways: via gem using a gem path under 'chef/handler/, via gem using a gem path not under 'chef' namespace, or a flat file in the configured handler directory.

3. Change the namespace that chef-handler-datadog lives in. Right now, it's a first-class Gem, living at ::Datadog. Move the code to namespace it as Chef::Handler::Datadog, similar to the built-in handlers like JsonFile.

Option 1 isn't really backwards compatible, as it breaks on older versions of RubyGems, such as 1.3.7 which ships with CentOS 6.x (basically anything prior to 1.8.0), since they don't have any method to suss out the gem's dir without a lot of extra hackery.
Now, most people should be using something newer than Ruby 1.8.7 and RubyGems 1.3.7 - like the Opscode omnibus installer that much newer versions. But anyone using vaniall install of Chef gem via system Ruby & Rubygems is in for trouble.

Option 2 isn't great, as it has to be much more generalized and compatible with other deployments. This was partially discussed in IRC, and while some ideas were floated, I don't think it's really a great approach right now.

Option 3 seems the most likely, but will require a release of both the updated gem, as well as the cookbook, as the source statement will change to the new path.
The extent of the change seems to be moving a file into the correct location in the gemfile and releasing a new gem.

Feedback needed.

Remove extraneous package dependencies

As referenced in #6, adding ruby-dev packages may not be required by the dd-handler recipe, as the handler is executed at compile time, and the packages would be installed at execution time.

Lines: https://github.com/DataDog/chef-datadog/blob/master/recipes/dd-handler.rb#L23-38

Add dependency management in dd-agent.rb

E.g. if I have redis on, I should install the right redis package as part of the agent installation, same thing for mysql, postgresql, etc.

We should stop short of making modifications to the underlying service configuration (e.g. adding datadog to pg_hba.conf) to avoid surprises. We may decide to log a message if we can detect that the mysql dependencies are installed but the check is failing (bonus points for that).

redisdb, default port should be 6379

The default port for redis is 6379. Your redisdb.yaml.erb has 6739.

chef-handler-datadog 0.1.1 does not run on Chef 0.9.18

• chef 0.9.18 via apt-get
• chef-handler-datadog 0.0.1
• chef 0.16.12 via gems

#<NameError: uninitialized constant Kernel::Datadog>
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:81:in `const_get'
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:81:in `klass'
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:32:in `inject'
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:81:in `each'
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:81:in `inject'
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:81:in `klass'
/var/cache/chef/cookbooks/chef_handler/providers/default.rb:33:in `class_from_file'
/usr/lib/ruby/1.8/chef/provider.rb:104:in `instance_eval'
/usr/lib/ruby/1.8/chef/provider.rb:104:in `action_enable'
/usr/lib/ruby/1.8/chef/resource.rb:395:in `send'
/usr/lib/ruby/1.8/chef/resource.rb:395:in `run_action'
/var/cache/chef/cookbooks/datadog/recipes/dd-handler.rb:76:in `from_file'
/usr/lib/ruby/1.8/chef/cookbook_version.rb:473:in `load_recipe'
/usr/lib/ruby/1.8/chef/mixin/language_include_recipe.rb:40:in `include_recipe'
/usr/lib/ruby/1.8/chef/mixin/language_include_recipe.rb:27:in `each'
/usr/lib/ruby/1.8/chef/mixin/language_include_recipe.rb:27:in `include_recipe'
/usr/lib/ruby/1.8/chef/run_context.rb:94:in `load'
/usr/lib/ruby/1.8/chef/run_context.rb:91:in `each'
/usr/lib/ruby/1.8/chef/run_context.rb:91:in `load'
/usr/lib/ruby/1.8/chef/run_context.rb:55:in `initialize'
/usr/lib/ruby/1.8/chef/client.rb:166:in `new'
/usr/lib/ruby/1.8/chef/client.rb:166:in `run'
/usr/lib/ruby/1.8/chef/application/client.rb:222:in `run_application'
/usr/lib/ruby/1.8/chef/application/client.rb:212:in `loop'
/usr/lib/ruby/1.8/chef/application/client.rb:212:in `run_application'
/usr/lib/ruby/1.8/chef/application.rb:62:in `run'
/usr/bin/chef-client:26

Implement datadog::nagios recipe

Even though it's not using checks.d, it's worth making it easy to implement.

Cannot change agent hostname

Hi there,

Great work on Datadog, the dd-agent and this cookbook - we're huge fans!

Currently the /etc/dd-agent/datadog.conf template file "hostname" value is set to the value of "node.name".
See: https://github.com/evolution7/chef-datadog/blob/master/templates/default/datadog.conf.erb

We would like the ability to change the hostname.

It would be great if we could make this an attribute that can be overridden!

Cheers,
Ryan

jmx yaml recipe doesn't list attributes correctly

These lines:

https://github.com/DataDog/chef-datadog/blob/master/templates/default/jmx.yaml.erb#L19-L24

result in attributes attribute0, attribute1, etc. instead of the attribute name

add kafka jmx and consumer check

https://github.com/DataDog/dd-agent/blob/master/conf.d/kafka.yaml.example
https://github.com/DataDog/dd-agent/blob/master/conf.d/kafka_consumer.yaml.example

Chef Solo Role Tags Warning

Hi,

When using chef solo I get this warning after a successful chef run on and Amazon EC2 instance:

[2013-12-29T03:40:36+00:00] WARN: Could not submit ["role:base", "role:meanr", "role:master_branch"] tags for ip-10-238-139-249.ec2.internal to Datadog: 
["202", {"status"=>"ok", "event"=>{"priority"=>"low", "date_happened"=>1388288436, "handle"=>nil, "title"=>"Chef completed in 651 seconds on ip-10-238-139-249.ec2.internal ", "url"=>"https://app.datadoghq.com/event/jump_to?event_id=2080803783805692861", "text"=>"Chef updated 82 resources out of 107 resources total.

So it's not an error per say, but is there any way I can make this from not appearing?

From the nodes/ .json file the run list reads:

"run_list": [
    "role[base]",
    "role[meanr]",
    "role[master_branch]"
]

unnecessary files in the datadog cookbook

Someone in #datadog let us know that there are some unnecessary files in this cookbook, like .rvmrc, .gitignore, Berksfile.lock etc

Malformed version number string 2.7.5+

Environment

• Debian jessie (testing)
• Chef 11.8.0
• Ohai 6.20.0
• Python 2.7.5+

(what's the +? Read on...)

Problem

When loading the default attributes file, Chef raises with this lovely exception:

================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/datadog/attributes/default.rb
================================================================================

ArgumentError
-------------
Malformed version number string 2.7.5+

Cookbook Trace:
---------------
  /var/chef/cache/cookbooks/datadog/attributes/default.rb:47:in `new'
  /var/chef/cache/cookbooks/datadog/attributes/default.rb:47:in `from_file'

Relevant File Content:
----------------------
/var/chef/cache/cookbooks/datadog/attributes/default.rb:

 47>>   default['datadog']['install_base'] = Gem::Version.new(node['languages']['python']['version']) < Gem::Version.new('2.6.0')

For some reason, a + was appended to the default version of Python in Debian "jessie" (testing).

jtimberman@jenkins:~$ python --version
Python 2.7.5+

On Debian 7.2 (wheezy), this is not the case, so the error doesn't manifest itself.

jtimberman@debian-wheezy:~$ python --version
Python 2.7.3

Proposed Solution

A horrible #gsub could be sent to the python version attribute can be used to strip out the character(s) not allowed in Gem::Version comparison.

gpg key expired for apt repository

I been getting emails that the gpg key for your apt repository has expired.

W: GPG error: http://apt.datadoghq.com unstable Release: The following signatures were invalid: KEYEXPIRED 1394856986 KEYEXPIRED 1394856986 KEYEXPIRED 1394856986

Need to make sure apt/yum cookbooks are loaded when dd-handler is run

Can move this to default and include it in both dd-agent and dd-handler recipes.

chefignore is... ignored

Running 10.14.2 from Opscode's apt repo, I get, when I try to upload the cookbook, with chefignore in the root of the cookbook repository and in the datadog cookbook.

$ knife cookbook upload datadog
Uploading datadog        [0.0.13]
FATAL: Cookbook file spec/dd-agent_spec.rb has a ruby syntax error:
FATAL: /home/alq/chef-dogdev/datadog/spec/dd-agent_spec.rb:8: syntax error, unexpected tSYMBEG, expecting kDO or '{' or '('
FATAL:     Fauxhai.mock(platform:'ubuntu', version:'12.04') do |node|
FATAL:                                              ^
FATAL: /home/alq/chef-dogdev/datadog/spec/dd-agent_spec.rb:8: syntax error, unexpected ')', expecting kEND
FATAL:     Fauxhai.mock(platform:'ubuntu', version:'12.04') do |node|
FATAL:                                                     ^
FATAL: /home/alq/chef-dogdev/datadog/spec/dd-agent_spec.rb:31: syntax error, unexpected kEND, expecting $end

Process metrics can't be configured

While there is a template for the Process stats monitoring it doesn't have any parameterisation to let you specify what you want to monitor. It looks like it was added with no recipe/attributes to customise it.

dd-handler exception chef 10.12

Generated at 2012-09-10 18:43:11 +0000 ArgumentError: wrong number of arguments (1 for 0)
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/recipe.rb:68:in `node'
/var/chef/cache/cookbooks/datadog/recipes/dd-handler.rb:24:in `from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/mixin/from_file.rb:30:in `instance_eval'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/mixin/from_file.rb:30:in `from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/cookbook_version.rb:578:in `load_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/mixin/language_include_recipe.rb:46:in`load_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/mixin/language_include_recipe.rb:33:in `block in include_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/mixin/language_include_recipe.rb:27:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/mixin/language_include_recipe.rb:27:in `include_recipe' 
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/run_context.rb:72:in `block in load'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/run_context.rb:69:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/run_context.rb:69:in `load'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/client.rb:199:in `setup_run_context'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/client.rb:162:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/application/client.rb:254:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/application/client.rb:241:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/application/client.rb:241:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/lib/chef/application.rb:70:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.12.0/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:19:in `load'

Request: Add message / tags to dd-handler

It would be useful to be able to include bespoke text and/or set additional tags to events created by the dd handler.
My specific use-case is that we have a release cookbook that defines the version of components installed. I want to include the version of that cookbook that is in the run list.

Memcached recipe does not add all dependencies

The Datadog Memcached integration requires the python-memcache module, which is not currently added by the recipe.

Incorrect documentation for using instance id as agent hostname

Write now the documentation for use_ec2_instance_id in attributes says this:

# If running on ec2, if true, use the instance-id as the host identifier
# rather than the hostname for the agent or nodename for chef-handler.
default['datadog']['use_ec2_instance_id'] = false

This isn't correct since use_ec2_instance_id doesn't affect the agent hostname at all.

Examples of compatible defaults for default['datadog']['tags']

Ran into this issue recently: tags was set to "#{node.chef_environment}" while our handler is setting by default the "env" tag to be "env:#{node.chef_environment}". To be consistent and allow group-by's, I suggest that we at least document a good tag in the form of "env:...", but even set it by default to that value (if the version of chef supports it).

Support for encrypted data bags

The readme indicates that data bags were purposefully left out under the assumption that it's unlikely someone would have more than one api key. What has been overlooked is that many people prefer to store credentials of any kind (including api keys) in an encrypted data bag to avoid keeping sensitive information in node attributes.

I'm more than willing to open a PR with the necessary modifications to optionally support an encrypted data bag, but would like to gauge interest first. Would you accept such a feature?

Installation of the apt repository should be optional

The current datadog::dd-agent recipe does not provide a method of disabling the apt repository installation.

We mirror all packages into a single repository of our own to:

• Prevent our deployments depending on any 3rd parties
• Ensure we are in control of the timing of all upgrades
• Allow the testing of all upgrades on a staging environment

Default attributes set to nil can override attributes set at the same level

In one instance, using our cookbook with chef solo causes the Redis attributes to be set to nil by our cookbook thereby causing the Redis cookbook to run with nil attributes. There are workaround for this, e.g. set the Redis attributes at a higher priority than that of our cookbook but we're not playing nice.

Suggested solution #1: namespace our attributes to read from the Redis attributes with overriding them.

Default['datadog']['redis']['server']=default['redis']['server'] || nil

In the proper ruby way (making sure the above won't bomb)

Suggested solution #2: write a lightweight provider for our stuff (nice and idiomatic) so that you can say

Datadog_monitor(Redis, server, port)

And the rest is taken care of for you (nice in that it will hide our transition to split configuration files)

No way to install python-redis from package on rhel platform.

Agent install not working via chef (apt-1.8.0)

On ubuntu 12.04 with apt cookbook 1.8.0 and datadog cookbook 0.1.2 I get this error

ec2-23-20-41-214.compute-1.amazonaws.com [2013-01-23T20:59:38+00:00] INFO: Processing package[datadog-agent] action install (datadog::dd-agent line 45)
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com ================================================================================
ec2-23-20-41-214.compute-1.amazonaws.com Error executing action install on resource 'package[datadog-agent]'
ec2-23-20-41-214.compute-1.amazonaws.com ================================================================================
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com Chef::Exceptions::Package
ec2-23-20-41-214.compute-1.amazonaws.com -------------------------
ec2-23-20-41-214.compute-1.amazonaws.com No version specified, and no candidate version available for datadog-agent
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com Resource Declaration:
ec2-23-20-41-214.compute-1.amazonaws.com ---------------------
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com # In /var/chef/cache/cookbooks/datadog/recipes/dd-agent.rb
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com 45: package "datadog-agent"
ec2-23-20-41-214.compute-1.amazonaws.com 46:
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com Compiled Resource:
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com ------------------
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com # Declared in /var/chef/cache/cookbooks/datadog/recipes/dd-agent.rb:45:in `from_file'
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com package("datadog-agent") do
ec2-23-20-41-214.compute-1.amazonaws.com action :install
ec2-23-20-41-214.compute-1.amazonaws.com retries 0
ec2-23-20-41-214.compute-1.amazonaws.com retry_delay 2
ec2-23-20-41-214.compute-1.amazonaws.com package_name "datadog-agent"
ec2-23-20-41-214.compute-1.amazonaws.com cookbook_name "datadog"
ec2-23-20-41-214.compute-1.amazonaws.com recipe_name "dd-agent"
ec2-23-20-41-214.compute-1.amazonaws.com end
ec2-23-20-41-214.compute-1.amazonaws.com
ec2-23-20-41-214.compute-1.amazonaws.com [2013-01-23T20:59:43+00:00] ERROR: Running exception handlers
ec2-23-20-41-214.compute-1.amazonaws.com [2013-01-23T20:59:43+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json

If I add the following line after line 30 on ./recipes/dd-agent.rb (in the apt_repository stanza) then it works.

notifies :run, resources(:execute => "apt-get update"), :immediately

Not sure if this is due to a change in the apt cookbook or not, but I pulled that line from the README.md in the apt cookbook.

Allow configuration of collect_connection_state, and excluded_interfaces in network.yaml

I do not see a way to configure collect_connection_state, or excluded_interfaces.

recipe examples show 'host' parameter when it should be 'server'

for instance: https://github.com/DataDog/chef-datadog/blob/master/recipes/postgres.rb#L10

Request: Support Windows agent installation

Hi,

Do you plan to support installation of the Windows agent?

Add HTTP and TCP Check recipes

See https://github.com/DataDog/dd-agent/blob/master/conf.d/http_check.yaml.example
and https://github.com/DataDog/dd-agent/blob/master/conf.d/tcp_check.yaml.example

Support relations in postgres check

See new config here:
https://github.com/DataDog/dd-agent/blob/master/conf.d/postgres.yaml.example

Auto-generated chef tags include the Roles and Tags as well as the run list

It's at least as important to be able to filter alerts and graphs by role or chef tags as the run list

chef_handler dependency breaks berks upload

chef_handler v1.0.8 doesn't have the name parameter set in metadata.rb, it was added in v1.1.0, can we update the dependency restriction?

https://gist.github.com/bhouse/5249306

Better testing infrastructure

Using something like sous-chef

Version 0.1.1 Cookbook release

This release jumps the numbering scheme, but since version 0.1.0 has already been released on the community site, automated tools will select it when downloading the "most recent" cookbook.

Should incorporate:

• moving cookbook to first-class repo
• some automated travis testing
• some unit testing with ChefSpec

Add support for docker monitoring

Per: https://www.datadoghq.com/2014/06/monitor-docker-datadog/

Sample configuration: https://github.com/DataDog/dd-agent/blob/master/conf.d/docker.yaml.example

Support 'why-run' mode

Datadog agent template renders a lot of whitespace

It's not really needed just bloats the file, and we can remove a lot if it by ending conditionals with -%>.