datadog / dd-agent-omnibus

Omnibus project to build the Datadog Agent v5

Datadog Agent - Omnibus Project

This is an Omnibus project to build the Datadog Agent packages.

It's using a fork of the official 4.0.0 release of the Omnibus project.

Builds are run in docker containers with Circleci. See:

• https://github.com/DataDog/docker-dd-agent-build-deb-i386
• https://github.com/DataDog/docker-dd-agent-build-rpm-i386
• https://github.com/DataDog/docker-dd-agent-build-deb-x64
• https://github.com/DataDog/docker-dd-agent-build-rpm-x64

Build a package locally

• Install Docker

• Run the following script with the desired parameters

PLATFORM="deb-x64" # must be in "deb-x64", "deb-i386", "rpm-x64", "rpm-i386"
TRACE_AGENT_BRANCH="master" # Branch of the datadog-trace-agent repo to use, default "master"
PROCESS_AGENT_BRANCH="master" # Branch of the datadog-process-agent repo to use, default "master"
AGENT_BRANCH="master" # Branch of dd-agent repo to use, default "master"
OMNIBUS_BRANCH="master" # Branch of dd-agent-omnibus repo to use, default "master"
AGENT_VERSION="5.4.0" # default to the latest tag on that branch
LOG_LEVEL="debug" # default to "info"
LOCAL_AGENT_REPO="~/dd-agent" # Path to a local repo of the agent to build from. Defaut is not set and the build will be done against the github repo

# The passphrase of the key you want to use to sign your .rpm package (if
# building an RPM package). If you don't set this variable, the RPM won't be
# signed but the build should succeed. Note that you must also mount a volume
# under /keys and bind it to a folder containing an RPM-SIGNING-KEY.private
# file containing your exported signing key. Finally, be aware that the
# package_maintainer DSL defined in config/projects/datadog_agent.rb and the
# full key name (My Name (comments) <[email protected]>) must match.
RPM_SIGNING_PASSPHRASE="my_super_secret_passphrase"

mkdir -p pkg
mkdir -p "cache/$PLATFORM"
docker run --name "dd-agent-build-$PLATFORM" \
  -e OMNIBUS_BRANCH=$OMNIBUS_BRANCH \
  -e LOG_LEVEL=$LOG_LEVEL \
  -e AGENT_BRANCH=$AGENT_BRANCH \
  -e AGENT_VERSION=$AGENT_VERSION \
  -e TRACE_AGENT_BRANCH=$TRACE_AGENT_BRANCH \
  -e PROCESS_AGENT_BRANCH=$PROCESS_AGENT_BRANCH \
  -e RPM_SIGNING_PASSPHRASE=$RPM_SIGNING_PASSPHRASE \
  -e LOCAL_AGENT_REPO=$LOCAL_AGENT_REPO # Only to use if you want to build from a local repo \
  -v `pwd`/pkg:/dd-agent-omnibus/pkg \
  -v `pwd`/keys:/keys \
  -v "`pwd`/cache/$PLATFORM:/var/cache/omnibus" \
  -v $LOCAL_AGENT_REPO:/dd-agent-repo # Only to use if you want to build from a local repo \
  "datadog/docker-dd-agent-build-$PLATFORM"

# Cleanup (necessary to launch another build)
docker rm dd-agent-build-$PLATFORM

Build on Mac OS X

The Mac build platform should have:

• Xcode installed (type git in a terminal),
• Go installed,
• sudoer rights for the build user,
• Bundler installed: sudo gem install bundler,
• Important directories created: sudo mkdir -p /var/cache/omnibus /opt/datadog-agent,
• Owned by the right user: sudo chown $USER:nogroup /var/cache/omnibus /opt/datadog-agent.
• Xcode license accepted (to sign package) sudo xcodebuild -license
• Datadog signing key

Then run:

AGENT_BRANCH=<YOUR_AGENT_BRANCH> OMNIBUS_BRANCH=<YOUR_OMNIBUS_BRANCH> OMNIBUS_SOFTWARE_BRANCH=<YOUR_OMNIBUS_SOFTWARE_BRANCH> sh omnibus_build.sh

The env vars have the same meaning as the Dockerized build above. Omitting them will cause the default of master to be used for all 3

dd-agent-omnibus's Issues

auto_conf files not copied in the package

After a clean installation of the 5.12.0~rc.6 on Xenial, the folder /etc/dd-agent/conf.d/auto_conf is empty.
The culprit might be this: https://github.com/DataDog/dd-agent-omnibus/blob/master/config/software/datadog-agent-integrations.rb#L101 the name of the file is missing an underscore.

Package PREIN Distribution Check Fails when /etc/issue is non standard.

Installation of datadog agent fails due to the method of checking for distribution in the pre-install script:

https://github.com/DataDog/dd-agent-omnibus/blob/master/package-scripts/datadog-agent/preinst#L7

Which doesn't leave much room for custom /etc/issue files and I would like to respectfully ask this is reviewed to either make it more flexible or more use more OS resources to verify the specific release.

In our instance the /etc/issue file is built by Chef and includes the line:

System Info: rhel amazon 2016.09

Which is built from the following chef template configuration (the template itself has a legal warning followed by system info):

template "/etc/issue" do
  source "issue"
  mode 0644
  variables(
    "platform" => node["platform"],
    "platform_version" => node["platform_version"],
    "platform_family" => node["platform_family"]
  )
end

We use the same base cookbook across multiple distributions.

Unfortunately here the case doesn't match. An alternative file to check on Amazon Linux would be /etc/system-release which contains:

Amazon Linux AMI release 2016.09

A simple fix would be just to add case insensitivity to the grep statement which would fix my use case but that is selfish and suggest checking:

• lsb_release (if installed)
• All /etc/*-release files present
• /etc/issue
• uname -s

Adding dogapi?

It would be nice to have the dog cli script available which is provided by https://github.com/DataDog/dogapi https://github.com/DataDog/datadogpy - how about adding the python library to the omnibus distribution of dd-agent?

MySQL Max Connections

Hello,

This was very confusing for me before I looked into the code:
https://github.com/DataDog/dd-agent/blob/master/checks.d/mysql.py#L20

MySQL has a system variable called max_connections AND a status variable called max_used_connections. This name change threw me off a bit.

Consider including non-platform-specific dependencies in package

Scenario

In a recent case, a customer tried to use the SQL Server monitoring check on a Linux-based host.

The Agent failed with a message of:

sqlserver
---------
- initialize check class [ERROR]: ImportError('No module named adodbapi',)

We typically include this dependency only in our Windows-based builds, it may be simple enough to add this, and the Pyro4 dependency of adodbapi to TSCA builds.

Workaround

sudo /opt/datadog-agent/embedded/bin/pip install adodbapi Pyro4
sudo service datadog-agent restart

Upgrade to omnibus 4.0

Omnibus 4.0 has been out for a month now, we should try to keep up to date with it.

There are a few caveats though as we are currently using a forked version of omnibus 3.2.2 as we needed a few extra extra settings.

The main one will be the to make sure that we can pass this posttrans script to the RPM files
https://github.com/DataDog/dd-agent-omnibus/blob/master/package-scripts/datadog-agent/posttrans

That's the main reason we forked omnibus as they didn't provide the interface to do that.

Also omnibus 4.0.0 and uses system tooling so we'll need to make sure that this tooling will let us do that.

Delete unused branches?

This repo has a lot of unused branches, can you clean it up?

Bundle changelog in packaged dd-agent

In debian-style packages, one can update the debian/changelog file with information for new releases.
This usually shows in "need upgrading" reports (such as apticron's), and they're pretty useful for sysadmins.
I don't know if omnibus supports this.

https://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog

apticron report [Tue, 17 Nov 2015 09:09:18 +0100]
========================================================================

apticron has detected that some packages need upgrading on:

    XXXX 
    [ xxx.xxx.xxx.xxx xxxx:xxxx:xxxx:xxxx::1 ]

The following packages are currently pending an upgrade:

    datadog-agent 1:5.6.2-1


========================================================================

Package Details:

Reading changelogs...
========================================================================

Issue building dd-agent

Is anyone else seeing this, building an rpm64 from my mac, and having issues.

The following shell command exited with status 128:

$ git --git-dir=/var/cache/omnibus/cache/git_cache/opt/datadog-agent --work-tree=/opt/datadog-agent add -A -f

Output:

(nothing)

Error:

fatal: Will not add file alias 'embedded/share/terminfo/32/2621a' ('embedded/share/terminfo/32/2621A' already exists in index)

apt postinst script

I noticed that datadog-agent 5.9.0-1 debian packages do not have the updated postinst from #89.

As far as I can tell, 55d0ea8 exists in 5.9.0 and #89 shows up in https://github.com/DataDog/dd-agent/releases/tag/5.9.0

Is there anything I am missing?

https://s3.amazonaws.com/apt.datadoghq.com/pool/d/da/datadog-agent_5.9.0-1_amd64.deb
https://s3.amazonaws.com/apt.datadoghq.com/pool/d/da/datadog-agent_5.9.0-1_i386.deb

from

http://apt.datadoghq.com/pool/d/da/

52 MB of Python byte code

@olivielpeau Please exclude Python byte code from the Debian packages.

find /opt/datadog-agent "(" -name "*.pyc" -o -name "*.pyo" ")" -delete

It takes up additional 52 MB.
Thank you.

Lintian

Actually lintian suggests a day of work.
4842 lines excluding dir-or-file-in-opt-s.

Ubuntu/Debian debsums. wrong md5sum of /usr/bin/dd-agent

After proper install of datadog-agent from repository:
https://apt.datadoghq.com/ stable main

the file /usr/bin/dd-agent is reported on my hosts as modified.
I looked into that and I've found out that in postInst script
a /usr/bin/dd-agent file is overwritten with symbolic link:
ln -sf $INSTALL_DIR/agent/agent.py `/usr/bin/dd-agent`
as a result it still has a md5 hash of overwritten file in package md5sums file.
and my periodic debsums -c check reports an error on /usr/bin/dd-agent file

I consider it as a bug of install scripts and it would have been nice to get rid of this.

Additional environment details (Operating System, Cloud provider, etc):
i run Debian Wheezy,Debian Jessie hosts. some physical some virtual.
install process on these don't differ and error still occur.

Steps to reproduce the issue:

1. install datadog-agent package on Debian Jessie host. runapt-get install datadog-agent
2. after installation complete run debsums -c

Additional information you deem important (e.g. issue happens only occasionally):
The "bug" as i've called it does not affect overall datadog agent behavior or run but in my case i have to consider it as a minor security issue.

Using python-apt in checks

Hi! I'd like to write an agent check that uses the apt python library.

I know that on debian and ubuntu host, this is part of the python-apt package. Whose source is git://anonscm.debian.org/apt/python-apt.git But I don't know if that ties into omnibus.

I see https://pypi.python.org/pypi/apt/ but that looks surprisingly out of date.

Do you have any advice?

SSL libs may not be compiled in

In attempting to run the omnibus agent in an extremely limited environment, I attempted to run dogstatsd standalone mode, however when running, I receive the following error:

$ API_KEY=111 ./run-dogstatsd.sh
Traceback (most recent call last):
  File ".apt/opt/datadog-agent/agent/dogstatsd.py", line 446, in <module>
    sys.exit(main())
  File ".apt/opt/datadog-agent/agent/dogstatsd.py", line 415, in main
    reporter, server, cnf = init(config_path, use_watchdog=True, use_forwarder=opts.use_forwarder, args=args)
  File ".apt/opt/datadog-agent/agent/dogstatsd.py", line 395, in init
    reporter = Reporter(interval, aggregator, target, api_key, use_watchdog, event_chunk_size)
  File ".apt/opt/datadog-agent/agent/dogstatsd.py", line 90, in __init__
    self.http_conn_cls = http_client.HTTPSConnection
AttributeError: 'module' object has no attribute 'HTTPSConnection'

Sure enough, the module does not have the attribute:

$ .apt/opt/datadog-agent/embedded/bin/python  -c "import sys; print sys.version_info"
sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0)
$ .apt/opt/datadog-agent/embedded/bin/python  -c "import httplib; print filter(lambda attr: attr.startswith('HTTP'), dir(httplib))"
['HTTP', 'HTTPConnection', 'HTTPException', 'HTTPMessage', 'HTTPResponse', 'HTTPS_PORT', 'HTTP_PORT', 'HTTP_VERSION_NOT_SUPPORTED']

I then tried using the requests library, as that's the one we use in the forwarder, considering maybe we rewrite the dogstatsd standalone emitter to be the same as the forwarder, and didn't get much better results:

>>> import requests
>>> url = 'http://www.google.com/'
>>> r = requests.get(url)
>>> r.text
u'<!doctype html><html itemscope=""...........


>>> url = 'https://app.datadoghq.com'
>>> r = requests.get(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 722, in _validate_conn
    conn.connect()
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 169, in connect
    self.sock = ssl.wrap_socket(conn, self.key_file, self.cert_file)
AttributeError: 'NoneType' object has no attribute 'wrap_socket'


>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/.apt/opt/datadog-agent/embedded/lib/python2.7/ssl.py", line 60, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: /app/.apt/opt/datadog-agent/embedded/lib/python2.7/lib-dynload/_ssl.so: undefined symbol: SSLv2_method

This is datadog-agent 5.1.0, pyopenssl 0.14

This may be due to the correct ssl libs not being present during compile phase, which then leads to the HTTPS* attributes not being available.

An obscure thread on the topic mentions adding the C dev modules before compiling: pypa/pip#833 (comment)

I'm not entirely certain where we would place the apt-get install libssl-dev in the pipeline, as well as see why we're not getting Python 2.7.8 either.

package: http://packages.ubuntu.com/trusty/libssl-dev

A well-formed, ssl-compiled httplib output will look like this:

$ python -c "import httplib; print filter(lambda attr: attr.startswith('HTTP'), dir(httplib))"
['HTTP', 'HTTPConnection', 'HTTPException', 'HTTPMessage', 'HTTPResponse', 'HTTPS', 'HTTPSConnection', 'HTTPS_PORT', 'HTTP_PORT', 'HTTP_VERSION_NOT_SUPPORTED']

preinst script breaks on servers with restricted networks

On Debian based systems, the preinst script in the .deb package causes a failure when attempting to retrieve PGP keys.

I submitted a pull request which checks for existing PGP keys in apt-key, and also checks if a proxy is in use.
#143

Build error with datadog-gohai

Not exactly sure where to put this, but I'm getting a build error when I build dd-agent with the latest changes to gohai:

 [Builder: datadog-gohai] I | 2017-05-01T21:32:50+00:00 | Execute: `/usr/local/go/bin/go get -d -u github.com/DataDog/gohai': 1.145s
 [Builder: datadog-gohai] I | 2017-05-01T21:32:50+00:00 | Build datadog-gohai: 1.5283s
The following shell command exited with status 1:

    $ GOPATH=/var/cache/omnibus/src/datadog-gohai GOROOT=/usr/local/go /usr/local/go/bin/go get -d -u github.com/DataDog/gohai

Output:

    (nothing)

Error:

    package github.com/DataDog/gohai
	imports context: unrecognized import path "context"

Seems like this was recently merged: shirou/gopsutil@772ab1f

and "context" depends on go >= 1.5, but the version of go in datadog/docker-dd-agent-build-deb-x64 is 1.3.3.

How is the build working for you guys?