elastic / beats-packer

DEPRECATED: Moved to https://github.com/elastic/beats/tree/master/dev-tools/packer

Beats Packer

Tools, scripts and docker images for cross-compiling and packaging the Elastic Beats.

Prepare

You need Go and docker installed. This project uses several docker files, you can either build them with:

 make images

Or pull them from the Docker registry with:

 make pull-images

Prepare the rest with:

 make deps

Cross-compile

The cross compilation part is based on xgo, with some changes that add a bit more extensibility that we needed for the Beats (e.g. static compiling, custom docker image).

You can cross-compile one Beat for all platforms with (e.g.):

 make packetbeat

Packaging

For each OS (named platform here) we execute a build.sh script which is free to do whatever it is required to build the proper packages for that platform. This can include running docker containers with the right tools included or with that OS installed for native packaging.

The deb and rpm creation is based on fpm which is executed from a container.

Besides the platform, there are three other dimensions: architecture, beat and the release. Each of these is defined by YAML files in their folders. These dimensions only set static options, the platforms is the only one scripted.

The runner is currently (ab)using a Makefile, which is nice because it can parallelize things automatically, but it's hacky so we might replace it in a future.

Building all Beats for all platforms:

 make clean && make -j2

Which currently produces the following:

    topbeat-1.0.0-nightly.150813173034-darwin.tgz
    topbeat-1.0.0-nightly.150813173034-darwin.tgz.sha1
    topbeat_1.0.0-nightly.150813173034_i386.deb
    topbeat_1.0.0-nightly.150813173034_i386.deb.sha1
    topbeat-1.0.0-nightly.150813173034-windows.zip
    topbeat-1.0.0-nightly.150813173034-windows.zip.sha1
    topbeat-1.0.0-nightly.150813173034-i686.rpm
    topbeat-1.0.0-nightly.150813173034-i686.rpm
    topbeat_1.0.0-nightly.150813173034_amd64.deb
    topbeat_1.0.0-nightly.150813173034_amd64.deb.sha1
    topbeat-1.0.0-nightly.150813173034-x86_64.rpm
    topbeat-1.0.0-nightly.150813173034-x86_64.rpm
    packetbeat-1.0.0-nightly.150813173058-windows.zip
    packetbeat-1.0.0-nightly.150813173058-windows.zip.sha1
    packetbeat-1.0.0-nightly.150813173058-darwin.tgz
    packetbeat-1.0.0-nightly.150813173058-darwin.tgz.sha1
    packetbeat_1.0.0-nightly.150813173058_i386.deb
    packetbeat_1.0.0-nightly.150813173058_i386.deb.sha1
    packetbeat-1.0.0-nightly.150813173058-i686.rpm
    packetbeat-1.0.0-nightly.150813173058-i686.rpm
    packetbeat_1.0.0-nightly.150813173058_amd64.deb
    packetbeat_1.0.0-nightly.150813173058_amd64.deb.sha1
    packetbeat-1.0.0-nightly.150813173058-x86_64.rpm
    packetbeat-1.0.0-nightly.150813173058-x86_64.rpm

beats-packer's Issues

Add ARM packages

For some of the Beats, namely topbeat and filebeat, linux on ARM would be easy to support, because they have been reported to work already. So what's missing is some sort of packages. We could start with a simple .tar.gz containing the binary and the config file, then we could also look into creating deb packages for Rasbian and the like.

Remove first line of config

As the config file is now auto generated, the first line of the config file must be removed that states it should not be modified. In addition etc/beat.yml must be removed.

See here for more details: https://github.com/elastic/filebeat/pull/218

Unable to install nightly build on Debian: 'Version' field string 'nightly151208141331': version number does not start with digit

As requested by @ruflin after first reporting in beats repo: elastic/beats#481

dpkg -i filebeat_nightly.latest_amd64.deb
dpkg: error processing archive filebeat_nightly.latest_amd64.deb (--install):
parsing file '/var/lib/dpkg/tmp.ci/control' near line 2 package 'filebeat':
error in 'Version' field string 'nightly151208141331': version number does not start with digit
Errors were encountered while processing:
filebeat_nightly.latest_amd64.deb

Trying to install on Debian 8 64 bit VM. The stable release can be installed, but I would like to test some of the resolved issues in the latest nightly build. Got the filebeat nightlies from here: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

Build failure on master

Master currently fails because it tries to apply gopacket_pcap.patch to the beats framework, but this patch needs an update to apply cleanly

Purge old nightlies from the S3 bucket

On each commit to master and every night, a new set of build artifacts is published to the beats-nightlies S3 bucket. Should these artifacts be treated as transient and be purged after a period of time?

If so, we can configure S3 to manage the lifecycle of the artifacts for us. See https://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html

Amazon AMI support (fedora/rpm)

Amazon AMI (and other cloud solutions) use cloud-init to apply some additional configuration (e.g. configure hostname on startup). If cloud-init is run after beats are started reported hostname is wrong (which can become confusing in cloud environment).

See this discussion for solution in fedora based on systemd.

Missing repository for 5.0.0-alpha1 (or 5.x, in general)

You might be aware of this already, but I can't find an issue filed anywhere. Plus, I'm excited about the Elastic Pioneer Program ;)

My problem is that I can't find a repo for the 5.x beats, like there is one for Elasticsearch:
echo "deb http://packages.elastic.co/elasticsearch/5.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-5.x.list

And for Kibana:
echo "deb http://packages.elastic.co/kibana/5.0.0-alpha/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana.list

And for Logstash:
echo "deb http://packages.elastic.co/logstash/5.0/debian stable main" | sudo tee -a /etc/apt/sources.list

In the case of beats, there's no version in the URL:
echo "deb https://packages.elastic.co/beats/apt stable main" | sudo tee -a /etc/apt/sources.list.d/beats.list

I assume that when 5.0 GA will be released, the beats repo will point to it, but that might push an unwanted upgrade (i.e. between major releases). Ideally, the behavior would be consistent across the Elastic Stack, by having multiple repos limited to the same scope for all projects (IMO either to major versions, like ES is now, or to minor versions, like Logstash is - just in case something breaking comes up. I think Kibana's repo down to the patch level is too strict, as is the one-repo-to-rule-them-all current approach for beats). Having the same scope would make upgrades consistent.

Create nightlies continuosly

Either by using Travis-CI or Jenkins or both. Both should be possible because they support docker. Travis-CI might be problematic to trigger (we'd want to trigger it on code changes).

A statically linked filebeat binary

beats-packer produces a dynamically linked Filebeat binary when a statically linked binary would be preferred for portability.

See https://discuss.elastic.co/t/static-build/34354

Add support for winlogbeat

I'd do it sooner rather than later, so people can try the nightlies early.

Create package for Filebeat

Create a package for Filebeat, as well.

Add systemd support

Currently the packages provide SysV-style init scripts, but a lot of operating system primarily use systemd so we should provide packages that contain systemd scripts for those OS'es.

/cc: @dakrone

Write Git Hash into Nightly Build

To make the build of the nightlies easier, currently the Git hash is not part of the nightly build name (every beat would have a hash). To make it possible to identify exactly which git version was used to create the nightly, a file with the Git hash should be written into every nightly (or something similar).

Switch to Go 1.5.1

Install an /etc/filebeat/conf.d for Filebeat

When Filebeat is installed via RPM or DEB, the /etc/filebeat/conf.d directory should be created and the /etc/filebeat/filebeat.yml file should have config_dir: /etc/filebeat/conf.d.

Tilde breaks install on older versions of RHEL

This is only an issue for development builds where we add tilde to the version. Seems like tilde support was added around RHEL 6.4. I just wanted to document the issue.

Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
rpmlib(TildeInVersions) is needed by filebeat-1.1.0~SNAPSHOT-1.x86_64
Complete!
(1, [u'Please report this error in https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%205&component=yum'])

Remove 1.x pre-GA versions from yum repos

It seems that pre-GA versions confuse some implementations of the rpmvercmp algorithm (i.e. it thinks 1.0.0~beta4 is newer than 1.0.0).

To mitigate the issue we should remove the pre-GA versions from the repo.

See: https://discuss.elastic.co/t/beats-repo-and-spacewalk/36043