travis-ci / travis-cookbooks

Chef cookbook monolithic repo :book: :bomb:

Travis cookbooks!

Travis cookbooks are collections of Chef cookbooks used with Chef for setting up Linux VMs for running tests and Travis internal machines.

The wrapper cookbooks that compose together the cookbooks found here live over in the Travis CI Infrastructure Packer Templates repository.

Developing Cookbooks

Directory structure

There are two cookbook path directories in this repository:

• cookbooks authored by Travis CI
• community-cookbooks vendored community stuff

Requirements

There is no Gemfile for specifying Chef dependencies. Please install the ChefDK.

Testing

The script that's run on Travis is ./runtests, which by default runs on the ./cookbooks directory. Example usage:

./runtests

Any change to the cookbooks should also be tested on an actual build-VM, after building a test image. More information on how to trigger a test image build can be found in the packer-templates README.

Branches

There are two long-lived branches:

• master per tradition, which is compatible with Ubuntu 16.04
• trusty-stable, which is compatible with Ubuntu 14.04

Please target your patches accordingly.

License

See the LICENSE files.

travis-cookbooks's Issues

php.ini default name in php5.2 is php.ini-dist

After php5.3, php.ini default name is php.ini-[development,production]. So this is correct:
https://github.com/travis-ci/travis-cookbooks/blob/master/ci_environment/phpbuild/providers/build.rb#L17
./php-build -i development #{version} #{target_path}

But in php5.2, default name is php.ini-[dist,recommended]. So php5.2 (ie. php5.2.17), php-build requires:
./php-build -i dist #{version} #{target_path}

-i option is php.ini's suffix name.

via: php5.2 does not have loaded php.ini, so how to load php extension?
https://groups.google.com/forum/?fromgroups#!topic/travis-ci/XJTYdCCKgkI

Preinstall Cassandra

It is time to preinstall Cassandra (w/o starting it on boot), the same way we do for Neo4J and ElasticSearch.

[php] add pgsql support

Suggest to add the pgsql module to the default PHP installation. To enable this the ./configure line has to be edited to include --with-pgsql[=DIR].
ref: http://www.php.net/manual/en/pgsql.installation.php

Provision Cassandra from a tarball: Cassandra from apt repos fails to start with OpenVZ

Due to https://issues.apache.org/jira/browse/CASSANDRA-3636 Cassandra won't install when using apt.

We might want to change the recipe to install it from a binary instead.

Provision Erlang R15B1

Newer version of elasticsearch

Hiya

One of my dists depends on elasticsearch 0.19.7 http://www.elasticsearch.org/download/2012/06/26/0.19.7.html

What do I need to do (now and in the future) to make it easy for you to upgrade ES?

ta

clint

Preinstall HBase

It is time to preinstall HBase (w/o starting it on boot), the same way we do for Neo4J and ElasticSearch. I am not entirely sure what version it needs to be, Hadoop ecosystem version numbering does not exactly make sense.

Add one Perl version to each test VM

According to documentation and this repository, each test VM have one preinstalled version of Ruby, OpenJDK, Python and Node.js.

I think it will be good to include Perl with cpanm to this list.

rabbitmq-management-plugin support

The rabbitmq management plugin available here: http://www.rabbitmq.com/management.html provides a convenient REST api for creating virtualhosts. gen_bunny uses this to isolate test cases in it's integration tests, so it'd be useful if it were available in the travis-ci rabbitmq install. There are probably other use cases as well.

Provide CouchDB 1.1.

The version of CouchDB you are installing by default is outdated. The current version is 1.1.1. (Sven is standing next to me and said I should include this fact).

Could not open a socket to localhost and 127.0.0.1 on port 80

http://travis-ci.org/#!/renansaddam/cakephp/builds/354220

CakePHP tries to open a socket on port 80 on both localhost and 127.0.0.1
Using php.net/fsockopen to open the socket.

[php] fopen fails

When running my test suite agains php 5.3.2 some tests fail with: fopen(): Unable to find the wrapper "https" - did you forget to enable it when you configured PHP? but on php 5.3.10 the suite passes. As far as I can see nothing changed in fopen between those versions so this appears to be something in the php build on travis.

Chef-rvm

The current recipe for rvm is quite light (and it doesn't works out of the box).
There is chef-rvm, which is really great, and could perhaps be a good replacement.

Disable more services on boot

This is mostly to do with the SBT recipe, but is another an interesting issue in general.

When the Sbt recipe tries to run it needs to allocate a fair bit of memory. This is usually fine but if the chef run fails for some reason (maybe on the second to last recipe) then when you go to run it again it will fail due to memory being scarce.

The main offenders seem to be : riak, couchdb, and rabbitmq

Rabbit alone was taking 140+ megs.

All jobs successful but build doesn't finish

Hiya

Have a look at http://travis-ci.org/#!/clintongormley/Elastic-Model/builds/1808402

All its jobs completed successfully, but the build didn't finish

Sphinx doesn't check if it is already installed

When running chef over and over again on the same machine it seems Sphinx doesn't check if it is already installed and tries to install itself again.

Minor annoyance.

CouchDB version

gh-27 and http://about.travis-ci.org/docs/user/ci-environment/ indicate that travis should be running CouchDB 1.1.0, but I continue to see 1.0.1, e.g. here

(Note the line in the tests that echoes the database server's welcome message: {:couchdb Welcome, :version 1.0.1})

Add Perl 5.16

Hiya

It's been out over a month. Any chance we can get access to it?

ta

Fix gem uuid4r compilation

Hi,

I'm trying to test a new project of mien that has uuid4r as dependency.
When I start tests on travis-ci, it fail with:

Installing uuid4r (0.1.2) with native extensions /home/vagrant/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:552:in `rescue in block in build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)

/home/vagrant/.rvm/rubies/ruby-1.9.2-p290/bin/ruby extconf.rb
checking for uuid_export() in -luuid... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/home/vagrant/.rvm/rubies/ruby-1.9.2-p290/bin/ruby
--with-uuidlib
--without-uuidlib
OSSP uuid library 'uuid' required -- not found.

I had this issue on my Ubuntu box, and I've fixed it by installing the pakage libossp-uuid-dev.

I also found on Google that someone has the same issue beginning of August.

SBT runner script needs a variable for tweaking JVM options

PHP 5.2.x: failed to open stream: Too many open files

Originally posted in [travis-ci/travis-ci#463]:

Warning: require(/home/vagrant/.phpenv/versions/5.2.17/pear/PHPUnit/Util/Filter.php) [http://php.net/function.require]: failed to open stream: Too many open files in /home/vagrant/.phpenv/versions/5.2.17/pear/PHPUnit/Autoload.php on line 203

This only happens on the 5.2 builds. The 5.3 and 5.4 builds run fine.

http://travis-ci.org/#!/ceeram/cakephp/jobs/867439

Upgrade Leiningen to 1.7.0

Mysql PDO tries to connect on wrong host on PHP 5.4

http://travis-ci.org/#!/renansaddam/cakephp/builds/354092
Even when using the right DSN string.

mysql:host=0.0.0.0;port=3306;dbname=cakephp_test

Error: Database connection "SQLSTATE[HY000] [1045] Access denied for user ''@'localhost' (using password: NO)" is missing, or could not be created.

Consider supporting Jython

Would it be possible to add Jython to the list of supported Python interpreters?

http://packages.ubuntu.com/precise/jython

Add tests for provisioning

I still think it's a good idea to have all the tiny requirements (like, ~/.rvmrc ignores local .rvmrcs) documented in form of tests.

Make sure the cookbooks are re-runnable

Currently at least the mysql recipes fail when run for the second time because of wrong permissions

Provide avr32-toolchain preinstalled

This would enable CI for embedded C (32-bit AVRs). The toolchain can be easily built and installed with AVR32 Toolchain Builder, https://github.com/jsnyder/avr32-toolchain.

Work around Chef's double-linking bug in the neo4j-server recipe

[Sun, 13 May 2012 13:16:23 +0000] INFO: bash[chown -R neo4j:neo4j /var/lib/neo4j-server/] ran successfully
[Sun, 13 May 2012 13:16:23 +0000] INFO: bash[chown -R neo4j:neo4j /var/lib/neo4j-server/data/graph.db] ran successfully
[Sun, 13 May 2012 13:16:23 +0000] INFO: bash[chown -R neo4j:neo4j /usr/local/neo4j-server/data] ran successfully
[Sun, 13 May 2012 13:16:23 +0000] INFO: bash[chown -R neo4j:neo4j /usr/local/neo4j-server/system] ran successfully
[Sun, 13 May 2012 13:16:23 +0000] INFO: bash[chown -R neo4j:neo4j /usr/local/neo4j-server] ran successfully
[Sun, 13 May 2012 13:16:23 +0000] INFO: link[/usr/local/bin/neo4j] created
[Sun, 13 May 2012 13:16:23 +0000] ERROR: link[/usr/local/bin/neo4j] (neo4j-server::tarball line 95) has had an error
[Sun, 13 May 2012 13:16:23 +0000] ERROR: link[/usr/local/bin/neo4j] (/tmp/vm-provisioning/travis-cookbooks/ci_environment/neo4j-server/recipes/tarball.rb:95:in `block in from_file') had an error:
link[/usr/local/bin/neo4j] (neo4j-server::tarball line 95) had an error: TypeError: can't convert nil into String
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/file_access_control/unix.rb:106:in `stat'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/file_access_control/unix.rb:106:in `stat'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/file_access_control/unix.rb:61:in `set_owner'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/file_access_control/unix.rb:30:in `set_all'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/mixin/enforce_ownership_and_permissions.rb:33:in `enforce_ownership_and_permissions'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/provider/link.rb:96:in `action_create'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource.rb:454:in `run_action'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/runner.rb:49:in `run_action'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/runner.rb:85:in `block (2 levels) in converge'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/runner.rb:85:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/runner.rb:85:in `block in converge'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection.rb:94:in `block in execute_each_resource'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/resource_collection.rb:92:in `execute_each_resource'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/runner.rb:80:in `converge'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/client.rb:330:in `converge'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/client.rb:163:in `run'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/application/solo.rb:207:in `block in run_application'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/application/solo.rb:195:in `loop'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/application/solo.rb:195:in `run_application'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/lib/chef/application.rb:70:in `run'
/usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.10/bin/chef-solo:25:in `<top (required)>'
/usr/local/bin/chef-solo:23:in `load'
/usr/local/bin/chef-solo:23:in `<main>'
[Sun, 13 May 2012 13:16:23 +0000] ERROR: Running exception handlers
[Sun, 13 May 2012 13:16:23 +0000] ERROR: Exception handlers complete

The issue seems to be with neo4j-shell, if you do a which neo4j-shell it returns nothing.

GD is not compiled with FreeType support

Originally travis-ci/travis-ci#627

Provide NumPy preinstalled

Moved from travis-ci/travis-ci#534

As discussed on the travis-ci Google Group, many Python packages depend on NumPy so it would be helpful to have a recent NumPy installed for each supported Python platform (i.e. Python 2.5, 2.6, 2.7 and 3.2 at the time of writing).

Note PyPy is a special case (and now comes with its own built-in partial NumPy emulating library, NumPyPy)

Currently it appears NumPy must be installed from source, which takes a while and wastes bandwidth and CPU.

Preinstall Dist::Zilla and Moose for Perl

Hiya

Dist::Zilla and Moose are widely used by Perl modules and take ages to install. Is there any possibility we could get them preinstalled?

ta

Provide Leiningen 2.0 preview

The plan is to provide it as lein2 in the CI env and make Clojure builder recognize lein key which will have the value of "lein" by default but can be changed to "lein2" for projects that migrate to Leiningen 2.

Migrate standard VM image to Ubuntu 11.10

Our goal is to always be one Ubuntu release behind: this way we can provide very recent software and use a release developers are already familiar with (every new release changes something, for example, GCC version, libssl version or default socket path location for databases). 11.10 in particular will let us stop provisioning some stuff (like Maven 3 and some databases) from source/tarballs.

So when 12.04 is out in April, we will move to 11.10. Right now the only blocker for that is #30. I am already doing a lot of testing (roughly half of the time I use 11.10 to develop and test cookbooks) and all new cookbooks and changes to existing cookbooks will have to be tested against both 11.04 and 11.10 starting March 1st (so this week).

FYI: @joshk @mattmatt @svenfuchs @rkh

Build memcached with SASL support

Originally reported by @jaysonsantos in travis-ci/travis-ci#434:

As I talked with @michaelklishin on IRC it would be great if memached is build with sasl support.
We have talked about how ugly is build it with sources and maybe apt-get source should be a better approach as it is the same package as in production.
Here is a simple how to http://code.google.com/p/memcached/wiki/SASLHowto
Thank you for this great project

http proxy for workers

It would be neat if the workers had an env var set with http_proxy=<some travis proxy that supports caching headers>.

The reason I ask is that we at packagist.org get a lot of traffic from travis workers that have to download all the package metadata (5MB and growing) at every build of most php projects. If we had a known IP and it supported caching, it would be easy to add a custom rule in nginx to set Cache-Control: s-maxage: 120 or such. It would probably cut off the traffic and give us some breathing room until we can provide you with an easy enough mirror infrastructure.

Consider adding golang support

http://golang.org/doc/install#freebsd_linux

Pull Go test & xtest dependencies as well as include contained modules

First, thanks for being awesome and adding Go support! Great to have it as a supported language.

The current configuration works well for simple projects that:

1. Define only a single top level package.
2. Don't have dependencies to other local code (that is, depends on other modules in the same github repo). Obviously these also fit into 1.
3. Don't define test/xtest only dependencies.

Here's a annotated script that fixes these issues:

#!/bin/bash

set -e

if [ -z "$GOPATH" ]; then
  echo Expecting GOPATH to be set.
  exit 1
fi

REPO_DIR=$PWD
GITHUB_REPO=$(basename $PWD)
GITHUB_USER=$(basename $(cd .. && pwd))

# this ensures inter-package dependencies use the same version and don't fetch
# the "current" version. symlinks may have issues with ... hence a copy instead.
mkdir -p $GOPATH/src/github.com/$GITHUB_USER
cp -r $REPO_DIR $GOPATH/src/github.com/$GITHUB_USER/$GITHUB_REPO

# fetch dependencies for all contained packages using the ... syntax
echo Fetching package dependencies
go get -v github.com/$GITHUB_USER/$GITHUB_REPO/...

# explicitly fetch the test/xtest only dependencies
echo Fetching test dependencies
TEST_DEPS=$(go list -f '{{.TestImports}} {{.XTestImports}}' github.com/$GITHUB_USER/$GITHUB_REPO/... | sed -e 's/\[//g' | sed -e 's/\]//g')
if [ "$TEST_DEPS" ]; then
  go get -v $TEST_DEPS
fi

# run tests in all contained package using the ... syntax
go test -v github.com/$GITHUB_USER/$GITHUB_REPO/...

Also available here: https://github.com/daaku/go.travis-test-case/blob/master/install
A sample test run which doesn't work with the current stock config: http://travis-ci.org/#!/daaku/go.travis-test-case/builds/1869799
The same test run with the custom script described above: http://travis-ci.org/#!/daaku/go.travis-test-case/builds/1869860

Multiple PHP releases fail to compile on 11.10

Our PHP-related cookbooks use php-build with the following configuration flags.

The list of PHP versions we currently support (and would like to keep supporting) is in our documentation guide on CI environment. Currently several of them (5.2.17, 5.3.10, 5.3.8, 5.3.2) fail to build:

• 5.2 seemingly because of libssl in 11.10 not having SSLv2 enabled by default.
• 5.3.x fail when --enable-intl flag is used

We need help with identifying what extension causes this and if we can work around this by disabling SSL for it (for CI, SSL is much less important than for typical server environments) or fix the issue one way or another. We would prefer to not do things like recompiling libssl (too much maintenance headaches).

This is the last blocker with our 11.10 migration.

Firebird support

It would be nice to have the ability to test against the firebird DBMS as well.
I've started an implementation but my ruby/travis knowledge isn't that high, someone with more experience will most likely have to see whether this is heading in the right direction. This appears to correctly install and enable firebird but needs some more testing.
erikfrerejean/travis-cookbooks@master...firebird

Install a js engine in vagrant_base

Make sure .rvmrc contains the correct flags

Here's what we have on the production worker: https://gist.github.com/170a0dc957f76474ec7a

It should mean:

• users are not allowed to install their own rubies
• local .rvmrc are ignored

I'm not sure that's what these settings actually mean though.

Python 2.5 is not provided in the most recent PPP VM images

Moved from travis-ci/travis-ci#433.

From this build log:

$ export TRAVIS_PYTHON_VERSION=2.5
$ source ~/virtualenv/python2.5/bin/activate
-bash: /home/vagrant/virtualenv/python2.5/bin/activate: No such file or directory
$ python --version
Python 2.7.1+

Worker: php1.worker.travis-ci.org:travis-php-6

Make 1.9.3 the default Ruby

I think it's time to make 1.9.3 the default Ruby version. Obviously it does not mean that 1.8.7 won't be provided, but if a project does not specify the list of Rubies to use, Travis Ruby builder should pick 1.9.3.

1.8.7 is about a year from being completely unsupported. It's time.

Filed per discussion with @joshk (not completely set in stone yet).

Provide multiple JDKs

Our support for multiple runtimes is well known: 11 Rubies, 5 Pythons, 5 OTP releases, 6 PHP versions, etc. JVM languages, however, only have 1 JDK to test against. It would be a killer feature to provide multiple JDKs. For example, if we do so, JRuby will move to Travis. And we all want JRuby on Travis, right?

Installing multiple JDKs is not a problem, although we need to investigate licensing of IBM JDK (J9). JDK 7 .deb packages (the most important JDK at the moment, since JDK 6 is 6+ years old and will be EOL in November this year) are available in Ubuntu 11.10 repositories or multiple other sources.

The hardest part is implementing JDK switcher that uses Debian tools like update-alternatives or update-java-alternatives and also takes care of JAVA_HOME value and so on. We also need it to support aliases because update-java-alternatives aliases are long, irregular and impossible to remember (examples: java-1.7.0-openjdk-amd64 and java-1.6.0-openjdk).

Minimal viable implementation of this feature would include JDK switcher, OpenJDK 6 (available from packages in 11.10) and OpenJDK 7u4+ (available from packages in 11.10).

FYI: @joshk @mattmatt @svenfuchs @rkh

Preinstall ElasticSearch

We agreed that it's time to preinstall ElasticSearch. We won't start it on boot (à la what we have been doing with Neo4J) but it will be much easier for projects just start it than downloading & installing it every time. ES is popular enough by now for us to do it.

Bump stack size limit for SBT

Moved from travis-ci/travis-ci#595

Provision stable PyPy release

We need to evaluate their PPA for starters

Upgrade Redis to 2.4.14 or so

We haven't had a Redis upgrade in a while. fpm-cookery's Redis formula now builds 2.4.14, it's time to build a new package and update our cookbook to use it.

PyPy virtualenv location needs to conform to the python-VERSION pattern

Otherwise it cannot be activated, for example: http://travis-ci.org/#!/arigo/ffi/jobs/1489217/L20

php::multi fails to finish provisioning

New php::multi recipe fails to finish provisioning. I also had to make it install re2c.