lullabot / jenkins_github_drupal

Create a Drupal site from a new Pull Request on Github with Jenkins

Overview

Do you build Drupal sites? Do you use GitHub to manage your code for these projects? Do you use Jenkins to automate continuous integration? Well you should do all of these things! And if you do, then you should also use Lullabot's Magical Jenkins/Github/Drupal Scripts™.

These scripts will build your repository's branches so that you can test them in a full Drupal environment. No more local checkout of a branch, running updates, and synchronizing your database and files in order to test a branch. Just install the scripts and let the machine do it for you automatically every time you create a new pull request.

Brought to you by your friends at Lullabot!

If you'd like to read more, check out this article on Lullabot.com.

Installation

Please see INSTALL.md

Usage

This script should be executed within Jenkins, and will fail otherwise.

First, call the directory preparer, which moves the pull request to your webroot, and merges it into master.

prepare_dir.sh [-himvx] <webroot>

Then, call the site cloning script, which uses drush to clone an existing staging site.

clone_site.sh [-deghHilvxc] <source-drush-alias> <url>

To clean up afterwards, call cleanup.sh using the same pull request ID and location of your webroot.

cleanup.sh [-dhiluvxc]

What does it do?

• Moves the checked out repository to a unique directory in the workspace.
• Creates a symlink to the docroot of the drupal directory in the webroot.
• Creates a new branch from the pull request and merges that branch to master.
• Copies the settings.php from an existing site to this new drupal site.
• Clones the database from the source site, prefixing any tables with a unique identifier to this pull request.
• Rsyncs the files directory from the source site.

Requirements

• Drush
• A web accessible URL for the pull request. The location of the docroot for this URL should be specified with the -l option.
• An existing Drupal 7 site with a site alias, and empty prefix line in the database array in settings.php
• A Jenkins job that checks out the Pull Request to 'new_pull_request' directory inside the job workspace.

Arguments

<webroot>

Location of the parent directory of the web root this site will be hosted at. Defaults to the job workspace. Note, the Jenkins user must have write permissions to this directory.

<source-drush-alias>

The drush alias to the site whose database and files will be cloned to build the pull request test environment.

<url>

The parent URL that the destination site will be visible at. Defaults to 'http://default'. The domain name the site will be set up at. Note, the site will be in a subdirectory of this domain using the Pull Request ID, so if the Pull Request ID is 234, and you pass https://www.example.com/foo, the site will be at https://www.example.com/foo/234.

Options

• -c Optional. Specifies that the full database is cloned instead of using table prefixes.
• -e Optional. Extra settings to be concatenated to the settings.php file when it is cloned. Only used in clone_site.sh.
• -g Optional. The http server's group, such as www-data or apache. This is only used in clone_site.sh to ensure that the file permissions are set properly on the Drupal file directory.
• -h Show this message
• -H The directory to pass to --link-dir during rsync. This is only used in clone_site.sh to use a shared files directory to create hardlinks from. This is useful for sites that have large file directories, to avoid eating up disk space. It is recommended to keep this directory synced regularly with the stage files dir. See man rsync for more details.
• -i The Github pull request issue number.
• -l The location of the parent directory of the web root. Same as <webroot>.
• -m The branch the pull request should be merged to. Defaults to 'master'. This is only used with prepare_dir.sh.
• -d The path to drush. Defaults to drush.
• -u Optional. URIs of the sites to clean up when running cleanup.sh. Useful when there are symlinks in /sites.
• -v Verbose mode, passed to all drush commands.
• -x Turn on bash xtrace and drush debug mode.

jenkins_github_drupal's Issues

Allow specifying alternate settings.php file in drush clone-settings-php command

On some sites we've got settings.php comitted to the repository with immutable settings, and then move things like DB and other environment specific settings to a file like local.settings.php which is then included via a require_once from settings.php.

It would be nice if the drush clone-settings-php command could perform it's magic on a different file by allowing you to pass a different file name as an option to the command. Something like

drush clone-settings-php --settings-file="local.settings.php"

This would also entail updating the clone_site.sh script so that it takes a --settings-file argument as well and passes that along to the drush command.

license

Should we have an OSS license on here? If so which one?

sassy jenkins

One request by our front end team is to make sure that we have the ability to compile sass so that css stays out of the db. ref: https://github.com/Lullabot/sassy-jenkins

Add Behat testing

Is there any d8 fork over there?

git clean -f doesn't remove untracked directories

This can cause problems if for example you have modules that were installed, but then you uninstalled and removed the directories with git rm -rf.

The proper command to remove those directories is git clean -d -f

Lullabot server farm

I wonder if there are any plans to have this setup in a way on our own servers that we can easily utilize it for any of our projects. The ideal situation I would envision as something like:

1. Setup a new project in GitHub
2. Hook it up to Jenkins
   • ... (not sure what all this consists of yet)
3. Profit!

I don't know if this means more Linode servers, or if we would want to spin up amazon instances or what. One thing to consider in regards to this might be trying to replicate whatever project's server environment. Hopefully we could have a fairly typical server setup that would suffice for the purpose of testing branches.

Tear down environment on merge

Maybe I am missing something, but does an environment built for a pull request get teared down if the pull request is merged or closed?

If not, where shall I start looking for to implement it?

Excitement

I just wanted to say that I'm really excited about this project. I've been trying to think of ways to make pull requests more efficient so that we're not all so hesitant to use them and we have more eyes and discussion on the code going into our projects. I think this project will take away the main pain point of having to switch to another branch in order to test it.

Reviewing the code is pretty painless with GitHub, but getting your local code base to a place where you can test the pull request and then switch back to your previous branch can be daunting, especially if there are database changes to be made/rolled back.

With being able to look at someone's branch already built somewhere, there is a huge potential gain in accepting and using this workflow. So thank you James!

Better ReadMe

Could we add a paragraph that gives an example use-case for this project in lay-person speak?

The "what does it do" section lists off the mechanics of what happens, but it's not clear WHY someone would want to use this or HOW it makes life easier for Drupal developers.

Perhaps a "Synopsis", "Description", or "Overview" section at the top?

Honestly, I'm not sure what this does myself, but let me make up a paragraph that would say the kind of thing I'm looking for:

Do you build Drupal sites? Do you use GitHub to manage your code for these projects? Do you use Jenkins to automate continuous integration? Well you should do all of these things! And if you do, then you should also use Lullabot's Magical Jenkins/Github/Drupal Scripts™.

These scripts will {...} so that you can {...}. No more {...}, just install the scripts and {...} will happen automatically every time you {...}.

Brought to you by your friends at Lullabot!

Investigate mr-drupal

@wimleers has an article about using mr to help automate Drupal maintenance. Let's discuss if this is something that might be helpful to incorporate into this project.

After a quick read, it would seem to me that mr is useful for maintenance and deployment, but this project is not so much about deployment as getting new test environments quickly spun up to test branches. So I'm not really sure how this would fit in here if at all.