codeenigma / deployments Goto Github PK

Worth a look:
https://www.securify.nl/blog/SFY20150303/automating-security-tests-using-owasp-zap-and-jenkins.html

We've had a request to make sync jobs configurable and generic via the config ini / yaml file, the idea being you should be able to specify details of your various MySQL configs (probably their cnf files with connection details in, host and path) and the scripts should use those to sync databases from %env to %env.

Similarly, we should be able to set volatile asset directory paths and sync from %env to %env without making assumptions about locations of things.

Nice to have to be able to save a Sucuri API secret in Jenkins credentials and use it to clear the cache at the end of a build.

As part of starting to look into #210 I discovered that Fabric 2 basically isn't finished. And unless you're already a heavyweight Python coder it's really hard to do anything useful, because none of the 'glue' from Fabric 1 has been ported.

Enter Robo.li. For one, we're at a company full of PHP developers, and secondly it's really easy to use and really well documented.

So we're going to change tack, Fabric is dead (long live Fabric!) and I'm going to look at porting to Robo.li instead. This issue closes #210.

We have a big block of code like this at the top of drupal/fabfile.py:

  # Now let's fetch alterations to those defaults from config.ini, if present
  if config.has_section("Build"):
    print "=> We have some build options in config.ini"
    # Provide the path to an alternative deploy key for this project
    if config.has_option("Build", "ssh_key"):
      ssh_key = config.get("Build", "ssh_key")
      print "===> path to SSH key is %s" % ssh_key
etc.

We could do with a config loading wrapper that gets passed the config file, the config item to look up and an optional variable for those configs such as url that may have been passed in via the Jenkins execute build box. If it gets a value already, it doesn't change it - if it doesn't get a value, it looks in config.ini for one. If it finds one it returns it, otherwise it returns None.

That way we could call configs we need directly in function calls instead of having a big block of horrid code at the top of each main() function. A good example of this is in the same file with Emlyn's feature branching code:

  execute(InitialBuild.initial_build_vhost, repo, url, branch, build, alias, buildtype, FeatureBranches.ssl_enabled, FeatureBranches.ssl_cert, FeatureBranches.ssl_ip, FeatureBranches.httpauth_pass, FeatureBranches.drupal_common_config, webserverport)

It's loading the feature branch configs it needs inline in the function call, no need to pre-define them.

This would be much neater if things like ssh_key, for example, instead of being loaded in a big block at the top and then only used once, could be fetched with common.Utils.get_config(config, "ssh_key") when you need it.

If we run CodeSniffer from bash instead of trying to find code changes with a Drupal module, it can be used to check Drupal, but also other open source apps - and it can be moved to 'common'.

Everything that's reusable in DrupalTests for behat should move to common, as behat isn't just a Drupal thing! We should keep the module disabling and enabling behaviour separated and wrap the common behat functions with some Drupal behat functions that handle that, but the core of behat support should also work for other platforms.

At the moment the multisite loop causes the symlink for live to be removed and recreated several times (once for each site installed/updated). It should be moved outside of the multisite loop so it happens once.

However, this is tricky because several functions that occur after it use drush aliases to run, which requires the symlink to be correct. So we need to refactor all code using drush aliases so it actually uses with cd('/path/to/drupal'): drush blah instead of drush @alias_name blah so we can safely process the symlink move later on in the script.

When the backup parameter is passed into the script from a Jenkins job, it's passed in as a string rather than a boolean (this actually applies to all parameters).

So when backup=False is passed, the backup variable is set. That means, if when it's set to False, the following if statement is satisfied:

The execute(common.MySQL.mysql_backup_db, db_name, 'drush_command', True) call fails to run because the app_primary role hasn't been defined:

So, the solution here is we need ensure the backup parameter is a boolean and we need to define the app_primary role. I do not think we can use the common.Utils.define_roles() function to do this because a config.ini file does not exist. I think we can simply set that role in the fabfile-drushcommand.py script.

We have the ability to set a URL template in config.ini, like this:

[featurebranch]
urltemplate=reponame.branchname.example.com

In FeatureBranches.py, we set the url variable like so, if the urltemplate option is present. But before that, we define the global url variable:

However, we do not pass in the correct variable to be used here. We need to pass in the alias variable.

Plus, I'm not sure if the URL set in the FeatureBranches.py function will stick as back in fabfile.py, the url variable is then generated:

I'll run some tests once I've fixed the variable being passed in issue.

This just happened on a client build:

sudo: mysql --defaults-file=/etc/mysql/debian.cnf -e 'GRANT ALL ON `DATABASE`.* TO "USER"@"172.30.0.30" IDENTIFIED BY "PASSWORD"'
out: Could not open required defaults file: /etc/mysql/debian.cnf
out: Fatal error in defaults handling. Program aborted

It didn't fail, nor did it fail later:

===> Installing Drupal with MySQL string of mysql://USER:PASSWORD@RDS_URL/DATABASE
run: drush si minimal -y --db-url=mysql://USER:PASSWORD@RDS_URL/DATABASE
out: You are about to CREATE the 'DATABASE' database. Do you want to continue? (y/n): y
out: Failed to create database: ERROR 1045 (28000): Access denied for user    [error]
out: 'USER'@'172.30.0.30' (using password: YES)

It carried right on until it eventually failed trying to alter the user 1 password.

At the moment we check for the existence of a site with a drush sa command to list the available sites. However, with drush increasingly shipping with Drupal via composer and that being the desired behaviour for developers, the idea of a system-level drush, thus also system-level drush aliases, is going away.

While we can always install a system drush and figure out a way to enforce its use for alias checks, it will be an oddity in the future. The path of least resistance feels like refactoring this behaviour so it searches for sites based on link existence instead of drush site aliase presence.

Feedback from several developers has been the INI format is obsolete and people are generally more familiar with YAML these days, so it would benefit the uptake of the scripts if config.ini became config.yml.

Obviously we'll have to support both for a time, with a view to ultimately phasing out the INI type.

If the mysql_version in config.ini is set, it is not compared correctly when evaluated here:

For example, in the 8.x-multisite branch of the example-drupal repo, I had:

[Database]
mysql_version=5.5

However, during an initial build of that branch, I saw the following in the Jenkins console output:

===> MySQL version is 5.5, setting database name length to 60 and username length to 32.

Which is clearly wrong, as per the code above. That value has been evaluated as greater than 5.6, when it is not.

The ultimate purposes of this is to allow Symfony projects to have feature branching.

At the moment we can only seed databases with WordPress and Drupal. Symfony presents a challenge, as by its nature the apps tend to be bespoke, so we can't know what the parameters.yml.dist attributes for the db name, username and password will be in advance, nor can we create them and safely add them to the parameters.yml include we generate.

I think, in order to achieve this, we'll need to use the database generation code (which is already in common/MySQL.py) and then have some clever way of allowing developers to tell us which attributes are to be used for the different generated database attributes, which we'll then have to pipe into the correct attributes in the generated parameters.yml file in /var/www/config - it won't be trivial!

Be a good thing to sort out though, if we do it right it'll probably help us further abstract database handling, which is only a good thing.

While not common, it can happen that a static reset 'wipes' some configuration (held in settings.php), and it needs to be re-included.
Typically SimpleSamlPHP integration can break when the idp database settings can't be found.

Switching the 'include_once' to 'include' ensures all settings are always properly loaded, and I can't think of any drawback (the main settings.php is anyway very rarely included more than once).

MR on its way.

This is something of a WIP, but we're looking at supporting Selenium via Behat in the long run. We have Behat tests up and working (though they need a little refactoring). The next step is finding a consistent way to make Selenium tests work headlessly.

Take a look at this code:

Instead of having various symlink functions for handling different scenarios, we could have a generic create_symlinks() function that just takes a list of links it needs to create from config.ini/yml and makes them. That way it's platform/application agnostic, and we can have Code Enigma 'defaults' for Drupal, Magento, WordPress, etc.

On a build for a new customer we saw the secure_admin_password() function failed with the following errors:

20:54:13 ===> Setting secure username and password for uid 1 on training_toolkit site...
20:54:13 [dev4.int.codeenigma.net] run: drush sqlq "UPDATE users SET name = 'XXXXXXXXXXXXXXXXXXX' WHERE uid = 1" 
20:54:13 [dev4.int.codeenigma.net] out: Query failed.                                                            [error]
20:54:13 [dev4.int.codeenigma.net] out: 
20:54:13 
20:54:13 Warning: run() received nonzero return code 1 while executing 'drush sqlq "UPDATE users SET name = 'XXXXXXXXXXXXXXXXXXX' WHERE uid = 1"'!
20:54:13 
20:54:14 
20:54:14 [dev4.int.codeenigma.net] run: drush upwd XXXXXXXXXXXXXXXXXXX --password='XXXXXXXXXXXXXXXXXXX'
20:54:14 [dev4.int.codeenigma.net] out: exception 'Drush\User\UserListException' with message 'Unable to find    [error]
20:54:14 [dev4.int.codeenigma.net] out: a matching user for XXXXXXXXXXXXXXXXXXX.' in
20:54:14 [dev4.int.codeenigma.net] out: /opt/drush/8/vendor/drush/drush/lib/Drush/User/UserList.php:114
20:54:14 [dev4.int.codeenigma.net] out: Stack trace:
20:54:14 [dev4.int.codeenigma.net] out: #0 /opt/drush/8/vendor/drush/drush/lib/Drush/User/UserList.php(15):
20:54:14 [dev4.int.codeenigma.net] out: Drush\User\UserList::getFromParameters('XXXXXXXXXXXXXXXXXXX...')
20:54:14 [dev4.int.codeenigma.net] out: #1 /opt/drush/8/vendor/drush/drush/commands/user/user.drush.inc(355):
20:54:14 [dev4.int.codeenigma.net] out: Drush\User\UserList->__construct('XXXXXXXXXXXXXXXXXXX...')
20:54:14 [dev4.int.codeenigma.net] out: #2 [internal function]: drush_user_password('XXXXXXXXXXXXXXXXXXX...')
20:54:14 [dev4.int.codeenigma.net] out: #3 /opt/drush/8/vendor/drush/drush/includes/command.inc(422):
20:54:14 [dev4.int.codeenigma.net] out: call_user_func_array('drush_user_pass...', Array)
20:54:14 [dev4.int.codeenigma.net] out: #4 /opt/drush/8/vendor/drush/drush/includes/command.inc(231):
20:54:14 [dev4.int.codeenigma.net] out: _drush_invoke_hooks(Array, Array)
20:54:14 [dev4.int.codeenigma.net] out: #5 [internal function]: drush_command('XXXXXXXXXXXXXXXXXXX...')
20:54:14 [dev4.int.codeenigma.net] out: #6 /opt/drush/8/vendor/drush/drush/includes/command.inc(199):
20:54:14 [dev4.int.codeenigma.net] out: call_user_func_array('drush_command', Array)
20:54:14 [dev4.int.codeenigma.net] out: #7 /opt/drush/8/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(67):
20:54:14 [dev4.int.codeenigma.net] out: drush_dispatch(Array)
20:54:14 [dev4.int.codeenigma.net] out: #8 /opt/drush/8/vendor/drush/drush/includes/preflight.inc(66):
20:54:14 [dev4.int.codeenigma.net] out: Drush\Boot\BaseBoot->bootstrap_and_dispatch()
20:54:14 [dev4.int.codeenigma.net] out: #9 /opt/drush/8/vendor/drush/drush/drush.php(12): drush_main()
20:54:14 [dev4.int.codeenigma.net] out: #10 {main}
20:54:14 [dev4.int.codeenigma.net] out:

This site uses a prefix for the Drupal tables, drupal_ in this case. However, the command we run to update the username for uid/1 is:

run('drush sqlq "UPDATE users SET name = \'%s\' WHERE uid = 1"' % u1name)

In this example, users doesn't exit. It's drupal_users. As a result, we then can't update the password for that user.

So, we need a way to determine if there's a prefix for the Drupal tables. It's not a feature we ever use, hence not noticing until now.

Probably not a high priority, as only affects updating user 1's password.

Pretty much what it says on the tin. This isn't a priority (particularly for us at CE, as we have Nagios checking free disk space), but if Jenkins were to do a quick disk space check to make sure $freespace is available before a deployment, it could help avoid a disaster further down the line.

From a customer:

Changes in the output of drush status in Drush 9 have broken the teardown script.

This isn't super-urgent for us, as we're probably going to back away from Drush 9 for now, it's still buggy. However, it will need sorting at some point.

Details of the issue are here: drush-ops/drush#3076

The teardown script fails at this line as a result:

dbname = sudo("drush status | grep \"Database name \" | cut -d \":\" -f 2")

We have also noticed the code before you call the obfuscate Ruby script parses drush output without using --format so that will break with Drush 9 too.

So we probably need to address this sooner rather than later!

While we use Drupal's built in maintenance mode, this isn't suitable for other types of projects. It would be instead easier to use a 'flat' html page (path overridable in config.ini with a generic default (?)), that gets switched off/on, presumably at the vhost level.

This would make the deployment process suitable for any kind of stack, and prevent having to cope up with any Drupal changes going forward.

Where applicable, we need to make it so all deploy jobs check, early on, if the provided SSL certificate there on the target web server. If not, it should not even attempt to start a build. Remember, this needs to work for Apache and Nginx.

We've had instances of builds stopping web servers because of missing SSL certificates, which isn't good!

Some servers might have multiple versions of PHP installed, and we might want to specify the location of php.ini so different builds can use different versions with the CLI. PHP provides an environment variable for this, PHPRC, so we can do something like this:

def main(php_ini_file=None):
  # usual build stuff

  if php_ini_file:
    run("export PHPRC='%s'" % php_ini_file)

  # do all the CLI stuff

  if php_ini_file:
    run("export PHPRC=''")

  # restart services, etc.

Internal ticket reference r30298.

/var/www is explicitly defined as the webroot throughout these scripts.

This is fine most of the time, but does cause issues with the remove_old_builds script as on our server /var/www is a symlink to /srv/www and this causes the find command in that script to fail.

Ideally, we could pass the webroot in as a parameter (with /var/www as a fallback).

I'll try and raise a PR for this if/when I get time, but there are about 300 or so instances that will need changing!

Related to #22 we'll need to allow people to alter these Linux usernames and groups. For example, Red Hat uses 'apache' as the web server user, so our 'www-data' assumption breaks.

As Composer is increasingly becoming the way of building PHP projects (Magento, SimpleSAMLphp, Drupal > 7, Symfony, etc.) it makes sense to make our Composer bits central and commonly available.

We can detect the inclusion of Drush CMI Tools in a project as follows:

drush @site_alias help --format=yaml | grep "config-import-plus"

If that grep returns results then the Drush module is installed, and if it is installed we can safely assume it should be used and drush cimy should be preferred over drush cim.

As suggested internally, most of Sync.py could actually be common to other jobs. Worth exploring... would be better not to have to maintain sync scripts per application type.

When the autoscale cluster is rebuilt or a new EC2 instance is added, the webroot is now put into /var/www/project_branch_build_0, rather than just /var/www/project.

This means we don't need to remove the files at sites/default/*.settings.php during a deployment, nor should we. We had a case whereby the first deployment after the cluster had been rebuilt failed, so when Jenkins reverted the include_once() statement in settings.php, it referenced files that no longer existed, which resulted in a site that was unavailable.

It would be handy to have the ability to run a regex check on completion of a build (mainly production builds). If said regex isn't found, either the build could be reverted (though I'm not sure about this) or a hard-to-miss message displayed in the console output...maybe even an email being sent to a predefined address.

An example of hard-to-miss text would be:

################################
################################
################################

REGEX TEST FOR {site_name} FAILED! MANUAL CHECK REQUIRED.

################################
################################
################################

Or something like that.

When a build fails and the per-buildtype settings.php config in settings.php needs to be reverted, Revert.py currently uses the branch name rather than buildtype. If the repo has been setup whereby the branch and buildtype values don't match (i.e the production branch is master but the buildtype is prod), the correct overriding file is not included.

A fix for this will require updating the _revert_settings() function in Revert.py, along with making sure any function that function is called from has the buildtype parameter passed to it.

There's a lot of repeatable stuff, particularly in Drupal's InitialBuild.py, that should be in 'common'. Things like vhost handling, config directory checking, etc. We should make sure as much is shared across scripts as possible...

It's entirely feasible someone might use these scrips and not use Jenkins, so actually the CI username should be something you can configure which defaults to 'jenkins' - otherwise folk using other systems, or just bash scripts, will be obliged to create a 'jenkins' user.

I've noticed some teardowns failing when drush status returns non-zero.
The code falls back to trying to extract the database name from settings.php:

But that never succeeds because it greps for a database name based on the alias. However database names rarely match the alias because their names are deliberately truncated on creation:

A fix to get it to grep for the right thing would be complicated, as we'd need to know what the truncation was.

As a slight aside, in this case even though drush status was returning non-zero, it was still outputting the "Database name" correctly. So perhaps a good first fix would be not to immediately fail on non-zero, but rather check the output first.

We would like Symfony developers to be able to build spontaneous feature branches, as Drupal developers already can.

Parallel install of packages with composer:

https://github.com/hirak/prestissimo

We could install that before composer install to make builds a lot quicker! :-)

Referring to this code in common/Tests.py

# We cannot make phpunit work with PHP 5.x, too many problems
# Detect PHP version
php_version = run("php -v | head -1 | awk {'print $2'} | cut -d. -f1,2")
if any(version in php_version for version in [ '5.3', '5.4', '5.5', '5.6' ]):
  # Sorry, PHP too old!
  print "##### Sorry, this version of PHP is too old for current phpunit builds, we cannot run the tests."
  return phpunit_tests_failed

What are the issues getting this to run? I've able to do this locally with PHP 5.6 by running a command like cd www && ../vendor/bin/phpunit -c core modules/custom and ideally I'd be able to run them during the build to identify and regressions and fail the build automatically if a test fails.

Would be nice to provide a LetsEncrypt certificate automatically for SSL as part of the initial build behaviour, if SSL is required and no specific certificate is specified.

If we do this then we can use database backups across all build types, as long as the backend database is MySQL.

This should be renamed DBHost in config and not reference Drupal.

At the moment the output looks like this:

sudo /home/jenkins/revert -b  -d   -s /var/www/live.website.live -a website_live

When it should be:

sudo /home/jenkins/revert -b /var/www/website_branch_build_109 -d /home/jenkins/dbbackups/website_prior_to_build_110.sql.gz -s /var/www/live.website.live -a website_branch

We automatically add Environment Indicator colours if the module is present and turn it on. However, it has been noted the variables are injected into settings.inc after the per-buildtype settings include, so they cannot be overridden. This is not desired, it should be possible to override in branchname.settings.php in the repository.

We should be treating this as a proper integer, so that instead of things like drupal_version == '8' we should be able to do things like drupal_version > 7 - otherwise we'll have unnecessary complications / problems with Drupal 9.

It's the big one! But we got to do it... :-)

I'll set up a feature branch for this in the mainline repo and also get Python 3/Fabric 2 installed on a staging server so we can start tackling this.

Config in D8 can be quite tricky at times, but from my research, Drush CMI Tools offers more flexibility. At least one client wants to try it out and I wouldn't be surprised if others want to try it if there's the option there for it.

Historically we've depended on Puppet to put the necessary Nginx and Apache configurations in place. We should change this so the deploy scripts place the configs if they're not there. We should make the vhost config location a variable while we're at it, and put these configs in the same directory.

When the feature branch functionality was initially built, it was primarily done for Nginx webservers. However, the same level of functionality should be made available for Apache webservers.

For example:

• The ability to add HTTP auth to the feature branch being deployed
• The ability to use SSL on the new feature branch

The solution will be to create dummy Apache vhosts, much like the Nginx ones, which can be used during the initial deployment of a feature branch.

If you have a multisite architecture and one of the sites breaks, the script will stop and revert the database and settings for the site that failed but not any others it previously built in the process.

We'll need to somehow build a list of already 'treated' sites in a build and go back and revert each one in turn in the event of a failure further down the line.

Also, when drush updates fail for multisites it does not revert the link to the previous build directory.

So there are a few logical flaws to overcome for rolling back multisite builds. Should probably be tackled as a single refactoring job of revert behaviour to make it handle multisites and associated generic links.

If the server layout involves more than one app server, should the build fail, the live symlink will only be adjusted on the primary app server. The same applies to when the settings.php file is reverted to include the $buildtype.settings.php file from the previous working build.

Take the config_import() function as an example:

That function is only run on the primary app server, so when Revert._revert_settings(repo, branch, build, site, alias) is run, it's only run on what has been designated the primary app server.

So we need a way to run the revert functions on all app servers, if there are more than one.