engineyard / ey-cookbooks-stable-v5

Engine Yard Cloud v5 Chef Recipes

• This codebase and its cookbooks represent the latest version of Engine Yard's stable-v5-3.0 stack.

Dependencies

To upload and run the recipes from the CLI, you need the ey-core gem.

gem install ey-core

Usage

1. Create the cookbooks/ directory at the root of your application. If you prefer to keep the infrastructure code separate from application code, you can create a new repository.

2. For each custom cookbook that you want to use, do the following:

   • Create or edit cookbooks/ey-custom/recipes/after-main.rb and add the line:

   include_recipe 'custom-<recipe>'
   

   • Create or edit cookbooks/ey-custom/metadata.rb and add the line depends 'custom-<recipe>'
     • prepend name 'ey-custom' to cookbooks/ey-custom/metadata.rb in case of a creation
   • Download this repository and copy custom-cookbooks/<recipe>/cookbooks/custom-<recipe> to cookbooks. For example, to use memcached, copy custom-cookbooks/memcached/cookbooks/custom-memcached to cookbooks/custom-memcached.

3. Alternative to step #2 above: use ey-v5-starterkit to automate copying the recipe from the custom-cookbooks directory

4. To upload and apply the recipes, run

   ey-core recipes upload --environment <nameofenvironment> --apply
   

For more information about our V5 (Gentoo 16.06) Stack, please see https://support.cloud.engineyard.com/hc/en-us/sections/205071967-Engine-Yard-Gentoo-16-06

Contributing/Development

Please read our Contributions Guidelines.

ey-cookbooks-stable-v5's Issues

EFS removal cookbook incomplete

EFS removal cookbook doesn't remove the mount point causing issues if enabled again

Redis security

Redis recipe should have protected mode enabled by default, and Redis should listen only on the private IP. The recipe already sets up /etc/hosts and redis.yml to point to the redis instance's private IP.

Wrong recipe installation instruction in custom chef README

A number of custom chef recipes have this in their installation instructions:

ey-core recipes upload --environment <nameofenvironment> --path <pathtocookbooksfolder> --apply

ey-core does not support the path parameter. It throws the error:

The application encountered the following error:
  invalid option: --path=cookbooks

Please remove the --path option on the instructions.

This following recipes need to be fixed:

• cron
• Elasticsearch
• logrotate
• memcached
• packages
• papertrail
• redis
• resque
• resque-scheduler
• thinking sphinx

Configure Sidekiq workers independently

This feature was implemented in this PR: #64, but it was a breaking change.

Let's find a way to implement it so that users can upgrade to the newer version without breaking things.

monit.d directory is not symlinked to /data/monit.d

Stack version: stable-v5-3.0.36
Environment: Solo
Recipes used: memcached, redis

The redis recipe writes its monitrc in /data/monit.d/redis.monitrc
The memcached recipe writes in /etc/monit.d/memcached.monitrc.

By itself this shouldn't be a problem because /etc/monit.d/ is a symlink to /data/monit.d/.

However under certain circumstances - possibly when the memcached recipe is used, /etc/monit.d/ ends up as a directory and not a symlink to /data/monit.d.

This block in the memcached recipe is responsible: https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/memcached/recipes/install.rb#L43-L48

---

Possible workaround (not tested):

Following the approach described in https://github.com/engineyard/ey-cookbooks-stable-v5/wiki/Customizing-Your-Environment-Using-Overlay-Chef-Recipes, create an overlay recipe to override memcached/recipes/install.rb.

• create cookbooks/memcached/recipes/install.rb
• copy the code in https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/memcached/recipes/install.rb but remove the block identified in the bug
• upload the recipe and run chef

Consolidate application restarts into just one restart

After release v5.3.0.33, we'll now have the perform_restart flag in several recipes (env_vars, cdn_distribution). All of these should be consolidated and there should only be one app restart at the very end of the chef run.

New ElasticSearch 5 recipe does not configure limits

I tried to boot up ElasticSearch 5.5.0 from @204 on my EngineYard test environment.

It fails with this message:

[2] bootstrap checks failed
[1]: max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

I noticed that the limits are only applied if ElasticSearch is using version 2, it is defined here:

ey-cookbooks-stable-v5/cookbooks/elasticsearch/recipes/configure_limits.rb

Lines 6 to 7 in e390995

	is_V2 = node['elasticsearch']['version'].match(/^2/)
	if is_V2

Without any changes the /usr/local/bin/monit will look like this:

deploy@ip-172-31-25-112 ~ $ sudo cat /usr/local/bin/monit
#!/bin/sh

# set process limits
ulimit -n 16384

# launch monit with the arguments passed to us
exec /usr/bin/monit $@d

With the conditional negated, the monit file now looks like this:

#!/bin/sh

# set process limits
# NOTE: These changes are needed by Elasticsearch 5.x
ulimit -n 65536
sysctl -w vm.max_map_count=262144

# launch monit with the arguments passed to us
exec /usr/bin/monit $@

And /etc/security/limits.conf does contain the right settings:

*                soft    nofile          65535
*                hard    nofile          65535

elasticsearch    soft    nofile          65536
elasticsearch    hard    nofile          65536

But ElasticSearch continues to fail with the this error, so somehow all of these changes are not applied:

[1] bootstrap checks failed
[1]: max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536]

I think this is because the /etc/security/limits.conf is not applied to running sessions, which meant that after a reboot, the changes are picked up, but there should be a way to do this without requirering a reboot, should there not?

Custom cron recipe improvement - setup cron jobs on app instances

At present the custom cron recipe is only for setting up cron jobs on named utility instances.

For setting up cron jobs on app_master, the EY Cloud dashboard can be used.

There is no facility for setting up cron jobs on app instances, other than writing a custom chef recipe from scratch. We should modify the custom cron recipe to make it easier to setup cron jobs on app instances.

Delayed_Job4 should cleanup older monit configs

As the Sidekiq recipe does the delayed_job4 recipe should clean up for older/existing monit config files.

Whenever jobs cleared by `ey-cron` recipe

We have whenever generated tasks that are generated in the before_restart hook. We've found that these get cleared via a recipe in

ey-cookbooks-stable-v5/cookbooks/ey-cron/recipes/default.rb

Line 15 in 4fe1da3

execute "clearing old crons" do

is "clearing old crons" requiring us to re-deploy after each chef run.
There is no warning of this happening anywhere.
Can it at least be made optional?
Can we force a deploy after each chef run to avoid losing whenever generated tasks?

/etc/hosts is not updated when an instance is added

The ey-hosts recipe generates internal-use hostnames for all instances within the environment. When a chef 'integrate' run is done after an instance is added to the environment this recipe is not called resulting in the new host not being added to the list. Additionally, since db slaves are removed from environment configurations (database.yml) by chef before being removed from the environment an integrate run is done after their removal; this results in removed replicas remaining in the /etc/hosts entries until the next full chef run.

The workaround is to do a full chef run after adding instances or after removing database replicas. The fix is to call the ey-hosts recipe from the integrate run as well.

Add a recipe for redis-sentinel

Please port this V4 recipe: engineyard/ey-cloud-recipes#335

DelayedJob script improvement

We only wait for the bash command that launches DelayedJob: https://github.com/engineyard/ey-cookbooks-stable-v5/blame/next-release/cookbooks/delayed_job4/templates/default/dj.erb#L161

It's possible that the bash command stops and the actual dj process doesn't and its parent becomes PID 1

We should terminate both the bash command and the dj process

Default user for prompt cookbook.

Default user is set as deploy on prompt cookbook this will cause chef to fail if the person isn't using the standard deploy user and will need overlay chef recipe.

cookbooks/prompt/recipes/default.rb

users = ["deploy"]

logentries configure.rb does not allow filenames with dash '-'

In configure.rb, the regex is matching the filename part of the path on (\w+.log). \w does not match on dash (and many other valid filename characters). My workaround is to rename my files with _.

Not sure the script needs to be fixed. Maybe just update the README in custom-le to indicate that? Unless the log file is being written by some 3 party code and it's hard to change. But that seems unlikely.

Solr recipe needs upgraded dev-libs/nss package

The Solr recipe fails with this error:

Expected process to exit with [0], but received '1'
^[[0m---- Begin output of sleep 30 && /data/solr/bin/solr create_core -c default ----
^[[0mSTDOUT:
^[[0mSTDERR: Exception in thread "main" java.lang.InternalError
^[[0m   at sun.security.ec.SunEC.initialize(Native Method)
^[[0m   at sun.security.ec.SunEC.access$000(SunEC.java:49)
^[[0m   at sun.security.ec.SunEC$1.run(SunEC.java:61)
^[[0m   at sun.security.ec.SunEC$1.run(SunEC.java:58)

due to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1332456

I manually upgraded dev-libs/nss to version 3.23 and /data/solr/bin/solr create_core -c default worked after that.

Please update the recipe to install the dev-libs/nss version 3.23 before installing Solr.

DelayedJob4 recipe: worker does not get terminated if monit timeout is less than the DJ timeout

monit timeout is here: https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/delayed_job4/templates/default/dj.monitrc.erb#L4. Default: 60 seconds

worker timeout is set here: https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/delayed_job4/templates/default/dj.erb#L114. Default: 60 seconds

Result: if the DJ worker is unresponsive, it does not get terminated because the monit timeout kicks in first

Quick solution: increase the monit timeout to 90 seconds

no yarn recipe found

I'm trying to follow the instructions provided here to install Yarn, but custom-cookbooks/yarn/cookbooks/yarn is not found in this repo. What am I missing?

# Yarn

This cookbook makes it easy to install any version of `yarn` available in a downloadable form in [https://github.com/yarnpkg/yarn/releases/](https://github.com/yarnpkg/yarn/releases/).

...

3. Copy `custom-cookbooks/timezone/cookbooks/yarn` to `cookbooks/`

cd ~ # Change this to your preferred directory. Anywhere but inside the application

git clone https://github.com/engineyard/ey-cookbooks-stable-v5
cd ey-cookbooks-stable-v5
cp custom-cookbooks/yarn/cookbooks/yarn /path/to/app/cookbooks/

If you do not have `cookbooks/ey-custom` on your app repository, you can copy
`custom-cookbooks/yarn/cookbooks/ey-custom` to `/path/to/app/cookbooks` as well.

## Specifying the Yarn version

The recipe is installing `yarn` version 1.9.4 by default. You may change that by setting the version in `cookbooks/yarn/attributes/default.rb`.

logentries recipe improvement

End-users should be able to add logs to follow simply by adding the path to the logfile in custom-le/attributes/default.rb

Update database.yml pool size in sidekiq recipe

I have a custom recipe that is updating my database.yml file after the sidekiq recipe runs so my database.yml file has pool: #{node['sidekiq']['concurrency']+2}. Before this change I would get frequent ActiveRecord::ConnectionTimeoutError errors.

Im my case, my custom recipe is using sed to find a replace pool: 5 with pool: 7.
command "sed -ibak --follow-symlinks 's/pool: #{node['sidekiq']['concurrency']}/pool: #{(node['sidekiq']['concurrency']+2)}/g' #{db_yaml_file}"

Can this change be made in the main sidekiq recipe so I don't have to have a custom recipe?

Two lines need to be updated:

ey-cookbooks-stable-v5/cookbooks/sidekiq/recipes/setup.rb

Line 44 in ca3fa18

command "sed -ibak --follow-symlinks 's/reconnect/pool: #{node['sidekiq']['concurrency']}\\\n reconnect/g' #{db_yaml_file}"

ey-cookbooks-stable-v5/cookbooks/sidekiq/recipes/setup.rb

Line 46 in ca3fa18

only_if "test -f #{db_yaml_file} && ! grep 'pool: *#{node['sidekiq']['concurrency']}' #{db_yaml_file}"

passenger_monitor grace_time delay is wrong

Passing -w 60 triggers the kill -9 in 15 seconds instead of 60 seconds

https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/passenger5/files/default/passenger_monitor#L67-L86

After enabling the New Relic add-on and running chef it remains in a crashed state

Steps to replicate:

• Create a new environment
• Add the New Relic Infrastructure recipe to the environment recipes
• Do not enable the New Relic add-on for the environment
• Boot the environment

Result:
New Relic will be installed, but will be in a crashed state

Even after enabling the add-on for the environment, New Relic remains in a crashed state

I'm opening this issue to start the discussion on this.

Should the New Relic Infrastructure recipe include a restart of New Relic? This will solve the issue, but the downside is that there will be the New Relic restart that is unnecessary for 99.99% of chef runs.

Should there be a restart_newrelic_flag? It will introduce complexity to the recipe.

Should we just document this in the recipe README?

Do not run memached from the core recipes

Currently memcached is a no-op. There is a pull request to actually install memcached. The new memached recipe has checks, so if default['memcached']['perform_install'] is not set to true, then the whole recipe is effectively a no-op. This is good, but we should remove unnecessary calls to memcached in a future cleanup.

New Papertrail CA bundle next week

Papertrail will be updating papertrail-bundle.tar.gz on Tuesday, May 15. The new SHA-256 checksum will be be208e650e910106bc9d6c954807c875b22cd9fbe005aa59e0aad0ed13b0c6b6, but given that we might continue updating this file from time to time, I'd recommend against simply hard-coding a new checksum. Instead, either omit the checksum entirely, or embed the entire bundle rather than downloading it. Either option ensures that Papertrail is free to update the source without breaking anyone's deployment.

Consider using passenger-config detach instead of kill -9

If a Passenger5 worker is bloated after grace_time seconds, we terminate it with kill -9 here:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/passenger5/files/default/passenger_monitor#L78

We should consider using passenger-config detach instead. This solution was recommended here: brianmario/mysql2#772 (comment)

Sidekiq recipe README has wrong deploy hook

This part of the README recommends a deploy hook for restarting the sidekiq workers:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/custom-cookbooks/sidekiq/cookbooks/custom-sidekiq/README.md#restarting-your-workers

The monit group name in the deploy hook should be consistent with:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/sidekiq/templates/default/sidekiq.monitrc.erb#L8

The deploy hook code should be changed to:

sudo "monit -g #{config.app}_sidekiq restart all"

First run of Sidekiq creates logfiles owned by root

The fix for how Monit on v5 starts Sidekiq processes introduced this issue: the first time the command is run, the logfiles are created as with root as the owner, and then Sidekiq running as deploy can't open them.
The workaround for the time being is to manually run sudo chown deploy: /var/log/engineyard/app/<app_name>/sidekiq*.log*

PHP_ENV not set for cronjobs

The app environment is correctly exposed to cron jobs for Ruby/Rack, but not for PHP. From crontab -l:

RAILS_ENV="production"
RACK_ENV="production"

Just like with web and console scripts, I would expect to see PHP_ENV in there as well.

Sidekiq README mentions Redis is installed in the DB instance

https://github.com/engineyard/ey-cookbooks-stable-v5/tree/next-release/cookbooks/sidekiq#multi-instance-deploys

This is true for v2 and v4 but not for the v5 stack

Deployed Ruby 2.3(.6) has security vulnerabilities

In March this year, Ruby 2.3.7 was released, with multiple vulnerability fixes (see here for complete release notes).

If I'm not mistaken, EY instances using Ruby 2.3.x are still on 2.3.6, based on this source code and on ruby -v on a machine I manage.

Ruby 2.3 EOL is still in the future (2019-03-31) and officially supported by EY, so would you welcome a PR to bump Ruby version as well as Rubygems for 2.3?

Thanks!

V5: passenger_monitor cannot terminate orphan workers in a multi-application environment

passenger_monitor calls passenger-status to look for orphaned passenger workers:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/passenger5/files/default/passenger_monitor#L103

In a multiple-application environment there will be multiple passenger-standalone processes running. Calling passenger-status without a name parameter will result in an error like this, written to /var/log/syslog:

Apr  4 08:44:02 localhost passenger_monitor: It appears that multiple Phusion Passenger instances are running. Please select a specific one by running:    passenger-status <NAME>  The following Phusion Passenger instances are running:  Name                       PID      Description -------------------------------------------------------------------------- pgnOYJGM                   7081     nginx/1.12.1 Phusion_Passenger/5.1.8 keSiecR1                   16255    nginx/1.12.1 Phusion_Passenger/5.1.8
Apr  4 08:44:02 localhost passenger_monitor: Could not read passenger-status [Status:1]
Apr  4 08:44:02 localhost passenger_monitor: It appears that multiple Phusion Passenger instances are running. Please select a specific one by running:    passenger-status <NAME>  The following Phusion Passenger instances are running:  Name                       PID      Description -------------------------------------------------------------------------- pgnOYJGM                   7081     nginx/1.12.1 Phusion_Passenger/5.1.8 keSiecR1                   16255    nginx/1.12.1 Phusion_Passenger/5.1.8
Apr  4 08:44:15 localhost passenger_monitor: Could not read passenger-status [Status:1]
Apr  4 08:44:15 localhost passenger_monitor: It appears that multiple Phusion Passenger instances are running. Please select a specific one by running:    passenger-status <NAME>  The following Phusion Passenger instances are running:  Name                       PID      Description -------------------------------------------------------------------------- pgnOYJGM                   7081     nginx/1.12.1 Phusion_Passenger/5.1.8 keSiecR1                   16255    nginx/1.12.1 Phusion_Passenger/5.1.8

Improve how sidekiq recipe changes database.yml

The sidekiq recipe updates database.yml to set the number of DB connections according to the node['sidekiq']['concurrency'] setting:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/sidekiq/recipes/setup.rb#L44

This is OK as long as there is only one sidekiq process running.

The DB connection pool setting should be set to node['sidekiq']['workers'] * node['sidekiq']['concurrency]'

We provide instructions on how to customize the number of workers here:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/custom-cookbooks/sidekiq/cookbooks/custom-sidekiq/README.md#choose-the-number-of-sidekiq-processes

Make nginx easier to customize

Can we have an custom-nginx recipe, similar to the custom-redis recipe that makes redis easier to customize?

Changes are:

1. nginx recipe will load some variables from attributes, which can be customized from custom-nginx
2. nginx recipe will get its templates from custom-nginx (pass cookbook 'custom-nginx') to template calls in nginx recipes

Memcached and Memcached.yml not present

Currently memcached and memcached.yml aren't present or available as custom chef.

Port eybackup_slave recipe

V4 version: https://github.com/engineyard/ey-cloud-recipes/tree/master/cookbooks/eybackup_slave

Modify Papertrail recipe to specify instances to run on

Most of the custom chef recipes have a line like this:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/master/cookbooks/sidekiq/recipes/setup.rb#L6

so that it is possible to specify the instances to run on just by modifying custom-<recipe>/attributes/default.rb.

The papertrail recipe is not written this way. Can we please modify it to allow specifying instances to run on via custom-papertrail/attributes/default.rb?

[custom-cron] Temporarily empty crontab on apply

While deploying custom-cron, I discovered that the crontab is emptied for a good minute during the chef run.

To verify this, monitor the output of :

$ watch -d -n 0.1 "crontab -l | tee | wc -l"

Then in another shell, run:

$ ey-core recipes upload --environment=YourEnv --file=cookbooks --apply

While a cron job should be written in a way that having a "miss" in the occurrences is not a problem, this behaviour increases the risk of mis-beat unnecessarily IMO.

Reporting this here and I'll also open a ticket at EY support.

Redis recipe: add the option to configure a Redis slave instance

Please modify the Redis recipe to support booting a Redis slave instance. Add the SLAVEOF directive in the configuration file for the Redis slave instance.

redis force_upgrade doesn't work when moved from source to package installation

I have redis installed with custom-redis cookbook with attributes:

redis['install_from_source'] = true
redis['version'] = '2.4.6'

After that I decided to upgrade redis and change attributes to:

redis['install_from_source'] = false
redis['version'] = '3.2.0'
redis['force_upgrade'] = true

I upload recipes and aplly via dashboard but version remain unchanged and redis binaries in /usr/local/bin are old

Make the puma restart sleep timeout configurable

On this line: https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/puma/templates/default/app_control.erb#L127

Get the sleep timeout from the puma recipe's attributes/puma.rb file

This way it can be easier to customize how puma behaves simply by creating an overlay recipe that changes attributes/puma.rb

Monitor sidekiq orphan workers

Please port this V4 patch to the sidekiq recipe: engineyard/ey-cloud-recipes#343

Sidekiq should redirect STDOUT to the log file

The newer monit version on v5 can cause broken pipe errors when Sidekiq attempts to write to STDOUT. To avoid this we should redirect STDOUT to the log file.

Change this line:
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/cookbooks/sidekiq/files/default/sidekiq#L69

from

sudo -u "${app_user}" -E -H "${sidekiq}" -d -C "${conf_file}" -e "${rails_env}" -r "${app_root}" -P "${pid_file}" -L "${log_file}" -i ${worker_id} &

to

sudo -u "${app_user}" -H "${sidekiq}" -C "${conf_file}" -e "${rails_env}" -r "${app_root}" -P "${pid_file}" -L "${log_file}" > ${log_file} 2>&1 &

Update download URL for Elasticsearch recipe

The download URL for the 5.x.x series is:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.x.x.zip

The download URL for the 2.4.x series is:
https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.4.4/elasticsearch-2.4.x.zip

Please update the elasticsearch recipe with the correct URLs. Also update it so that it obtains the download URL from custom-elasticsearch, this way customers can easily adjust in case the URL changes in the future.

custom-delayd_job4 example error: Cookbook delayed_job4 not found

I have just copied recipes ey-custom and custom-delayed_job from delayed_job4 example

When I try to upload and apply recipes I received an error: Cookbook delayed_job4 not found. If you're loading delayed_job4 from another cookbook, make sure you configure the dependency in your metadata

delayed_job4dependency is in custom-delayed_job4 metadata file

Is there sth i did wrong? Or the example is not working?

Missing http2 variable from nginx recipes

Looks like there is a missing :http2 variable from nginx/recipes/default.rb at line 100 as proxy.conf template checks for http2.

managed_template "/data/nginx/common/proxy.conf" do
  owner node['owner_name']
  group node['owner_name']
  mode 0644
  source "common.proxy.conf.erb"
  variables({
    :use_msec => use_msec
  })
  notifies node['nginx'][:action], resources(:service => "nginx"), :delayed
end

Testing with HTTP2 enabled seems to show that configuration for https is missing from proxy.conf

proxy_set_header  X-Forwarded-Proto "https";
proxy_headers_hash_bucket_size 128;