etsy / mod_realdoc

Apache module to support atomic deploys - http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

mod_realdoc is an Apache module which does a realpath on the docroot symlink and sets the absolute path as the real document root for the remainder of the request.

It executes as soon as Apache is finished reading the request from the client.

The realpath frequency can be adjusted in httpd.conf using:

<IfModule mod_realdoc.c>
    RealpathEvery 2
</IfModule>

By resolving the configured symlinked docroot directory to an absolute path at the start of a request we can safely switch this symlink to point to another directory on a deploy. Requests that started before the symlink change will continue to execute on the previous symlink target and therefore will not be vulnerable to deploy race conditions.

This module is intended for the prefork mpm. Threaded mpms will incur race conditions.

Compile and install with:

apxs -c mod_realdoc.c
sudo apxs -i -a -n realdoc mod_realdoc.la

or just

make install

Please note that if you're granting access to your document root using a symlink, that will stop working, unless expanding the symlink also in your Apache configuration.  If, for instance, your symlink is named "current" and your releases are something like "releases/20160519102200" (with "current" pointing to last release), you need to adapt your configration. 

# Old configuration (without mod_realdoc):
<Directory "/var/www/your_host/current">
    Require all granted
</Directory>

# New configuration (with mod_realdoc):
<Directory ~ "/var/www/your_host/releases/\d{14}">
    Require all granted
</Directory>

You need to adapt the regular expression \d{14} if you use a different schema from timestamp in your releases.

If you want to map incremental releases to two static docroot symlinks in order to re-use your opcache cache entries, you can set

UseReadlink On

This means that instead of calling realpath() it will call readlink() on the first symlink it finds in your configured docroot path. This means you can do:

                                  /var/www/release-11
/var/www/current -> /var/www/A -> /var/www/release-12
                    /var/www/B -> /var/www/release-13

Then when release-13 is ready to go live, flip the symlink to B. For PHP you are going to need the resolve_symlinks patch in order to only have /var/www/A and /var/www/B opcache entries. Without that patch PHP will call realpath() and you will have /var/www/release-* cache entries which means no cache re-use.

There is also a small optimization. Instead of just turning it on, you can tell it where to start checking for symlinks from in your docroot path like this:

UseReadlink /var/www/current

this saves a couple of lstat() syscalls.

mod_realdoc's Issues

Change URL in description

Current URL doesn't work.

Change it to http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

VirtualHost/DocumentRoot conflict when using mod_realdoc

We've tried deploying mod_realdoc and phpinc to production but have run across a problem with requests being handled by the wrong VirtualHost. Each VirtualHost has a unique DocumentRoot and port, the Apache configs are similar to:

<VirtualHost *:80>
    DocumentRoot /var/www/proj/env/docroot1
    ServerName secure.hostname.tld
    UseCanonicalName On

    <Directory /var/www/proj/env>
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>

    <IfModule mod_php5.c>
        # The include_path replaces "/var/www/proj/env/docroot1" with DocumentRoot
        # when the php extension incpath is enabled, therefore the path must go up
        # one level, i.e. "/..". The initial include_path is also appended in case the
        # php extension gets disabled.
        php_value include_path ".:/var/www/proj/env/docroot1/../legacy/classes:/var/www/proj/env/HEAD/legacy/classes"

        # php incpath setting to resolve the correct DocumentRoot
        php_admin_value incpath.search_replace_pattern "/var/www/proj/env/docroot1"
    </IfModule>
</VirtualHost>

<VirtualHost *:24380>
    DocumentRoot /var/www/proj/env/docroot2
    ServerName secure.hostname.tld
    UseCanonicalName On

    <Directory /var/www/proj/env>
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>

    <IfModule mod_php5.c>
        # The include_path replaces "/var/www/proj/env/docroot2" with DocumentRoot
        # when the php extension incpath is enabled, therefore the path must go up
        # one level, i.e. "/..". The initial include_path is also appended in case the
        # php extension gets disabled.
        php_value include_path ".:/var/www/proj/env/docroot2/../legacy/classes:/var/www/proj/env/HEAD/legacy/classes"

        # php incpath setting to resolve the correct DocumentRoot
        php_admin_value incpath.search_replace_pattern "/var/www/proj/env/docroot2"
    </IfModule>
</VirtualHost>

Deployments are done via rsync with the --link-dist option, deployments reside in /var/www/proj/env/hash with symlinks created to point to the new HEAD:

$ ls -al /var/www/proj/env/
drwxr-xr-x 72ac812
lrwxrwxrwx docroot1 -> 72ac812/docroot1
lrwxrwxrwx docroot2 -> 72ac812/docroot2
lrwxrwxrwx HEAD -> 72ac812

DocumentRoot used to be HEAD (e.g. DocumentRoot /var/www/proj/env/HEAD/docroot1) but was changed to use the {docroot1,docroot2} links in case they fixed the problem

A couple of seconds after reloading the server, lots of 404 and 'File does not exist' errors are logged. The detailed error suggests that requests going to :80 are being resolved to the DocumentRoot served by :24380: File does not exist: /var/www/proj/master/ff8c48a/docroot2/js. Unloading mod_realdoc (by removing realdoc.conf) and reloading apache instantly stops more errors from happening.

This is happening on a CentOS 6.x / Apache 2.2.15 (prefork) server. Throughout is about 850rpm.

Apache 2.4 - Can't locate API module structure

Will this work with Apache 2.4? On CentOS, I'm getting the following:

Can't locate API module structure `realdoc' in file /etc/httpd/modules/mod_realdoc.so: /etc/httpd/modules/mod_realdoc.so: undefined symbol: realdoc

[root@centos-linux src]# ls -al /etc/httpd/modules/mod_realdoc.so
-rwxr-xr-x. 1 root root 7560 Oct 5 10:46 /etc/httpd/modules/mod_realdoc.so

File exists, but doesn't load.

I tried various different names passed into the apxs command to install it, as well as matching those in the LoadModule portion.

Removed the '-a' from the install command due to error about 'at least one LoadModule must exist' (on CentOS they are handled via the /etc/httpd/conf.modules.d directory include). Tried putting a dummy LoadModule in so that apxs would generate the LoadModule statement, which did work, but Apache still failed to start with the same error.

Thanks

Potential race-condition when using mod_realdoc in a threaded/asynchronous environment

r->server is shared between all requests handled by a httpd process on the same VirtualHost (at least in httpd 2.2), therefore changing it in a threaded/asynchronous environment may lead to unexpected race-conditions, where one request has changed the document root after another request has read it for translating to script filename.

mpm_worker - will be affected
mpm_event - I'm unsure how much of the request is handled asynchronously

The scenario that might be possible is:

Request 1: in mod_realdoc post_read_request hook, loads a cached resolved document root
Request 1: runs it's translate_name hooks
Request 2: in mod_realdoc post_read_request hook, finds cache is expired, so saves a new document root entry
Request 2: runs it's translate_name hooks
Request 1: sends the new DOCUMENT_ROOT and old SCRIPT_FILENAME to page processor (e.g. php)
Request 2: sends the new DOCUMENT_ROOT and new SCRIPT_FILENAME to page processor (e.g. php)

The only real solution I believe is to use translate_name to only update the uri to filename handling when in these modes. This wouldn't change the document root however.

Access denied

As soon as I enable mod_realdoc all my virtual hosts that use a symlinked document root stop to work, giving a 403 error, like the following:

[Thu May 19 09:00:41.113107 2016] [authz_core:error] [pid 26472] [client 66.249.78.105:51750] AH01630: client denied by server configuration: /full_path/releases/20160506140516/web/app.php

where releases/20160506140516 is symlinked from current and virtual host has document root pointed to /full_path/current/web/ and the following configuration:

    <Directory "/full_path/current/web">
        Require all granted
    </Directory>

Apache is 2.4.20 on Ubuntu 14.04.4

Can't install because mod_realdoc.la is missing

I found the README installation instructions a bit misleading.

As written, the README seemed to imply that "make install" was equivalent to doing both apxs commands, not just the second one:

Compile and install with:

apxs -c mod_realdoc.c
sudo apxs -i -a -n realdoc mod_realdoc.la

or just

make install