Shell scripts to manage database backups, asset backups, file permissions, asset syncing, cache clearing, and database syncing between Craft CMS environments
There are several scripts included in craft-scripts
, each of which perform different functions. They all use a shared .env.sh
to function. This .env.sh
should be created on each environment where you wish to run the craft-scripts
, and it should be excluded from your git repo via .gitignore
.
Craft-Scripts works with both Craft 2.x & Craft 3.x, and has built-in support for both mysql
as well as postgres
databases.
To upgrade to a later version of Craft-Scripts, replace the contents of your scripts
folder with the newest Craft-Scripts, while preserving your existing .env.sh
file.
Craft-Scripts comes with defaults so that even older .env.sh
files should work with the latest Craft-Scripts.
The set_perms.sh
script sets the Craft CMS install file permissions in a strict manner, to assist in hardening Craft CMS installs.
See Hardening Craft CMS Permissions for a detailed writeup.
Note: if you use git
, please see the Permissions and Git section below.
The clear_caches.sh
script clears the Craft CMS caches by removing all of the craft/storage/runtime/
cache dirs, as well as emptying the craft_templatecaches
db table.
It can also clear Redis db caches if LOCAL_REDIS_DB_ID
is set, and it can clear FastCGI Cache if LOCAL_FASTCGI_CACHE_DIR
is set.
If you want to add this to your Forge / DeployBot / Buddy.works deploy script so that caches are auto-cleared on deploy, set up the .env.sh
on your remote server(s) and then add this to your deploy script:
scripts/clear_caches.sh
The above assumes that the current working directory is the project root already.
The pull_db.sh
script pulls down a database dump from a remote server, and then dumps it into your local database. It backs up your local database before doing the dump.
The db dumps that craft-scripts
does will exclude tables that are temporary/cache tables that we don't want in our backups/restores, such as the templatecaches
table.
See Database & Asset Syncing Between Environments in Craft CMS for a detailed writeup.
The pull_assets.sh
script pulls down an arbitrary number of asset directories from a remote server, since we keep client-uploadable assets out of the git repo. The directories it will pull down are specified in LOCAL_ASSETS_DIRS
It will also pull down the Craft userphotos
and rebrand
directories from craft/storage
by default. The directories it will pull down are specified in LOCAL_CRAFT_FILE_DIRS
See Database & Asset Syncing Between Environments in Craft CMS for a detailed writeup.
The pull_backups.sh
script pulls down the backups created by craft-scripts
from a remote server, and synced into the LOCAL_BACKUPS_PATH
For database backups, a sub-directory REMOTE_DB_NAME/db
inside the REMOTE_BACKUPS_PATH
directory is used for the database backups.
For asset backups, a sub-directory REMOTE_DB_NAME/assets
inside the REMOTE_BACKUPS_PATH
directory is used for the asset backups.
Because rsync
is used for these backups, you can put a .rsync-filter
in any directory to define files/folders to ignore. More info
See Mitigating Disaster via Website Backups for a detailed writeup.
The sync_backups_to_s3.sh
script syncs the backups from LOCAL_BACKUPS_PATH
to the Amazon S3 bucket specified in REMOTE_S3_BUCKET
.
If you have defined a optional subfolder, it will contain the backups to the path defined in REMOTE_S3_PATH
.
This script assumes that you have already installed awscli and have configured it with your credentials.
It's recommended that you set up a separate user with access to only S3, and set up a private S3 bucket for your backups.
See Mitigating Disaster via Website Backups for a detailed writeup.
The backup_db.sh
script backs up the local database into a timestamped, gzip
compressed archive into the directory set via LOCAL_BACKUPS_PATH
. It will also automatically rotate out (delete) any backups that are older than GLOBAL_DB_BACKUPS_MAX_AGE
old.
The database backups exclude temporary/cache tables, and are stored in the sub-directory LOCAL_DB_NAME/db
, inside of LOCAL_BACKUPS_PATH
.
The numbers at the end of the backup archive are a timestamp in the format of YYYYMMDD-HHMMSS
.
See the Automated Script Execution section below for details on how to run this automatically
See Mitigating Disaster via Website Backups for a detailed writeup.
The backup_assets.sh
script backs up an arbitrary number of asset directories to the directory specified in LOCAL_BACKUPS_PATH
. The directories it backs are up specified in LOCAL_ASSETS_DIRS
, just as they were for the pull_assets.sh
script.
It will also back up the Craft userphotos
and rebrand
directories from craft/storage
by default. The directories it will backup are specified in LOCAL_CRAFT_FILE_DIRS
Because rsync
is used for these backups, you can put a .rsync-filter
in any directory to define files/folders to ignore. More info
For example, if you don't want any Craft image transforms backed up, your .rsync-filter
file in each assets directory might look like this:
# This file allows you to add filter rules to rsync, one per line, preceded by either
# `-` or `exclude` and then a pattern to exclude, or `+` or `include` and then a pattern
# to include. More info: http://askubuntu.com/questions/291322/how-to-exclude-files-in-rsync
- _*/**
See the Automated Script Execution section below for details on how to run this automatically
See Mitigating Disaster via Website Backups for a detailed writeup.
The backup_dirs.sh
script backs up an arbitrary number of directories to the directory specified in LOCAL_BACKUPS_PATH
. The directories it backs are up specified in LOCAL_DIRS_TO_BACKUP
.
This script is provided in case you have other files outside of your project that need backing up. For example, you might have a separate wiki or directory of config files.
Because rsync
is used for these backups, you can put a .rsync-filter
in any directory to define files/folders to ignore. More info
For example, if you have a wiki with data/cache
and data/tmp
directories that you don't want backed up, your .rsync-filter
file in the wiki directory might look like this:
# This file allows you to add filter rules to rsync, one per line, preceded by either
# `-` or `exclude` and then a pattern to exclude, or `+` or `include` and then a pattern
# to include. More info: http://askubuntu.com/questions/291322/how-to-exclude-files-in-rsync
- public/data/cache
- public/data/tmp
See the Automated Script Execution section below for details on how to run this automatically
See Mitigating Disaster via Website Backups for a detailed writeup.
The restore_db.sh
restores the local database to the database dumb passed in via command line argument. It backs up your local database before doing the restore.
You can pass in either a path to a .sql
file or .gz
file to restore_db.sh
, and it will do the right thing based on the file type.
See Mitigating Disaster via Website Backups for a detailed writeup.
- Download or clone the
craft-scripts
git repo - Copy the
scripts
directory into the root directory of your Craft CMS project - In the
scripts
directory, duplicate thecraft2-example.env.sh
(for Craft 2.x projects) orcraft3-example.env.sh
(for Craft 3.x projects) file, and rename it to.env.sh
. These*-example.env.sh
files are largely the same, just with some different defaults for Craft 2.x and Craft 3.x. - Add
.env.sh
to your.gitignore
file - Then open up the
.env.sh
file into your favorite editor, and replaceREPLACE_ME
with the appropriate settings.
All configuration is done in the .env.sh
file, rather than in the scripts themselves. This is is so that the same scripts can be used in multiple environments such as local
dev, staging
, and live
production without modification. Just create a .env.sh
file in each environment, and keep it out of your git repo via .gitignore
.
All settings that are prefaced with GLOBAL_
apply to all environments.
GLOBAL_DB_TABLE_PREFIX
is the Craft database table prefix, usually craft_
GLOBAL_CRAFT_PATH
is the path of the craft
folder, relative to the root path. This should normally be craft/
, unless you have moved it elsewhere. Paths should always have a trailing /
GLOBAL_DB_BACKUPS_MAX_AGE
Is the maximum age of local backups in days; backups older than this will be automatically rotated out (removed).
GLOBAL_DB_DRIVER
is the database driver for this Craft install (mysql
or pgsql
)
All settings that are prefaced with LOCAL_
refer to the local environment where the script will be run, not your local
dev environment.
LOCAL_ROOT_PATH
is the absolute path to the root of your local Craft install, with a trailing /
after it.
LOCAL_ASSETS_PATH
is the relative path to your local assets directories, with a trailing /
after it.
LOCAL_CHOWN_USER
is the user that is the owner of your entire Craft install.
LOCAL_CHOWN_GROUP
is your webserver's group, usually either nginx
or apache
.
LOCAL_WRITEABLE_DIRS
is a quoted list of directories relative to LOCAL_ROOT_PATH
that should be writeable by your webserver.
LOCAL_ASSETS_DIRS
is a quoted list of asset directories relative to LOCAL_ASSETS_PATH
that you want to pull down from the remote server. It's done this way in case you wish to sync some asset directories, but not others. If you want to pull down all asset directories in LOCAL_ASSETS_PATH
, just leave one blank quoted string in this array
LOCAL_CRAFT_FILE_DIRS
is a quoted list of Craft file directories relative to LOCAL_CRAFT_FILES_PATH
that you want to pull down from the remote server. By default, it will pull down the userphotos
and rebrand
directories in craft/storage
, which typically are not kept in git
. If you don't want it to sync anything, just leave the setting empty, e.g.: LOCAL_CRAFT_FILE_DIRS=()
LOCAL_DIRS_TO_BACKUP
is an array of bsolute paths to directories to back up, in addition to LOCAL_ASSETS_DIRS
and LOCAL_CRAFT_FILE_DIRS
LOCAL_FASTCGI_CACHE_DIR
is the local FastCGI Cache path; leave it empty ("") if you're not using FastCGI Cache; paths should always have a trailing /
. The clear_caches.sh
script will delete everything in this directory when it is executed (say, on deploy)
LOCAL_REDIS_DB_ID
is the local Redis database ID; leave it empty ("") if you're not using Redis. The clear_caches.sh
script will purge this Redis database when it is executed (say, on deploy)
LOCAL_DB_NAME
is the name of the local mysql Craft CMS database
LOCAL_DB_PASSWORD
is the password for the local mysql Craft CMS database
LOCAL_DB_USER
is the user for the local mysql Craft CMS database
LOCAL_DB_HOST
is the host name of the local mysql database host. This is normally localhost
LOCAL_DB_PORT
is the port number of the local mysql database host. This is normally 3306
for mysql
, and 5432
for postgres
.
LOCAL_MYSQL_CMD
is the command for the local mysql executable, normally just mysql
. It is provided because some setups like MAMP require a full path to a copy of mysql
inside of the application bundle.
LOCAL_MYSQLDUMP_CMD
is the command for the local mysqldump executable, normally just mysqldump
. It is provided because some setups like MAMP require a full path to a copy of mysqldump
inside of the application bundle.
LOCAL_PSQL_CMD
is the command for the local postgres executable, normally just psql
.
LOCAL_PG_DUMP_CMD
is the command for the local pg_dump executable, normally just pg_dump
.
LOCAL_DB_LOGIN_PATH
if this is set, it will use --login-path=
for your local db credentials instead of sending them in via the commandline (see below)
LOCAL_BACKUPS_PATH
is the absolute path to the directory where local backups should be stored. For database backups, a sub-directory LOCAL_DB_NAME/db
will be created inside the LOCAL_BACKUPS_PATH
directory to store the database backups. Paths should always have a trailing /
LOCAL_MYSQL_CMD
which is normally just mysql
, is prepended with docker exec -i CONTAINER_NAME
to execute the command within the container. (Example: docker exec -i container_mysql_1 mysql
)
LOCAL_MYSQLDUMP_CMD
which is normally just mysqldump
, is prepended with docker exec CONTAINER_NAME
to execute the command within the container. (Example: docker exec container_mysql_1 mysqldump
)
All settings that are prefaced with REMOTE_
refer to the remote environment where assets and the database will be pulled from.
REMOTE_SSH_LOGIN
is your ssh login to the remote server, e.g.: [email protected]
REMOTE_SSH_PORT
is the port to use for ssh on the remote server. This is normally 22
REMOTE_DB_USING_SSH
determines whether the database connection needs to be done over ssh
, or the database should be directly connected to (such as for Heroku or Amazon RDS services). This is normally yes
REMOTE_ROOT_PATH
is the absolute path to the root of your Craft install on the remote server, with a trailing /
after it.
REMOTE_ASSETS_PATH
is the relative path to the remote assets directories, with a trailing /
after it.
REMOTE_DB_NAME
is the name of the remote mysql Craft CMS database
REMOTE_DB_PASSWORD
is the password for the remote mysql Craft CMS database
REMOTE_DB_USER
is the user for the remote mysql Craft CMS database
REMOTE_DB_HOST
is the host name of the remote mysql database host. This is normally localhost
REMOTE_DB_PORT
is the port number of the remote mysql database host. This is normally 3306
for mysql
, and 5432
for postgres
.
REMOTE_MYSQL_CMD
is the command for the local mysql executable, normally just mysql
.
REMOTE_MYSQLDUMP_CMD
is the command for the local mysqldump executable, normally just mysqldump
.
REMOTE_PSQL_CMD
is the command for the remote postgres executable, normally just psql
.
REMOTE_PG_DUMP_CMD
is the command for the remote pg_dump executable, normally just pg_dump
.
REMOTE_DB_LOGIN_PATH
if this is set, it will use --login-path=
for your remote db credentials instead of sending them in via the commandline (see below)
REMOTE_BACKUPS_PATH
is the absolute path to the directory where the remote backups are stored. For database backups, a sub-directory REMOTE_DB_NAME/db
inside the REMOTE_BACKUPS_PATH
directory is used for the database backups. Paths should always have a trailing /
REMOTE_S3_BUCKET
is the name of the Amazon S3 bucket to backup to via the sync_backups_to_s3.sh
script
REMOTE_S3_PATH
is a optional path relative to the Amazon S3 bucket where the sync_backups_to_s3.sh
script will contain the backups if specified
Normally when you ssh
into a remote server (as some of the craft-scripts
do), you have to enter your password. Best practices from a security POV is to not allow for password-based logins, but instead use SSH Keys.
The day in, day out benefit of setting up SSH Keys is that you never have to enter your password again, so it allows for automated execution of the various craft-scripts
. Use the excellent How To Set Up SSH Keys artice as a guide for setting up your SSH keys.
If you use git
, a sample .gitignore
file that you can modify & use for your Craft CMS projects is included in craft-scripts
as example.gitignore
. If you wish to use it, the file should be copied to your Craft CMS project root, and renamed .gitignore
If you change file permissions on your remote server, you may encounter git complaining about overwriting existing local changes
when you try to deploy. This is because git considers changing the executable flag to be a change in the file, so it thinks you changed the files on your server (and the changes are not checked into your git repo).
To fix this, we just need to tell git to ignore permission changes on the server. You can change the fileMode
setting for git
on your server, telling it to ignore permission changes of the files on the server:
git config --global core.fileMode false
See the git-config man page for details.
The other way to fix this is to set the permission using set_perms.sh
in local
dev, and then check the files into your git repo. This will cause them to be saved with the correct permissions in your git repo to begin with.
The downside to the latter approach is that you must have matching user/groups in both local
dev and on live
production.
If you want to run any of these scripts automatically at a set schedule, here's how to do it. We'll use the backup_db.sh
script as an example, but the same applies to any of the scripts.
Please see the Setting up SSH Keys section and set up your SSH keys before you set up automatic script execution.
If you're using Forge you can set the backup_db.sh
script to run nightly (or whatever interval you want) via the Scheduler. If you're using ServerPilot.io or are managing the server yourself, just set the backup_db.sh
script to run via cron
at whatever interval you desire.
craft-scripts
includes a crontab-helper.txt
that you can add to your crontab
to make configuring cron
easier. Remember to use full, absolute paths to the scripts when running them via cron
, as cron
does not have access to your environment paths, e.g.:
/home/forge/nystudio107.com/scripts/backup_db.sh
If you're using a Mac and you want to execute the script locally, Apple uses Launch Daemons instead of cron
.
N.B.: Even if you are on a Mac, if you run your local
dev in a VM like Vagrant/Homestead, you'll want to execute the craft-scripts
from inside of the VM itself, not on your local Mac. If you use something like Valet or Mamp, read on.
Included in craft-scripts
is a com.example.launch_daemon.plist
to help you get started. This file is an XML
file, and the name should be a unique, reverse-DNS-style name suffixed with .plist
. This file is analogous to a single line in a crontab
file.
Rename com.example.launch_daemon.plist
to something unique to your project/script, e.g.: com.clientdomain.backup_db.plist
and place it in /Library/LaunchDaemons/
(you'll need to sudo
to do this).
The Launch Daemon .plist
file is an XML
file with a series of <key></key>
s followed by some type that is a value for that key. The value for the <key>Label</key>
should match the name of the file, minus the .plist
extension, e.g.: <string>com.clientdomain.backup_db</string>
. The value for the <key>UserName</key>
should be the user name that you want the task to run as, e.g.: <string>andrew</string>
The value for the <key>Program</key>
is a path to the command to execute, e.g.: <string>/Users/andrew/webdev/sites/nystudio107/scripts/backup_db.sh</string>
Launch Daemons offer any number of ways to schedule when and how they execute; please see the Launch Daemon documentation for details.
Once the file has been created in /Library/LaunchDaemons/
, it'll need to be loaded (you only need to do this once) via launchctl
, e.g.:
sudo launchctl load /Library/LaunchDaemons/com.clientdomain.backup_db.plist
For more information on configuring Launch Daemons, please see the excellent launchd.info website.
If you're using mysql 5.6
or later, you’ll note the warning from mysql (this is not an issue if you’re using MariaDB):
mysql: [Warning] Using a password on the command line interface can be insecure.
What the craft-scripts
is doing isn’t any less secure than if you typed it on the command line yourself; everything sent over the wire is always encrypted via ssh. However, you can set up login-path
to store your credentials in an encrypted file as per the Passwordless authentication using mysql_config_editor with MySQL 5.6 article.
If you set LOCAL_DB_LOGIN_PATH
or REMOTE_DB_LOGIN_PATH
it will use --login-path=
for your db credentials on the respective environments instead of sending them in via the commandline.
For example, for my local
dev setup:
mysql_config_editor set --login-path=localdev --user=homestead --host=localhost --port=3306 --password
...and then enter the password for that user. And then in the .env.sh
I set it to:
LOCAL_DB_LOGIN_PATH="localdev"
...and it will use my stored, encrypted credentials instead of passing them in via commandline. You can also set this up on your remote server, and then set it via REMOTE_DB_LOGIN_PATH
Brought to you by nystudio107