meteogrid / mason

Cross platform package manager for C/C++ apps

Mason

Mason can compile C/C++ code, publish packages to S3, and install those packages via binaries.

Mason does all its work locally inside ./mason_packages and therefore does not require privileges for, or conflict with, system-wide packages.

Mason works on both OS X and Linux.

Goals

Mason is a package manager designed for developers who package standalone applications and who need complete control over dependency versions.

Because Mason is developed by Mapbox the default S3 bucket and set of available packages are driven by Mapbox developers. For other developers: 1) fork Mason and ideally rename it to something like mason-{yourorg}, 2) configure it against your own s3 bucket, and 3) go ahead an publish your own packages to an S3 bucket of your creation.

Comparisons

Mason is like npm because it installs packages in the current working directory (./mason_packages) rather than globally.

Mason is like homebrew because it requires no use of sudo to install packages.

Mason is like linux package managers like apt-get or yum because it works on linux.

Mason is unlike all of the above package managers because:

• Mason runs on both Linux and OS X and creates a single set of binaries that work on >= OS X 10.8 and >= Ubuntu Precise (rather than building binaries per version).
• Mason strongly prefers static libraries over shared libraries
• Mason has zero understanding of dependency trees: it leaves complete control to the developer for how packages relate.
• Mason does not depend on any specific runtime language (like python, node.js or ruby). It is a just a few bash scripts.
• Mason depends on travis.ci for creating and publishing binaries.

Installation

You need to install Mason to your user directory into ~/.mason.

git clone -b master --single-branch https://github.com/mapbox/mason.git ~/.mason
sudo ln -s ~/.mason/mason /usr/local/bin/mason

The second line is optional.

Usage

Most commands are structured like this:

mason <command> <library> <version>

The command can be one of the following

• install: Installs the specified library/version
• remove: Removes the specified library/version
• build: Forces a build from source (= skip pre-built binary detection)
• publish: Uploads the built binaries to the S3 bucket
• prefix: Prints the absolute path to the library installation directory
• version: Prints the actual version of the library (only useful when version is system)
• cflags: Prints C/C++ compiler flags
• ldflags: Prints linker flags
• link: Creates symlinks for packages in mason_packages/.link
• trigger: Trigger a build and publish operation on Travis CI

install

$ mason install libuv 0.11.29
* Downloading binary package osx-10.10/libuv/0.11.29.tar.gz...
######################################################################## 100.0%
* Installed binary package at /Users/user/mason_packages/osx-10.10/libuv/0.11.29

Installs libuv into the current folder in the mason_packages directory. Libraries are versioned by platform and version number, so you can install several different versions of the same library along each other. Similarly, you can also install libraries for different platforms alongside each other, for example library binaries for OS X and iOS.

The install command first checks if the specified library/version is already present for this platform, and if so, exits. This means you can run it multiple times (e.g. as part of a configuration script) without doing unnecessary work.

Next, Mason checks whether there are pre-built binaries available in the S3 bucket for the current platform. If that is the case, they are downloaded and unzipped and the installation is complete.

If no pre-built binaries are available, Mason is going to build the library according to the script in the mason_packages/.build folder, and install into the platform- and library-specific directory.

remove

$ mason remove libuv 0.11.29
* Removing existing package...
/Users/user/mason_packages/osx-10.10/libuv/0.11.29/include/uv-darwin.h
[...]
/Users/user/mason_packages/osx-10.10/libuv/0.11.29

Removes the specified library/version from the package directory.

build

This command works like the install command, except that it doesn't check for existing library installations, and that it doesn't check for pre-built binaries. I.e. it first removes the current installation and always builds the library from source. This is useful when you are working on a build script and want to fresh builds.

publish

Creates a gzipped tarball of the specified platform/library/version and uploads it to the mason-binaries S3 bucket. If you want to use this feature, you need write access to the bucket and need to specify the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

prefix

~ $ mason prefix libuv 0.11.29
/Users/user/mason_packages/osx-10.10/libuv/0.11.29

This prints the absolute path to the installation directory of the the library/version. Likely, this folder has the typical include and lib folders.

cflags

~ $ mason cflags libuv 0.11.29
-I/Users/user/mason_packages/osx-10.10/libuv/0.11.29/include

Prints the C/C++ compiler flags that are required to compile source code with this library. Likely, this is just the include path, but may also contain other flags.

ldflags

~ $ mason ldflags libuv 0.11.29
-L/Users/user/mason_packages/osx-10.10/libuv/0.11.29/lib -luv -lpthread -ldl

Prints the linker flags that are required to link against this library.

link

~ $ mason link libuv 0.11.29

This command only works if the package has already been installed. When run it symlinks the versioned lib, include, share, and bin folders of the package into a shared structure that is unversioned. For example if mason prefix libuv 0.11.29 was ./mason_packages/osx-10.10/libuv/0.11.29 then the library would become available at ./mason_packages/.link/lib/libuv.a

trigger

In order to ensure that all prebuild binaries are consistent and reproducible, we perform the final build and publish operation on Travis CI. Use the trigger command to kick this off:

~ $ mason trigger libuv 0.11.29

Run this command from the root of a local mason repository checkout. It makes a request to the Travis API to build and publish the specified version of the package, using the Travis configuration in ./scripts/${MASON_NAME}/${MASON_VERSION}/.travis.yml. It requires the TRAVIS_TOKEN environment variable to be set. You can obtain an appropriate value using the Travis command line client's travis token command.

Writing build scripts

Each version of each package has its own directory in scripts/${package}/${version}, e.g. scripts/libuv/0.11.29. The directory must contain a file called script.sh, which is structured like this:

#!/usr/bin/env bash

MASON_NAME=libuv
MASON_VERSION=0.11.29
MASON_LIB_FILE=lib/libuv.a
MASON_PKGCONFIG_FILE=lib/pkgconfig/libuv.pc

Declare these variables first. MASON_NAME and MASON_VERSION are mandatory. If the install script build a static library, specify the relative path in the installation directory in MASON_LIB_FILE. This is used to check whether an installation actually exists before proceeding to download/build the library anew. You can optionally specify MASON_PKGCONFIG_FILE as the relative path to the pig-config file if the library has one. If the library doesn't have one, you need to override the functions mason_cflags and mason_ldflags (see below).

Then, we're loading the build system with

. ${MASON_DIR}/mason.sh

Next, we're defining a function that obtains the source code and unzips it:

function mason_load_source {
    mason_download \
        https://github.com/joyent/libuv/archive/v0.11.29.tar.gz \
        5bf49a8652f680557cbaf335a160187b2da3bf7f

    mason_extract_tar_gz

    export MASON_BUILD_PATH=${MASON_ROOT}/.build/libuv-${MASON_VERSION}
}

In that function, you should use mason_download as a shortcut to download the tarball. The second argument to is a hash generated with git hash-object and used to verify that the source code downloaded matches the expected file. The function also caches downloaded tarballs in the mason_packages/.cache folder.

mason_extract_tar_gz unpacks the archive into the mason_packages/.build folder. If the tarball is BZip2 compressed, you can also use mason_extract_tar_bz2 instead.

Lastly, the MASON_BUILD_PATH variable contains the path to the unpacked folder inside the .build directory.

Then, you can optionally specify a function that is run before compiling, e.g. a script that generates configuration files:

function mason_prepare_compile {
    ./autogen.sh
}

The heart of the script is the mason_compile function because it performs the actual build of the source code. There are a few variables available that you need to use to make sure that the package will work correctly.

function mason_compile {
    ./configure \
        --prefix=${MASON_PREFIX} \
        ${MASON_HOST_ARG} \
        --enable-static \
        --disable-shared \
        --disable-dependency-tracking

    make install -j${MASON_CONCURRENCY}
}

In particular, you have to set the build system's installation prefix to MASON_PREFIX. For cross-platform builds, you have to specify the MASON_HOST_ARG, which is empty for regular builds and is set to the correct host platform for cross-compiles (e.g. iOS builds use --host=arm-apple-darwin).

If the build system supports building concurrently, you can tell it do do so by providing the number of parallel tasks from MASON_CONCURRENCY.

Next, the mason_clean function tells Mason how to clean up the build folder. This is required for multi-architecture builds. E.g. iOS builds perform a Simulator (Intel architecture) build first, then an iOS (ARM architecture) build. The results are lipoed into one universal archive file.

function mason_clean {
    make clean
}

Finally, we're going to run the everything:

mason_run "$@"

Variables

Name	Description
MASON_DIR	The directory where Mason itself is installed. Defaults to the current directory.
MASON_ROOT	Absolute path the mason_packages directory. Example: /Users/user/mason_packages.
MASON_PLATFORM	Platform of the current invocation. Currently one of osx, ios or linux.
MASON_PLATFORM_VERSION	Version of the platform. It must include the architecture if the produced binaries are architecture-specific (e.g. on Linux). Example: 10.10
MASON_NAME	Name specified in the script.sh file. Example: libuv
MASON_VERSION	Version specified in the script.sh file. Example: 0.11.29
MASON_SLUG	Combination of the name and version. Example: libuv-0.11.29
MASON_PREFIX	Absolute installation path. Example: /Users/user/mason_packages/osx-10.10/libuv/0.11.29
MASON_BUILD_PATH	Absolute path to the build root. Example: /Users/user/mason_packages/.build/libuv-0.11.29
MASON_BUCKET	S3 bucket that is used for storing pre-built binary packages. Example: mason-binaries
MASON_BINARIES	Relative path to the gzipped tarball in the .binaries directory. Example: osx-10.10/libuv/0.11.29.tar.gz
MASON_BINARIES_PATH	Absolute path to the gzipped tarball. Example: /Users/user/mason_packages/.binaries/osx-10.10/libuv/0.11.29.tar.gz
MASON_CONCURRENCY	Number of CPU cores. Example: 8
MASON_HOST_ARG	Cross-compilation arguments. Example: --host=x86_64-apple-darwin
MASON_LIB_FILE	Relative path to the library file in the install directory. Example: lib/libuv.a
MASON_PKGCONFIG_FILE	Relative path to the pkg-config file in the install directory. Example: lib/pkgconfig/libuv.pc
MASON_XCODE_ROOT	OS X specific; Path to the Xcode Developer directory. Example: /Applications/Xcode.app/Contents/Developer

Customization

In addition to the override functions described above, you can also override the mason_cflags and mason_ldflags functions. By default, they're using pkg-config to determine these flags and print them to standard output. If a library doesn't include a .pc file, or has some other mechanism for determining the build flags, you can run them instead:

function mason_ldflags {
    ${MASON_PREFIX}/bin/curl-config --static-libs`
}

System packages

Some packages ship with operating systems, or can be easily installed with operating-specific package managers. For example, libpng is available on most systems and the version you're using doesn't really matter since it is mature and hasn't added any significant new APIs in recent years. To create a system package for it, use this script.sh file:

#!/usr/bin/env bash

MASON_NAME=libpng
MASON_VERSION=system
MASON_SYSTEM_PACKAGE=true

. ${MASON_DIR}/mason.sh

if [ ! $(pkg-config libpng --exists; echo $?) = 0 ]; then
    mason_error "Cannot find libpng with pkg-config"
    exit 1
fi

function mason_system_version {
    mkdir -p "${MASON_PREFIX}"
    cd "${MASON_PREFIX}"
    if [ ! -f version ]; then
        echo "#include <png.h>
#include <stdio.h>
#include <assert.h>
int main() {
    assert(PNG_LIBPNG_VER == png_access_version_number());
    printf(\"%s\", PNG_LIBPNG_VER_STRING);
    return 0;
}
" > version.c && ${CC:-cc} version.c $(mason_cflags) $(mason_ldflags) -o version
    fi
    ./version
}

function mason_compile {
    :
}

function mason_cflags {
    pkg-config libpng --cflags
}

function mason_ldflags {
    pkg-config libpng --libs
}

mason_run "$@"

System packages are marked with MASON_SYSTEM_PACKAGE=true. We're also first using pkg-config to check whether the library is present at all. The mason_system_version function creates a small executable which outputs the actual version. It is the only thing that is cached in the installation directory.

We have to override the mason_cflags and mason_ldflags commands since the regular commands return flags for static libraries, but in the case of system packages, we want to dynamically link against the package.

Troubleshooting

Downloaded source tarballs are cached in mason_packages/.cache. If for some reason the initial download failed, but it still left a file in that directory, make sure you delete the partial download there.