davidwittman / ansible-mesos-playbook

An ansible playbook for launching a mesos cluster with docker and marathon support.

ansible-mesos-playbook

An ansible playbook for launching a mesos cluster with native docker and mesos executors, along with Marathon), Consul and HAProxy support. Run this on Ubuntu 14.04 LTS (preferred) or Centos/RHEL 6. Read the blog post for a descriptive overview.

Getting Started

• Install ansible, version >= 1.7.
• Install librarian-ansible via gem install librarian-ansible
• Run librarian-ansible install
• Spin up a bunch of Ubuntu 14.04 servers, say 5, on your favorite cloud provider.
• cp hosts.sample hosts and update the mesos_masters and mesos_slaves groups.
• cp ansible.cfg.sample ansible.cfg to ensure librarian_roles is in the ansible path (ansible.cfg is git-ignored).
• Run ansible-playbook playbook.yml.

The Setup

• zookeeper, haproxy, mesos-master, consul and marathon with ha mode run on nodes in the mesos_primaries group. The zoo_id host variable is used to configure zookeeper, and consul_bootstrap is set on one node to initialize the cluster.
• mesos-slave runs in the mesos_workers group and are passed the list of mesos_primaries for coordination.
• Docker and native mesos are configured as containerizers on mesos-slaves.
• A cron job on each master is set up to query the marathon api and configure HAProxy.
• HAProxy routes a frontend (listening on port 80) to backends based on marathon tasks.
• Consul for service discovery. It's not hooked into any other services, just part of the default setup.
• You probably want to tweak the HAProxy configuration script (in /opt/marathon/bin) for your needs. With the current setup you can have a wildcard dns prefix route to a backend matching the marathon name: i.e. www.example.com would do a least-connection proxy to the www task.

Altering the Playbook

There are a variety of tweaks you can make to this playbook for your needs.

• Don't want Marathon, Consul or HAProxy? Simply remove roles from mesos_primaries.yml
• Don't want to overload the primaries? Simply add new groups and remap roles appropriately.

Troubelshooting

If you have trouble, /var/log/syslog on Ubuntu and /var/log/messages on RHEL is your friend. For Zookeeper, try /var/log/zookeeper/zookeeper.log. You can try re-running the playbook; the roles aren't perfect but most are idempotent.

Ansible lets you perform actions on groups of servers. You can try query or restart zookeeper and/or mesos:

$ ansible mesos_primaries -a "sudo status zookeeper"
$ ansible mesos_primaries -a "sudo restart zookeeper"

Notes

Currently this installs Mesos 0.20.1 with Marathon 0.7.3.

Launching a Container

POST to /v2/apps:

{
    "id": "mlh", 
    "container": {
        "docker": {
            "image": "mhamrah/mesos-sample",
            "network": "BRIDGE",
            "portMappings": [
                { "containerPort": 8080, "hostPort": 0, "protocol": "tcp" }
            ]
        },
        "type": "DOCKER"
    },
    "cpus": 0.5,
    "mem": 512,
    "instances": 1
}