morganstanley / treadmill

lightweight, container-based compute fabric

Treadmill

Download

wget https://github.com/Morgan-Stanley/treadmill/releases/download/0.0.1/treadmill -O /usr/bin/treadmill && chmod +x /usr/bin/treadmill

This will put treadmill in your path.

Vagrant setup for development

Prerequisites

• VirtualBox
• Vagrant
• Install the Guest Additions plugin for Vagrant

vagrant plugin install vagrant-vbguest

Vagrant Setup

• Clone the treadmill repo.
• Clone the treadmill-pid1 repo.
• Run the following commands

cd treadmill
git checkout standard_setup
vagrant up
vagrant ssh

• Create the cgroup folders as root

sudo su -
cd /sys/fs/cgroup
for i in *; do mkdir -p $i/treadmill/apps $i/treadmill/core $i/system ; done
cd -

• Start the zookeeper service

cd /home/centos && zookeeper-3.4.9/bin/zkServer.sh start

• Make the mount private

sudo mount --make-rprivate /

• Treadmill should now be available on the box

source /home/centos/env/bin/activate
treadmill --help

Treadmill cli cheatsheet

treadmill sproc scheduler /tmp
treadmill sproc service --root-dir /tmp/treadmill/ localdisk --reserve 20G --img-location /tmp/treadmill --default-read-bps 100M --default-write-bps 100M --default-read-iops 300 --default-write-iops 300
treadmill sproc service --root-dir /tmp/treadmill/ network
treadmill sproc service --root-dir /tmp/treadmill/ cgroup

# Zookeeper shell
create /scheduled/centos.bar#123 {"memory":"100M","cpu":"10%","disk":"500M","proid":"centos","affinity":"centos.bar","services":[{"name":"sleep","command":"/bin/top","restart":{"limit":5,"interval":60}}]}
create /servers/localhost.localdomain {"parent":"all:unknown","features":[],"traits":[],"label":null,"valid_until":1488573090.0}
create /cell/all:unknown {}
create /buckets/all:unknown {"parent":null,"traits":0}

# System shell
treadmill sproc init --approot /tmp/treadmill/
cd /tmp/treadmill/running && nohup /bin/s6-svscan > s6_svscan.out & && cd -
treadmill sproc eventdaemon
treadmill sproc appcfgmgr

treadmill's Issues

README.md does not provide adequate information about application purpose

README.md should have an introduction part which explains the use of the application

Non-existent modules on the Internet

When running nosetests as outlined by issue #5, the following modules do not seem to exist. They are likely internal modules that should either be removed or bundled with Treadmill:

webauthd
proiddb
plugins.fs

The following is the trail of errors illustrating which tests are picking these up:

ERROR: Tests optional container features.

Traceback (most recent call last):
File "/home/tome/treadmill/tests/appmgr/manifest_test.py", line 266, in test__add_features
treadmill.appmgr.manifest._add_features(manifest)
File "/home/tome/treadmill/lib/python/treadmill/appmgr/manifest.py", line 188, in _add_features
raise Exception('Unsupported feature: ' + feature)
Exception: Unsupported feature: webauthd
-------------------- >> begin captured logging << --------------------
treadmill.appmgr.manifest: ERROR: Unable to load feature: webauthd
Traceback (most recent call last):
File "/home/tome/treadmill/lib/python/treadmill/appmgr/manifest.py", line 183, in _add_features
'treadmill.appmgr.features.' + feature
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
ImportError: No module named webauthd
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Test the normalization of manifests.

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1297, in patched
arg = patching.enter()
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1353, in enter
self.target = self.getter()
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1523, in
getter = lambda: _importer(target)
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1210, in _importer
thing = _dot_lookup(thing, comp, import_path)
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1199, in _dot_lookup
import(import_path)
ImportError: No module named proiddb

======================================================================
ERROR: Test AppMgr environment initialization.

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "/home/tome/treadmill/tests/appmgr_test.py", line 67, in test_initialize
app_init.initialize(self.app_env)
File "/home/tome/treadmill/lib/python/treadmill/appmgr/initialize.py", line 16, in initialize
nodeinit.initialize(tm_env)
File "/home/tome/treadmill/lib/python/treadmill/osmodules/linux/nodeinit.py", line 20, in initialize
fs.init_plugins(tm_env.root)
File "/home/tome/treadmill/lib/python/treadmill/fs.py", line 253, in init_plugins
for mod in _iter_plugins():
File "/home/tome/treadmill/lib/python/treadmill/fs.py", line 235, in _iter_plugins
fsplugins = importlib.import_module('treadmill.plugins.fs')
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
ImportError: No module named plugins.fs

installation failure: access to treadmill.repo is denied

Documentation is outdated:

The following command fails:
git checkout standard_setup

If used doc from here: https://treadmill.readthedocs.io/en/latest/getting_started.html
cd treadmill
vagrant up --provision master
That also fails because Vagrantfile is missing.

If use the following commands:
cd treadmill/vagrant
vagrant up --provision master

Provisioning fails as the following:
master: Running: /var/folders/bx/7y0szfxx0rd0rhk1xf5jxy5c0000gn/T/vagrant-shell20181017-15754-13zl08c.sh
master: /home/vagrant/treadmill/vagrant/scripts/install-base.sh is up to date.
master: md5sum: /root/.provision/install-master-packages.sh: No such file or directory
master: Installing master packages.
master: Loaded plugins: fastestmirror
master:
master:
master: File contains no section headers.
master: file: file:///etc/yum.repos.d/treadmill.repo, line: 1
master: '\n'
master: /home/vagrant/treadmill/vagrant/scripts/install-master-packages.sh failed.

The reason is that access is closed to treadmill.repo:
https://s3.amazonaws.com/yum_repo_dev/treadmill.repo
<Error> <Code>AllAccessDisabled</Code> <Message>All access to this object has been disabled</Message> <RequestId>34CE3501DF55EB80</RequestId> <HostId> pL/9g/enbg0LQTU1pXTMX0qbvQgkAMmkam9D/lRhVqggHcpwYxYE6T9t6zISBG8Hix8523gPLDE= </HostId> </Error>

jsonpointer needs to support versions .7 and up

Modern versions of jsonpointer do not have set_pointer() method. When running through the tests, I'm getting the following error:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "/home/tome/treadmill/tests/api/schema_test.py", line 318, in test_allocation
_ok(api.create, 'aaa/prod', _patch(good, '/environment', 'qa'))
File "/home/tome/treadmill/tests/api/schema_test.py", line 46, in _patch
return jsonpointer.set_pointer(obj, jsonptr, value, inplace=False)
AttributeError: 'module' object has no attribute 'set_pointer'

I cannot downgrade jsonpointer. It looks like we need to support newer versions. I found this bit of code that has a fix to update set_pointer() in some other project that is using this:
peritus/robotframework-httplibrary@aa83809

Get all tests to work and document how to run tests

This issue should remain open until all tests are fixed and how to run them are documented

Here are the steps to install dependencies and run tests on a clean ubuntu box. There are a number of errors that are still occurring and will call them out independently in issues of their own. These instructions should be wrapped in setuptools and each module installed locally with setup.py. However, until that time, I'm just adding them here so that others can help hack away at getting the tests to work.

sudo apt-get install python-dev
sudo apt-get install krb5-config (just click enter on default kerb realm)
sudo apt-get install libkrb5-dev
sudo apt-get install python-pip

sudo locale-gen en_US
sudo update-locale

sudo pip install nose
sudo pip install kazoo
sudo pip install mock
sudo pip install jinja2
sudo pip install flask
sudo pip install tornado
sudo pip install enum34
sudo pip install ldap3
sudo pip install kerberos
sudo pip install decorator
sudo pip install dnspython
sudo pip install jsonschema
sudo pip install netifaces
sudo pip install flask_restplus
sudo pip install requests_kerberos
sudo pip install urllib2_kerberos
sudo pip install numpy
sudo pip install pandas
sudo pip install simplejson
sudo pip install websocket
sudo pip install jsonpointer --upgrade

The following modules do not exist on the Internet and probably need to be dropped/rethought re: why they are needed in public version of Treadmill:
webauthd
plugin.fs
proiddb

With the above in place, cd to treadmill dir and run:
nosetests

LDAP Exceptions do not exist and are failing in tests

Getting a lot of errors like these when running tests:

File "/home/tome/treadmill/tests/context_test.py", line 47, in test_ldap_resolve
treadmill.admin.Cell.get.side_effect = ldap3.LDAPNoSuchObjectResult
AttributeError: 'module' object has no attribute 'LDAPNoSuchObjectResult'

Looking at ldap3 module, it looks like exceptions are in ldap3.core.exceptions not ldap3.exceptions.

utils_test failure

When running nosetests as outlined in Issue #5, I am getting the following error:

======================================================================
ERROR: Tests conversion of values into human readable format.

Traceback (most recent call last):
File "/home/tome/treadmill/tests/utils_test.py", line 233, in test_humanreadable
self.assertEquals('2,310', utils.cpu_to_readable(2310))
File "/home/tome/treadmill/lib/python/treadmill/utils.py", line 348, in cpu_to_readable
locale.setlocale(locale.LC_ALL, 'en_US')
File "/usr/lib/python2.7/locale.py", line 579, in setlocale
return _setlocale(category, locale)
Error: unsupported locale setting

README.md outdated

README mentioned the following
git checkout standard_setup
but actually "standard_setup" doesn't exist

Documentation forthcoming?

Any plans to beef up the documentation on this? Building/running?

Missing test_setup.py

test_setup.py is needed by nosetests - this is the script that loads the lib paths and tests paths that are located in /lib and /tests.

LDAP schema is missing

Some of the tests are failing for more schema files, like:

======================================================================
ERROR: test missing defaults which cause the app to fail

Traceback (most recent call last):
File "/home/centos/treadmill/.tox/py27/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "/home/centos/treadmill/tests/api/instance_test.py", line 42, in test_normalize_run_once
new_doc = self.instance.create("proid.app", yaml.load(doc))
File "", line 2, in create
File "/home/centos/treadmill/treadmill/schema.py", line 64, in decorated
return func(*valid_args, **valid_kwargs)
File "/home/centos/treadmill/treadmill/api/instance.py", line 90, in create
'Missing required attribute: proid')
TreadmillError: Missing required attribute: proid
-------------------- >> begin captured logging << --------------------
treadmill.api.instance: INFO: Unable to load auth plugin: No module named plugins.api.instance
treadmill.api.instance: INFO: create: count = 1, proid.app {'services': [{'name': 'sleep1m', 'restart': {'limit': 0}, 'command': '/bin/sleep 1m'}], 'memory': '150M', 'cpu': '10%', 'disk': '100M'}
--------------------- >> end captured logging << ---------------------

Proper Documentation?

is this the most updated version of the treadmill install as this differs from the git page readme a bit? http://treadmill.readthedocs.io/en/latest/getting_started.html

[Bug] Fail to run `treadmill admin invoke` in Vagrant VM

Hi, I am trying to call treadmill admin invoke in vagrant, and it returns an error:

$ vagrant up && vagrant ssh master
# In master
$ treadmill admin invoke
Unable to load plugin: invoke [ /tmp/tmpj3mkfik5 ]
Usage: treadmill admin [OPTIONS] COMMAND [ARGS]...

Error: No such command "invoke".
$ cat /tmp/tmpj3mkfik5
Traceback (most recent call last):
  File "/home/vagrant/treadmill/treadmill/cli/admin/invoke.py", line 47, in _import_resource_mod
    return importlib.import_module('treadmill.api.' + resource_type)
  File "/usr/lib64/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
ImportError: No module named 'treadmill.api.scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/treadmill/treadmill/cli/__init__.py", line 77, in get_command
    return mod.init()
  File "/home/vagrant/treadmill/treadmill/cli/admin/invoke.py", line 181, in init
    make_resource_group(ctx, invoke, resource)
  File "/home/vagrant/treadmill/treadmill/cli/admin/invoke.py", line 150, in make_resource_group
    make_resource_group(ctx, _rsrc_group, verb, func)
  File "/home/vagrant/treadmill/treadmill/cli/admin/invoke.py", line 128, in make_resource_group
    mod = _import_resource_mod(resource_type)
  File "/home/vagrant/treadmill/treadmill/cli/admin/invoke.py", line 50, in _import_resource_mod
    raise click.BadParameter(resource_type)
click.exceptions.BadParameter: scheduler

Removing the content in treadmill/api/cron.py could hack the issue.

0.0.1/treadmill returning 404

From the root README:

wget https://github.com/Morgan-Stanley/treadmill/releases/download/0.0.1/treadmill -O /usr/bin/treadmill && chmod +x /usr/bin/treadmill is returning a 404.

Attempting to install on Ubuntu 16.04.

Log below:

--2017-11-07 13:17:44--  https://github.com/Morgan-Stanley/treadmill/releases/download/0.0.1/treadmill
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-11-07 13:17:44 ERROR 404: Not Found.