mozilla / build-relengapi

INACTIVE - http://mzl.la/ghe-archive - Your Interface to Release Engineering Automation -

RelengAPI

Continuous Integration: https://travis-ci.org/mozilla/build-relengapi

Your Interface to Release Engineering Automation.

This is the framework behind https://api.pub.build.mozilla.org/. It is a Flask-based framework for building and hosting releng-related APIs.

Goals

• Simple self-service usage for consumers

  • Industry-standard access mechanisms (REST, oAuth2, etc.) that require no client-side custom libraries
  • One or very few endpoints (e.g., https://api.pub.build.mozilla.org)
  • Self-documenting tools
  • Semantic versioning

• Simple, rapid implementation of new apps

  • Common requirements such as authentication, database access, scheduled tasks, configuration handling are already satisfied
  • All apps use the same technologies (language, web framework, DB framework, etc.), so the learning curve from one app to the next is small
  • Tailored for easy local development - minimal requirements, minimal installed components, etc.

• Operations-friendly

  • Horizontally scalable using normal webops techniques
  • Easily deployed in multiple environments with normal devops processes
  • Resilient to failure: no in-memory state

Documentation

RelengAPI documents itself.
See https://api.pub.build.mozilla.org/docs for the documentation of the currently-deployed version.

Info for Developers

See the "Installation" page of the deployment documentation for information on required operating system packages.

Structure

RelengAPI is a Flask application. It is composed of several Python distributions (packages). Each distribution can contain several Flask Blueprints -- web application components. Each Git repository can contain multiple distributions.

The base is in the relengapi distribution, implemented in this package. It implements the root app, with lots of common support functionality, and a number of blueprints. It also searches its python environment for other distributions that can provide blueprints for the Releng API. These act as plugins, adding extra endpoints and other functionality to the API.

Other top-level directories of this repository contain other related distributions with more blueprints. Other repositories contain even more distributions, with even more blueprints.

All of this is drawn together in production by installing the appropriate distributions on the releng web cluster. When developing, though, only the relengapi distribution and the distribution you're hacking on are required.

Running RelengAPI

To run the tool for development, pip install the requirements into your virtualenv:

pip install -e .[test]

The [test] installs the requirements for testing as well. Omit this if you won't be running tests.

[optional] Build the docs:

relengapi build-docs 

[optional] Set up your settings file:

cp settings_example.py settings.py
vim settings.py
export RELENGAPI_SETTINGS=$PWD/settings.py

Create the databases for the installed blueprints:

relengapi createdb

And finally run the server:

relengapi serve -p 8010

The relengapi tool has many useful subcommands. See its help for more information.

More

See the Releng API documentation for more information on development and deployment of the releng API. This is available at https://api.pub.build.mozilla.org/docs or, If you've installed the docs blueprint, at the same path on your own instance.

build-relengapi's Issues

Add a `relengapi test` command

The nosetests command has some issues:

• it stumbles over the namespaced modules, and thus has a hard time taking a filename as an argument (it can't convert that filename to a module path)
• it doesn't necessarily import relengapi.app first, so it sometimes races with the population of the relengapi namespace.
  This can probably be fixed by a relengapi test command that takes the same args as nosetests, but sets up the environment correctly first.

https://nose.readthedocs.org/en/latest/doc_tests/test_addplugins/test_addplugins.html

Implement slavealloc UI

The slavealloc UI and API should be moved to releng API.

It might make sense to move the allocator as well, although the design for
releng API was to not put mission-critical services directly on the API.
However, slavealloc clients are resilient to failure, so perhaps it isn't
mission-critical?

nosetests loads modules in a funny order

..and this causes failures.

======================================================================

ERROR: Failure: ImportError (cannot import name subcommands)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7.6/lib/python2.7/site-packages/nose/loader.py", line 411, in loadTestsFromName

addr.filename, addr.module)

File "/home/travis/virtualenv/python2.7.6/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath

return self.importFromDir(dir_path, fqname)

File "/home/travis/virtualenv/python2.7.6/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir

mod = load_module(part_fqname, fh, filename, desc)

File "/home/travis/build/petemoore/build-relengapi/docs/relengapi/blueprints/docs/__init__.py", line 6, in <module>

from relengapi import subcommands

ImportError: cannot import name subcommands

----------------------------------------------------------------------

Generate a sample config using a subcommand

There's a settings_example.py in the root directory now, but that's not
especially useful to users.

More useful would be something like

$ relengapi-setup .

which creates a settings.py with the basic required settings included, along
with comments.

Tests for UTCDateTime

@Callek said he'd take care of this in the next week or so.

Token Improvements

Exactly how this is implemented depends on what we need. It's entirely possible to allow an HTTP request authenticated with a token to issue a new token (with the same or fewer actions). In that case, we'd want the ability to trace the tree of generated tokens in case, for example, a parent token had to be revoked.

Alternatively, we may want to be able to trace tokens back to the user who created them (perhaps indirectly).

Finally, we may just want some record of how the token was created, as a hint to deciphering its use.

Options:

1. Use to track "token tree" of tokens issued by tokens
2. Always trace back to a human
3. Just informational, so either an email or "token ###"

Implement LDAP group-based authorization

Within Mozilla, the best way to determine different permissions for different
people is to based decisions on LDAP group membership.

Add a context manager for DB connections

Blueprints should be able to open a DB connection only for a short segment of code, e.g.,

with g.db.connect('relengapi') as conn:
    conn.execute(..)

Bug in sdist packaging of relengapi - files missing

See djmitche@917f5c9#commitcomment-6389808

Document the JS environment for embedded browser stuff

Releng API isn't meant to host big, cool web apps -- those should be on other hosts, and use oAuth to talk to the Releng API.

But we still need some amount of JS goodness to build some basic UIs for things like approving oAuth requests, issuing tokens, and so on. Those should all work in the same JS environment for consistency.

At the moment, despite my eagerness to try out AngularJS, that's limited to JQuery and the Altertify tool. Those limitations should be documented.

Replace production-masters.json, production-branches.json

Set up travis

Travis should

• run tests
• verify packaging is catching all required files

Travis job should ensure architecture is clean from slave loan tool

A thought just occurred - we agreed that architecturally we would make sure no dependencies will get introduced on slave loan tool from relengapi, and that they would live in different directories and have their own python packages that don't contain any code crossover.

We can test this in the CI - we can either delete / move the slave loan tool subdirectory when building relengapi and vice versa - to make sure neither has a dependency on the other being present in the source tree. This gives us confidence that the split will be easier ready for relengapi 1.0.

I would even propose we fully remove one when we build/test the other - so we have two complete cycles in the travis setup - first delete slave loan tool, and build/test everything for relengapi, and then the other way around - restore slaveloan subdirectory, and completely ditch the relengapi base directory and other top level artifacts it contains, leaving just the slave loan subdirectory - and then do the full build/test cycle in that.

Connect with New Relic

New Relic is disabled on the releng cluster at the moment, since it assumes Python-2.6.

That should be fixed, and minor modifications made to feed relevant data to NR.

Deploy script should validate database schema is ok

When deploying relengapi components using the update script on relengwebadm, it should at least try to see if the schema is correct, and either fix the problem (run relengapi createdb) or fail with an explanation.

Recently after a deployment, we had problems, which could maybe have been caught at deploy phase: https://bugzilla.mozilla.org/show_bug.cgi?id=1019847#c6

Please also note that the createdb action was run to try to fix the problem, it reported no problems, but did not fix the problem (probably because it was a column rename rather than simply adding new schema objects such as tables).

I know we are planning to use a database migration tool, so until this is done, maybe a simple fail would be enough. Maybe we can run "relengapi createdb" against a "spare" schema, and then compare the two schema definitions (the spare vs actual schemas) and if there are differences, fail. This way we can drop and recreate the spare schema without losing data in the actual schema.

I'm putting this as a relengapi-1.0.0 feature, as I think this is pretty important, when we go live, to avoid bustage (as relengapi will very soon be a potentially tree-closing service, if it goes down - due to mozharness b2g_build.py reliance on mapper service).

Log instead of printing at startup

At startup, relengapi prints the blueprints it is loading. That information should be presented via Python logging, instead.

Document using joins in queries

Based on djmitche/mapper#3

Port clobberer

Clobberer's UI should be in the releng API.

The buildslave interface should probably stay a separate service, so that
upgrades to relengapi aren't a high risk.

Make deployment of changes easy

RelengAPI is deployed as versioned distributions, so the process looks something like this right now:

• bump the version number for the relevant distribution
• make an sdist tarball
• upload to releng pypi
• update requirements.txt on the admin host
• run ./update

We should automate some of that, at least, keeping in mind:

• upgrades of production code are risky
• incompatible API changes for consumers will be disruptive
• visibility of running versions for consumers is important

Automatically document required permissions for API endpoints

This is a leftover from #29.

Document development setup

It'd be helpful to have some really simple instructions for getting your own copy of relengapi running on your laptop.

• finish up skeleton and publicize in documentation

The username link points to an old location

If you login and click your email address, you're linked to /userauth/account rather than the correct /auth/account.

Enable logging of SQL statements

It's sometimes useful to see all of the SQL queries that things issue in the logs.

SQLAlchemy supports this with an echo=True argument to engine construction. We don't want to turn this on all the time, though!

E:224,34: Instance of 'tuple' has no 'utctimetuple' member (but some types could not be inferred) (maybe-no-member)

Need to disable this pylint test.

PROPOSAL: monkeypatch datetime module to set tzinfo for utc functions

I have a feeling we'll hit needs for tz conversions often, so I feel we should monkeypatch datetime's module classes for all utc specific things, by default, to provide tzinfo in their return values.

We could also provide a helper to make no tz naive returns, (e.g. by using the tzlocal module and/or enforcing no tz-unaware calls)

@djmitche thoughts before I start to implement?

Fix pep8 failing for base/relengapi/blueprints/docs/src/conf.py, base/setup.py and settings_example.py

I think in the relengapi pep8 checks, these files aren't being checked, so I happened to spot this because of e.g. this failure in mapper:

https://travis-ci.org/petemoore/mapper/builds/26239093#L511

Here, it is running the pep8 checks against everything, and since we install build-relengapi using pip install -e it finds it.

I could get around this by checking out somewhere else - but I think it is good to have these files pep8 compliant anyway (and maybe we should have a separate issue to see why they are not checked in the relengapi pep8 tests - haven't looked into this yet).

Port AWS management

Everything that awsmanager1 does could be done via relengapi. Among other things, that would avoid the desire to run automated SSH's to awsmanager1 so that it can use Amazon's HTTP API.

Ability to instantiate cron-like tasks (tasks on a schedule)

For slaveloan at least we'll need the ability to have something run periodically, (ala cron). This is primarily incase an older task fails we will want the ability to retry it.

Useful features could include:

• escalation in case of repeat failures
• One invocation-at-a-time (hard across multiple webheads)
• able to retrieve run-logs/status/return-code/value somehow --- especially if using celery tasks

@djmitche - Any thoughts as to implementation details, eta in terms of being able to achieve this, etc?

Use 'blessings' to generate nicely colored output

Erik Rose has a nice library called "blessings" that makes it easy to colorize output. Let's use that to beautify the console logging when developing.

Obviously this is minor and trivial :)

Use 'wrapt' to build solid decorators

I learned about 'wrapt' at PyCon. It lets you build decorators that can work on instance methods, class methods, static methods, and functions.

Automatically index mentions of authz permissions

Refer to these using something like

  :ra:permission:`base.tokens.issue`

and then index every occurrence.

Support Flask Flashing

Flask has a nice feature for "flashing" messages on the next page load. The UI
template should support that.

Merge the 'relengapi-docs' distribution into 'relengapi'

Keeping these two separate, and in subdirectories, is just confusing for me and probably others.

hlep

Implement a frontend for Mozpool

This can begin as simple proxy to the backend, with the appropriate backend
server selected intelligently.

It could morph into a more thorough implementation of the fronend, with AMQP
used to communicate with the imaging servers.

Port Buildapi

Buildapi is already mostly an API, and can be ported relatively easily to relengapi.

It is written in Paste/Pylons, so it's not a copy/paste job, but close.

The use of RabbitMQ on the backend to talk to master should probably be
replaced with Celery nodes running on the buildmasters.

Implement token auth for automated access

We'll need some way for machines to authenticate to releng API. oAuth2 is not
suitable here, because its tokens are tied back to a "real" user. We don't
want all of our buidlslaves to stop working when someone gets removed from
LDAP!

Simple randomly-generated tokens could do the trick, but Hawk looks like a
promising way to implement this. It has the advantage of short-lived tokens,
in case they get disclosed.

Port jacuzzi allocator

The "frontend" is fairly simple, and a working version exists in the git history (git log jacuzzi-allocator should find it).

However, https://github.com/bhearsum/static-jacuzzis has two scripts to generate its content -- allocate.py and manage_jacuzzis.py, each of which are fairly expensive in their own way. Both draw configuration from the repository itself, and that configuration is accessible by sheriffs as well as by releng.

So we'd need to replace config.json by some kind of configuration table with a nice UI, and run the equivalent of allocate.py and manage_jacuzzis.py periodically in a celery task, with the output being written to the database for consumption by the "frontend".

These scripts depend on

• slavealloc (#11)
• schedulerdb - perhaps data available via buildapi (#13)
• usable_slaves.json (#14, I guess .. I'm not sure where the countless little crontasks fall right now)
  It doesn't need to wait until those are ported, but it might help.

Document oAuth2 support

(assuming it's being kept)

UI to list authenticated apps, available clients, add/remove clients

Users need some UI to be able to manage their oAuth2 associations and see their
permissions.

Port releng reports to relengapi

• Buildfaster (https://hg.mozilla.org/build/braindump/file/efb2abc7fba1/reports/buildfaster_report.py)
• Buildduty
• Slave-health
• Reportor

Reports should be generated on celery nodes, using beat for scheduling

Use Bootstrap to be less ugly

Bootstrap makes it at least a little bit difficult to look bad.

Remaining:

• update documentation to include bootstrap and updated angular version
• add moment.js and use it for badpenny (and doc)
• use directives to make badpenny work better (TODO items)
• use directives to make tokens work better (TODO item)
• X fix documentation navbar
• remove reference to badpenny.css

Add a way for blueprints to add top-level UI links

The HTML view of the main landing page for the API should have a nicely
formatted list of links to interesting bits of any installed blueprints.

This should be simple to implement for blueprints.

Document how to create a many-to-many relationship

This is done in some of the blueprints already, but isn't obvious from the SQLAlchemy docs.

Use Alembic for DB migrations

This will allow us to support automatic DB updates.

Implement scheduled tasks (badpenny)

This should be possible using Celery with Beat.

This will need some mechanism to ensure that one and only one Beat instance is running at any time.

• "badpenny" blueprint with a table of tasks, their last run, last status, etc.
• a way to register a task for scheduled execution
• display registered tasks in the UI
• 'relengapi badpenny --run' for crontasks
• logging support
  • move logs to their own table
  • connect with Python logging during Celery task execution
• locking support
  • as in, don't run a task while it's still running; but this needs a way to avoid tasks getting "wedged" when for example a celery node fails
• alerting support
  • probably left for after 1.0.0 -- a way to indicate what a "hard" failure of a task looks like, e.g., 10 out of the last 15 jobs failed, and use that to differentiate a "warning" task status from a "failed" task status.

Port slaveapi

Slaveapi is mostly backend tasks, which currently run on a single host.
They'll need to be switched around to run as Celery tasks, and tasks that must
wait for a significant time should operate in a way that they can continue if a
celery node fails.

Automatically document DB tables

Databases are specified in code, so it should be possible to include them in the generated documentation.

Test auth blueprint

Branched from #2

Connect with graphite

Metrics about relengapi should get fed to graphite for trending.