srault95 / task

A high-performance contender for asynchronous task execution; simpler to use than Celery.

Marrow Task

© 2013-2015 Alice Bevan-McGregor and contributors.

https://github.com/marrow/task

1. What is Marrow Task?

Marrow Task is a light-weight yet powerful, fully tested, Python 2.7+ and 3.3+ compatible distributed task execution system. It is built entirely around several core (though under-utilized) features of MongoDB and builds upon the powerful MongoEngine object document mapper.

It is similar to Celery or other distributed task systems excluding the fact that if your application already uses MongoDB, Marrow Task can be far easier to set up and integrate. It is substantially simpler to configure, and supports several preculiar features out of the box:

• Deferred execution of generators. Write code that yields data on one host, and consumes the data as it is yielded on another.
• Track task progress in realtime.
• Gather statistics on task execution automatically.
• Avoid data serialization formats (i.e. pickle) by using only native BSON types.
• The ability to execute any function that can be imported, including classmethods and staticmethods of classes.
• The ability to subclass the Task type and execute methods on instances of MongoEngine database objects.
• Being Futures-compatible you can use Marrow Task as a drop-in replacement for a more naive thread or process pool.
• Explicitly does not add any new infrastructure dependencies to your own application: no need for an additional queue like Zeromq or Rabbitmq, or storage system like Redis, and the accompanying service monitoring or even extra hardware required by such.

1.1 Goals

Marrow Task was created to be a light-weight alternative to systems like Celery that, as noted above, explicitly does not add to the infrastructure maintenance burden of systems administrators. Additionally, it was intended to fill several gaps in functionality in competing systems, often excluded for performance reasons, such as task timeline analytics, while providing an interface compatible with Futures, a universally standardized API for deferred execution.

While Celery didn't exactly fit some of our own requirements, Marrow Task might not fit yours. Always plan out your application's requirements before deciding on the right tools for the job.

2. Installation

Installing marrow.task is easy, just execute the following in a terminal:

pip install marrow.task

If you add marrow.task to the install_requires argument of the call to setup() in your applicaiton's setup.py file, Marrow Task will be automatically installed and made available when your own application or library is installed. We recommend using "less than" version numbers to ensure there are no unintentional side-effects when updating. Use marrow.task<1.1 to get all bugfixes for the current release, and marrow.task<2.0 to get bugfixes and feature updates while ensuring that large breaking changes are not installed.

2.1. Requirements

Due to some of the fairly advanced things Marrow Task does, certain versions of Python are explicitly not supported. Notably due to the removal of function wrappers (which add im_class to the mix) in Python 3.0, only Python 3 versions later than 3.3 (which introduced __qualname__ to replace im_class magic) will work if you make use of static or class methods as execution targets.

This restriction is soft; if you choose to install Marrow Task on earlier Python 3 versions we will issue a huge warning up front, but the remainder of the functionality should continue to operate.

Additionally, several other components are needed in order to make use of Marrow Task:

• A MongoDB 2.0 or later service.
• MongoEngine

2.2. Development Version

Development takes place on GitHub in the marrow.task project. Issue tracking, documentation, and downloads are provided there.

Installing the current development version requires Git, a distributed source code management system. If you have Git you can run the following to download and link the development version into your Python runtime:

git clone https://github.com/marrow/task.git
(cd task; python setup.py develop)

You can then upgrade to the latest version at any time:

(cd task; git pull; python setup.py develop)

If you would like to make changes and contribute them back to the project, fork the GitHub project, make your changes, and submit a pull request. This process is beyond the scope of this documentation; for more information see GitHub's documentation.

3. Basic Concepts

TBD

4. Version History

Version 1.0

• Initial release.

5. License

Marrow Task has been released under the MIT Open Source license.

5.1. The MIT License

Copyright © 2013-2015 Alice Bevan-McGregor and contributors.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.