ambientlighter / celery-director

Simple and rapid framework to build workflows with Celery

---

Director is a simple and rapid framework used to manage tasks and build workflows using Celery.

The objective is to make Celery easier to use by providing :

• a WebUI to track the tasks states,
• an API and a CLI to manage and execute the workflows,
• a YAML syntax used to combine tasks into workflows,
• the ability to periodically launch a whole workflow,
• and many others.

See how to use Director with the quickstart and guides in the documentation.

Installation

Install the latest version of Director with pip (requires Python 3.6 at least):

pip install celery-director

Usage

Write your code in Python

# tasks/orders.py
from director import task
from .utils import Order, Mail

@task(name="ORDER_PRODUCT")
def order_product(*args, **kwargs):
    order = Order(
      user=kwargs["payload"]["user"],
      product=kwargs["payload"]["product"]
    ).save()
    return {"id": order.id}

@task(name="SEND_MAIL")
def send_mail(*args, **kwargs):
    order_id = args[0]["id"]
    mail = Mail(
      title=f"Your order #{order_id} has been received",
      user=kwargs["payload"]["user"]
    )
    mail.send()

Build your workflows in YAML

# workflows.yml
product.ORDER:
  tasks:
    - ORDER_PRODUCT
    - SEND_MAIL

Run it

You can simply test your workflow in local :

$ director workflow run product.ORDER '{"user": 1234, "product": 1000}'

And run it in production using the director API :

$ curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"project": "product", "name": "ORDER", "payload": {"user": 1234, "product": 1000}}' \
  http://localhost:8000/api/workflows

Read the documentation to try the quickstart and see advanced usages of Celery Director.

Project layout

.env                # The configuration file.
workflows.yml       # The workflows definition.
tasks/
    example.py      # A file containing some tasks.
    ...             # Other files containing other tasks.

Commands

• director init [path] - Create a new project.
• director celery [worker|beat|flower] - Start Celery daemons.
• director webserver - Start the webserver.
• director workflow [list|show|run] - Manage your project workflows.

License

See https://github.com/ovh/celery-director/blob/master/LICENSE