georgeyk / loafer

Asynchronous message dispatcher - Currently using asyncio and amazon SQS

Loafer

---

Loafer is an asynchronous message dispatcher for concurrent tasks processing.

Currently, only AWS SQS is supported

Features:

• Encourages decoupling from message providers and consumers
• Easy to extend and customize
• Easy error handling, including integration with sentry
• Easy to create one or multiple services
• Generic Handlers
• Amazon SQS integration

It requires Python 3.6+ and is very experimental at the moment, expect a lot of changes until the first major version.

Example

A simple message forwader, from source-queue to destination-queue:

from loafer.ext.aws.handlers import SQSHandler
from loafer.ext.aws.routes import SQSRoute
from loafer.managers import LoaferManager


routes = [
    SQSRoute('source-queue', handler=SQSHandler('destination-queue')),
]


if __name__ == '__main__':
    manager = LoaferManager(routes)
    manager.run()

Documentation

Check out the latest Loafer full documentation at Read the Docs website.

loafer's Issues

Add meta attributes to message handler

Force handlers to receive at least 2 parameters: message, meta
"meta" wll be a dictionary containing the message metadata (initially, source name and/or route name)

Another option is to create a fixed message structure like:
message = {'content': message. 'meta': {}}

First approach seems a little better but will break existing implementations.

Error consuming sqs messages

I am getting the following error:

exception (<class 'ValueError'>, ValueError("<loafer.ext.aws.message_translators.SQSMessageTranslator object at 0x10faf1610> failed to translate message={'MessageId': 'd20819f2-406f-4f96-a618-544b477b1a3d', 'ReceiptHandle': 'AQEBFK5WHeaLkDQ2nOLhsr7BBeyVIAsSGManlBfE8i8g5gI5Rh16QkTONbtE+cMgZSzlTnOlLAgzxab3ZSABycM9lQ2HOQtx/x6fc1c3y7VL9f9jwD0jxtU992sPUxtk2uf8I1wACULQh6TbnnK6hXbCB/4yZnWuP09xVlbQhyZTWOPjB//JKsjx6LXphHohmguw++kgzndVIL8FzW8alQACGxu3kXN5nMgFBLSNZYFXEkOJeS/0QaTS7qgK9uIqmFFSEZRGImrMv7CwhBRgFRhKPmrAQAFt3DJHlhyz27WXEuQZm36Jxfb/MMrFaD3rhjpf4p5ZvcnQ7WTA+0+fZ6fTs5nr7TsFCkZfoJS130OjIqxCUoHmo7npWum2Xyt0k4kZQcrwhKBrwlab72D7nnYrkQ==', 'MD5OfBody': '8ca1da029d8479102410a867ed661515', 'Body': '36|0|123456789'}"), <traceback object at 0x111d87870>) received

Are you still very experimental?

enable parametrized credentials

fix instantiation of https://github.com/georgeyk/loafer/blob/master/loafer/ext/aws/bases.py#L21

circleci 2 + tox + pyenv

Find out how to setup tox and execute the tests against multiple python versions without duplicating entries in circleci config.

Add retrying feature

https://github.com/rholder/retrying or https://github.com/danielfm/pybreaker

brainstorming

Enable creation of multiple instances of the same route/handler

Check usage of .env files from prettyconf

It seems we are not using prettyconf correctly with .env files, requiring the file to exists in the package root (loafer package).
It should work with .env files from the user project root.

use internal asyncio.Queue

loafer stops unexpectedly the receive of the poll of messages after time

Hello, I have a problem with the worker stopping receiving messages.

Environment

I will try to explain the situation. I have two SQS Queues. The loafer is v1.3.2.

The loafer, as well as queues, is set to receive every 10 messages (maximum) in short polling mode (WaitTimeSeconds = 0).

At AWS, at ECS, I have 6 consumers for each queue. About 60,000 messages are processed every 5 minutes when all consumers are fully operational.

The first worker with loafer usually receives a good amount of messages within minutes (approximately 80,000 messages for 5 minutes).

The handler function processes and can send between 1 and 12 messages to the second queue (using SQSHandler, publish method).

The second worker, also with loafer, is responsible for consuming the messages produced by the first worker handler.

Behavior

After a while, only the consumers from queue 1 stop receiving messages. The strange thing is that no errors are logged, and the containers are not dropped. The cpu usage is 0% (from docker stats).

Has anyone had this problem?
Does the SQS have a limit?

What am I doing?

To get around the problem, I am monitoring the ECS logs, and when there is no further message processing, I kill the task so that ECS uploads a new task and continues processing.

concurrent and multithreaded

Also related with #29
We should consider something like this:

• the main thread and loop control/receive the signals and manages "worker" threads and loops
• every loop own a thread (or threads) or every thread own a loop ?

Check the thread-pool executor for study and reference.
We should be able to configure the number of threads globally or by route ?

Support sentry_sdk

The library supports sending errors to Sentry, but only using the deprecated Raven, and not the new sentry_sdk.

Provide HTTP factories

The library does not work with aiobotocore v0.12.0

There's a change in aiobotocore's version 0.12.0 that removes a loop parameter from a get_session method and it makes loafer stop working.

Route switcher flag

Add enabled boolean attribute to route, so we can restrict the route execution using the flag.

Refactor message translations

Create an isolated pipeline of chainable/reusable "translators"

Support error handlers

Requires python 3.5.2:
https://docs.python.org/3/library/asyncio-eventloop.html#error-handling-api

Support for aiobotocore >= 1.0.0

The old version of aiobotore returns directly to the client when you call the create_client. Now with this last version the client needs to be instantiated with an async context manager, broken some pieces because they receive a context generator, not the botocore client:

old version:
https://github.com/aio-libs/aiobotocore/blob/904332b2ab0a3cf44c49af13ac04173a71f9e647/aiobotocore/session.py#L19

new version:
https://github.com/aio-libs/aiobotocore/blob/b86e8ce76c8282fd71b78770b00b7e23a6cf545a/aiobotocore/session.py#L43

parametrize max_workers in LoaferManager

It should change LoaferDIspatcher and LoaferRunner creation

with await lock Deprecation on python 3.7

loafer/loafer/dispatchers.py

Line 24 in e878f27

with await self._semaphore:

My pytest shows a deprecation on line 24 (It isn't supposed to fix it immediatelly):

DeprecationWarning: 'with await lock' is deprecated use 'async with lock' instead

😁