GithubHelp home page GithubHelp logo

thorn's Introduction

http://thorn.readthedocs.io/en/latest/_images/thorn_banner.png

Python Stream Processing

Build status coverage BSD License Thorn can be installed via wheel Supported Python versions. Support Python implementations.

Version:1.5.2
Web:http://thorn.readthedocs.io/
Download:http://pypi.python.org/pypi/thorn/
Source:http://github.com/robinhood/thorn/
Keywords:event driven, webhooks, callback, http, django

Thorn is a webhook framework for Python, focusing on flexibility and ease of use, both when getting started and when maintaining a production system.

The goal is for webhooks to thrive on the web, by providing Python projects with an easy solution to implement them and keeping a repository of patterns evolved by the Python community.

  • Simple

    Add webhook capabilities to your database models using a single decorator, including filtering for specific changes to the model.

  • Flexible

    All Thorn components are pluggable, reusable and extendable.

  • Scalable

    Thorn can perform millions of HTTP requests every second by taking advantage of Celery for asynchronous processing.

A webhook is a fancy name for an HTTP callback.

Users and other services can subscribe to events happening in your system by registering a URL to be called whenever the event occurs.

The canonical example would be GitHub where you can register URLs to be called whenever a new change is committed to your repository, a new bugtracker issue is created, someone publishes a comment, and so on.

Another example is communication between internal systems, traditionally dominated by complicated message consumer daemons, using webhooks is an elegant and REST friendly way to implement event driven systems, requiring only a web-server (and optimally a separate service to dispatch the HTTP callback requests).

Webhooks are also composable, so you can combine multiple HTTP callbacks to form complicated workflows, executed as events happen across multiple systems.

Notable examples of webhooks in use are:

Site Documentation
Github https://developer.github.com/webhooks/
Stripe https://stripe.com/docs/webhooks
PayPal http://bit.ly/1TbDtvj

This example adds four webhook events to the Article model of an imaginary blog engine:

from django.urls import reverse
from thorn import ModelEvent, webhook_model

@webhook_model   # <--- activate webhooks for this model
class Article(models.Model):
    uuid = models.UUIDField()
    title = models.CharField(max_length=100)
    body = models.TextField()

    class webhooks:
        on_create = ModelEvent('article.created')
        on_change = ModelEvent('article.changed'),
        on_delete = ModelEvent('article.removed'),
        on_publish = ModelEvent(
            'article.published',
            state__eq='PUBLISHED',
        ).dispatches_on_change(),

    def get_absolute_url(self):
        return reverse('article:detail', kwargs={'uuid': self.uuid})

Users can now subscribe to the four events individually, or all of them by subscribing to article.*, and will be notified every time an article is created, changed, removed or published:

$ curl -X POST                                                      \
> -H "Authorization: Bearer <secret login token>"                   \
> -H "Content-Type: application/json"                               \
> -d '{"event": "article.*", "url": "https://e.com/h/article?u=1"}' \
> http://example.com/hooks/

The API is expressive, so may require you to learn more about the arguments to understand it fully. Luckily it's all described in the Events Guide for you to consult after reading the quick start tutorial.

Version Requirements

Thorn version 1.0 runs on

  • Python (2.7, 3.4, 3.5)
  • PyPy (5.1.1)
  • Jython (2.7).
  • Django (1.8, 1.9, 1.10)
    Django 1.9 adds the transaction.on_commit() feature, and Thorn takes advantage of this to send events only when the transaction is committed.

Thorn currently only supports Django, and an API for subscribing to events is only provided for Django REST Framework.

Extending Thorn is simple so you can also contribute support for your favorite frameworks.

For dispatching web requests we recommend using Celery, but you can get started immediately by dispatching requests locally.

Using Celery for dispatching requests will require a message transport like RabbitMQ or Redis.

You can also write custom dispatchers if you have an idea for efficient payload delivery, or just want to reuse a technology you already deploy in production.

Go immediately to the django-guide guide to get started using Thorn in your Django projects.

If you are using a different web framework, please consider contributing to the project by implementing a new environment type.

Thorn was inspired by multiple Python projects:

You can install thorn either via the Python Package Index (PyPI) or from source.

To install using pip,:

$ pip install -U thorn

Download the latest version of thorn from http://pypi.python.org/pypi/thorn/

You can install it by doing the following,:

$ tar xvfz thorn-0.0.0.tar.gz
$ cd thorn-0.0.0
$ python setup.py build
# python setup.py install

The last command must be executed as a privileged user if you are not currently using a virtualenv.

You can install the latest snapshot of thorn using the following pip command:

$ pip install https://github.com/robinhood/thorn/zipball/master#egg=thorn

For discussions about the usage, development, and future of Thorn, please join the thorn-users mailing list.

Come chat with us on IRC. The #thorn channel is located at the Freenode network.

If you have any suggestions, bug reports or annoyances please report them to our issue tracker at https://github.com/robinhood/thorn/issues/

Development of Thorn happens at GitHub: https://github.com/robinhood/thorn

You are highly encouraged to participate in the development of thorn. If you don't like GitHub (for some reason) you're welcome to send regular patches.

Be sure to also read the Contributing to Thorn section in the documentation.

This software is licensed under the New BSD License. See the LICENSE file in the top distribution directory for the full license text.

thorn's People

Contributors

ask avatar fcurella avatar imshriram avatar jamshedvesuna avatar jbinney avatar jefftriplett avatar jonashaag avatar joshdrake avatar moiseshiraldo avatar saevarom avatar smn avatar viyatb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

thorn's Issues

Customizable webhook headers

Currently, it's possible to customize the payload sent by using webhook_payload.

I think it could be useful to add a way to customize headers as well. It would allow, for example, to include an Authorize header so that the receiving endpoint could associate the event to a user (eg a 'system' user with admin permission).

This could be implemented as either as webhook_header in the model (that'd be my favorite option), or even just some kind of global settings.

If this is something you think it's useful, I can work on a PR.

Celery dispatcher does not set custom headers

Problem

Celery dispatcher does not set custom HTTP headers for webhook requests.

I am pretty sure the headers are being dropped because they are not included in the returned dict when the Request.as_dict method is called from here

I propose to ensure that Request.as_dict returns the existing dict, but with the request instances headers (_headers) included. If this seems like a reasonable solution please let me know and I'll get a PR submitted.

Environment

thorn 1.5.0

Remove unique_together (url, event) constraint

We'd like to have more than one subscription to the same event/URL combination.

I tried to remove the constraint in #24 and it seems to pass the test suite. So it's not actually assumed anywhere in the code.

is there any ways to format values with just one line?

I've get the sql command like insert a(k,k,k) values (a,a,a),(b,b,b),(c,c,c);

Once I formatted the sql commands, it ran out

insert into
  a (
    k,
    k,
    k
  )
values
  (
    a,
    a,
    a
  ),
  (
    b,
    b,
    b
  );

is there anyways to format the commands to

insert into a(k,k,k) values
(a,a,a),
(b,b,b);

Is this project maintained?

I see some indicators that this project may no longer be maintained. I am considering incorporating this into our code base and I wanted to know if it would be possible to get some sort of confirmation from Robinhood that there is an intention to maintain this project over the long term.

Any comment from Robinhood on this would be greatly appreciated.

Tagging @ask due to status of largest contributor and apparent status of current staff at Robinhood.

Change THORN_DISPATCHER during runtime

I'm working on writing some unit / integration tests for our webhook system and was wondering if there is a way to change the THORN_DISPATCHER setting during individual tests. or maybe just "reload" thorn somehow. My goal is to disable thorn during testing and then turn it on again for specific tests.

Typically I would just do this with something like:

    def setUp(self):
        settings.THORN_DISPATCHER = 'celery'

    def tearDown(self):
        settings.THORN_DISPATCHER = 'disabled'

But when I do this it looks like it still uses the default settings that were originally set. I think think this is because of the cached_property usage.

I looked through the thorn tests and couldn't find anything that resembled this and the google didn't seem very helpful...

Ordering guarantees

Hi,
Great stuff! ๐Ÿ‘

I didn't see any info in the documentation regarding ordering guarantees of posting callbacks (in success and failure scenarios).

Could you please elaborate?

Thanks!
Erik.

Local dispatcher error handlers don't work

Using the local dispatcher (THORN_DISPATCHER = 'default'), not the Celery-based one, if the webhook raises an error (e.g. due to a timeout), the on_timeout and on_error handlers passed to send() aren't called. Instead the following internal error is raised:

  File "thorn/events.py", line 124, in send
    timeout=timeout, on_timeout=on_timeout,
  File "thorn/events.py", line 146, in _send
    allow_keepalive=self.allow_keepalive,
  File "thorn/dispatch/base.py", line 73, in send
    **kwargs
  File "vine/synchronization.py", line 57, in __init__
    [self.add_noincr(p) for p in promises or []]
  File "vine/synchronization.py", line 57, in <listcomp>
    [self.add_noincr(p) for p in promises or []]
  File "vine/synchronization.py", line 82, in add_noincr
    p.then(self)
AttributeError: 'NoneType' object has no attribute 'then'

I fixed (?) this with the following patch, although I am very unsure if it's correct. jonashaag@15453b2

Test case:

import django; django.setup()

import thorn
import thorn.django.models

thorn.django.models.Subscriber.objects.create(event='foo.*', url='http://8.8.8.8')

e = thorn.Event('foo.bar')
e.send({})

With my fix (?):

Traceback (most recent call last):
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connection.py", line 166, in connect
    conn = self._new_conn()
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connection.py", line 146, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x10faddda0>, 'Connection to 8.8.8.8 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "thorn/env3/lib/python3.6/site-packages/requests-2.18.4-py3.6.egg/requests/adapters.py", line 440, in send
    timeout=timeout
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "thorn/env3/lib/python3.6/site-packages/urllib3-1.22-py3.6.egg/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='8.8.8.8', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x10faddda0>, 'Connection to 8.8.8.8 timed out. (connect timeout=1)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    e.send({})
  File "thorn/thorn/events.py", line 124, in send
    timeout=timeout, on_timeout=on_timeout,
  File "thorn/thorn/events.py", line 146, in _send
    allow_keepalive=self.allow_keepalive,
  File "thorn/thorn/dispatch/base.py", line 73, in send
    **kwargs
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/synchronization.py", line 57, in __init__
    [self.add_noincr(p) for p in promises or []]
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/synchronization.py", line 57, in <listcomp>
    [self.add_noincr(p) for p in promises or []]
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/synchronization.py", line 82, in add_noincr
    p.then(self)
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/abstract.py", line 53, in then
    return self._p.then(on_success, on_error)
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/promises.py", line 173, in then
    callback.throw(self.reason)
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/synchronization.py", line 94, in throw
    self.p.throw(*args, **kwargs)
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/promises.py", line 217, in throw
    reraise(type(exc), exc, tb)
  File "thorn/env3/lib/python3.6/site-packages/vine-1.1.4-py3.6.egg/vine/five.py", line 178, in reraise
    raise value.with_traceback(tb)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='8.8.8.8', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x10faddda0>, 'Connection to 8.8.8.8 timed out. (connect timeout=1)'))

Generator expression must be parenthesized

I am currently using python 3.8, Django 3.0.2 thorn 1.5.2 in Arch Linux. When I try to import thorn to my script it returns following error:

Traceback (most recent call last):
  File "authbot.py", line 10, in <module>
    from thorn import ModelEvent, webhook_model
  File "/home/yorozuya3/Desktop/authtelegrambot/env/lib/python3.8/site-packages/thorn/__init__.py", line 9, in <module>
    from .events import Event, ModelEvent
  File "/home/yorozuya3/Desktop/authtelegrambot/env/lib/python3.8/site-packages/thorn/events.py", line 282
    '__now_' in k for k in keys(self.filter_fields),
    ^
SyntaxError: Generator expression must be parenthesized

Filtering Subscriptions - Sender Field

Hi,
I've looked at issue #7 and I'm not sure if it solves my issue, or if my case is more complex.

Given entities Transport Order, Company, User and Application.

A Transport Order is created by and "belongs to" a Producer Company. It is consumed by Consumer Companies. It maintains a m2m "consumers" relationship with companies.
Users are employees of the companies
Consumer companies only wish to be notified about updates/ deletes to Transport Orders they consume.
Applications are OAuth apps created by developers, to act on behalf of Users.

Ideal Flow

  1. Application on behalf of an employee of a consumer company registers a subscription to transport_order.change.

  2. Producer company edits an order consumed by the consumer company

  3. Consumer company's webhook is hit by the change, and not with changes to orders it does not subscribe to.

I don't think the sender_field is advanced enough to handle this, but hopefully I'm wrong?

Enable callback functions for the Celery dispatcher

The on_success, on_error and on_timeout callbacks are really useful (e.g. I intend to use them to store/show recent webhook deliveries to the customers, same way GitHub does).

I think it would be safe to enable them for the celery dispatcher if we look at Python's what can be pickled list:

  • functions defined at the top level of a module
  • built-in functions defined at the top level of a module
  • classes that are defined at the top level of a module
  • instances of such classes whose dict or the result of calling getstate() is picklable

I've done it myself and it's working fine, I've also fixed the corresponding tests (see the commit below). Would you consider merging the branch?

Thank you for your work!

instance-based events

It would be useful if subscribers could receive events only related to a specific instance.

Scenario:

A web application lets users create their own AddressBook. Each AddressBook have many Contacts:

class AddressBook(models.Model):
    user = models.ForeignKey('auth.User')

@webhook_model(
    on_create=ModelEvent('contact.created'),
)
class Contact(models.Model):
    name = models.CharField(max_length=255)
    address_book = models.ForeignKey(AddressBook, related_name='contacts')

There doesn't seem to be a way for a subscriber to receive contact.created events only for Contacts of a specific AddressBook.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.