Add support for asyncio (Python>=3.4) about backoff

[bgreen-litl]

Hey, this is great. Thanks.

It had recently crossed my mind to see if I could support asyncio but I hadn't put any real thought into it. I'm by no means an expert in asynchronous python. It is great that you have put some real thought into how it might work.

I'm inclined to explore if we could roll async support into the core library (option 1. above) but I don't yet have a clear sense of what that entails. I'm quite fond of the current API and I wouldn't want to do it if it ended up polluting it with clunky options or an undue expansion of its surface.

A few questions off the top of my head:

1. Could the API potentially be exactly the same and the implementation switching based only on whether the the target and handlers functions are coroutines? Or would we need something more explicit?
2. Do you know of any other libraries that support something similar? i.e. is there a best practice out there already for this sort of thing?
3. If the decorated target is a coroutine, does that imply that the handlers have to be too? Can those be mix and matched?
4. Some keywords args can also be callable so their values can be evaluated at runtime. Would we have to or want to support async for those as well?

[rutsky]

Could the API potentially be exactly the same and the implementation switching based only on whether the the target and handlers functions are coroutines? Or would we need something more explicit?

Yes, API can left exactly the same. You can see in #23 that API of on_predicate/on_exception hasn't changed (they are still normal decorators with the same parameters as in backoff).

Do you know of any other libraries that support something similar? i.e. is there a best practice out there already for this sort of thing?

Other libraries that implements exponential and other backoffs functionality? I haven't found any compatible with asyncio.

If the decorated target is a coroutine, does that imply that the handlers have to be too? Can those be mix and matched?

Handlers can be regular functions even if target is coroutine.
From coroutines you can freely call regular functions with the exception that those functions must be non-blocking (i.e. time.sleep(), long computations, blocking networking are not allowed).
In #23 I explicitly convert handlers to coroutines, which should work assuming that handler doesn't do any blocking operations.

Some keywords args can also be callable so their values can be evaluated at runtime. Would we have to or want to support async for those as well?

It depends. Support both sync and async versions of them is easy, but I don't think it's feasible to have them async --- is meaningful for them to read/compute value from network/database (i.e. value obtaining may require async operation)?

I'm inclined to explore if we could roll async support into the core library (option 1. above) but I don't yet have a clear sense of what that entails. I'm quite fond of the current API and I wouldn't want to do it if it ended up polluting it with clunky options or an undue expansion of its surface.

External interface can be left the same, as seen in #23 . But implementation of on_predicate/on_exception will be either messy with conditions/switches, or be duplicated.
I see that on_predicate and on_exception shares a lot in common, but you haven't merged their implementations (probably because it will lead to messy code).

If we decide to left sync and async implementations of on_predicate/on_exception in separate function it should be quite easy to implement. Something like:

def on_predicate(...):
    ...

    def decorate(target):
        @functools.wraps(target)
        def retry_sync(*args, **kwargs):
            ...

        @functools.wraps(target)
        @asyncio.coroutine
        def retry_async(*args, **kwargs):
            ...

        if asyncio.iscouroutinefunction(target):
            return retry_async
        else:
            return retry_sync

IMO I see two things relatively complicated:

1. Trying to extract common functionality from sync/async versions of on_predicate/on_exception. Or forget about it and accurately maintain four versions of retry.

2. Properly handle old Pythons, where asyncio is not available in backoff.py, tests, CI, etc.

[bgreen-litl]

External interface can be left the same, as seen in #23 . But implementation of on_predicate/on_exception will be either messy with conditions/switches, or be duplicated.
I see that on_predicate and on_exception shares a lot in common, but you haven't merged their implementations (probably because it will lead to messy code).

Ok, great. I think you're exactly right about where the difficulties will lie.

I'm most concerned with keeping the API clean and slightly less concerned with what the implementation looks like. I think that's the best policy for library code. However, of course maintenance is a concern, and you're exactly right the 2 current implementations are similar but mostly don't share code. The reason is these were once much simpler and grew up to their present state as features were added. Looking now with a critical eye, we could at minimum factor out a few helper functions. This might help to make the 4 implementations we end up with a little less unwieldy. Maybe I'll see how far I get doing this as a first step, since it probably makes sense anyway.

2. Properly handle old Pythons, where asyncio is not available in backoff.py, tests, CI, etc.

I did a little exploratory coding for this, and I think the biggest problem here is going to be with new Python 3 syntax. If the asyncio library isn't available in an python 2.7, that's not a problem because we can just conditionally not import it at runtime, and conditionally not define the retry_async version of the retry function. But if the code fails to compile because of new syntax e.g. yield from then it won't work and we are stuck. So the question is: is that new syntax required for the async path to work?

[rutsky]

2. Properly handle old Pythons, where asyncio is not available in backoff.py, tests, CI, etc.

I did a little exploratory coding for this, and I think the biggest problem here is going to be with new python 3 syntax. If the asyncio library isn't available in an python 2.7, that's not a problem because we can just conditionally not import it at runtime, and conditionally not define the retry_async version of the retry function. But if the code failed to compile because of new syntax e.g. yield from then it won't work and we are stuck. So the question is: is that new syntax required for the async path to work?

Uh, I forgot that yield from appeared only in Python 3.3. Yes, new syntax is required to use asyncio's coroutines (yield from and asyncio.coroutine are used in Python >=3.4, await and async def keywords may be used in Python >= 3.5).
We can move all async stuff to separate module, like _backoff_async.py, and import it only on Python >= 3.4.

[bgreen-litl]

A separate _backoff_async module which is only imported in Python >=3.4 sounds like a good idea if we can make it work. I know that 3.5 has those new keywords, but I believe we could stick with a 3.4 style implementation without keywords that would still work in 3.5. Is that true?

I suspect we're going to end up with the four different implementations, so as a first step, I did some simple refactoring to share more code in the form of helper functions. A PR for it is here: #24 I think at at minimum it makes the two current implementations a little easier to follow. The async implementations should be able to be modified to use those as well.

[rutsky]

A separate _backoff_async module which is only imported in Python >=3.4 sounds like a good idea if we can make it work. I know that 3.5 has those new keywords, but I believe we could stick with a 3.4 style implementation without keywords that would still work in 3.5. Is that true?

Yes, it should be enough to support Python 3.4 style only --- it works under Python >= 3.4, and we don't need at this moment syntax features appeared in Python 3.5 and 3.6.

I suspect we're going to end up with the four different implementations, so as a first step, I did some simple refactoring to share more code in the form of helper functions. A PR for it is here: #24 I think at at minimum it makes the two current implementations a little easier to follow. The async implementations should be able to be modified to use those as well.

Looks good to me. Would you like to convert single-file backoff.py to package with set of modules?
This way we can create _common.py and move to it common functions for sync and async implementations (_maybe_call, _init_wait_gen, _next_wait).
Create _wait_gens (or any other name) to put expo, fibo, constant; and probably other private modules.

backoff.__init__.py will import _backoff_async.py and _backoff_async.py should not import backoff.__init__.py to prevent circular dependencies, so we need to place all common functions in separate modules.

[bgreen-litl]

Looks good to me. Would you like to convert single-file backoff.py to package with set of modules? This way we can create _common.py and move to it common functions for sync and async implementations (_maybe_call, _init_wait_gen, _next_wait). Create _wait_gens (or any other name) to put expo, fibo, constant; and probably other private modules.

Yes, this sounds good to me.

backoff.init.py will import _backoff_async.py and _backoff_async.py should not import backoff.init.py to prevent circular dependencies, so we need to place all common functions in separate modules.

Yes, sounds right.

Do you want to put a PR together for this? If not, I can see how far I get as I have time.

[rutsky]

Do you want to put a PR together for this? If not, I can see how far I get as I have time.

If you can do initial split --- this would be great, then I can do PR adding _backoff_async.py and tests for it (in separate file too, so probably it is a good idea to move backoff_tests.py to tests subdirectory; I would also rename it to test_backoff.py and run pytest on that tests directory; async tests may be in test_backoff_async.py).
I'm limited in time too, but if you won't be able to do splitting, I can try to do PR for that.

[bgreen-litl]

Here's my pass at making backoff into a package with private submodules. The idea is that you would just need to implement the two functions in _async.py Let me know if this looks like what you need. #25

[bgreen-litl]

#25 is merged to master!

[bgreen-litl]

#26 is merged to master! I also sorted out the pep8, tests, and coverage across platforms so all Travis builds pass 'make check' now.

I'll need to work on some documentation next. Let me know if you have any compelling simple examples which you think would be good in the docs.

Thanks for your work on this!

[rutsky]

@bgreen-litl your changes for checking code LGTM, thanks!

I opened #28 with README update, which should complete this issue (only changelog update and release publishing will be left).

Here is complete example of async usage of backoff parts of which I used in README, in case you would like to experiment with it: https://gist.github.com/rutsky/6b2b891dff399536ce98c3d7445e39c7

[bgreen-litl]

@rutsky 1.4.0 with async support is released!

[rutsky]

Great, thanks!

Would you like to make announcement on the aio-libs mailing list?
I can do this for you, if you want.

[bgreen-litl]

Sure. That would be great. Thanks.

…

On Sun, Feb 5, 2017, 3:17 PM Vladimir Rutsky ***@***.***> wrote: Great, thanks! Would you like to make announcement on the aio-libs mailing list <https://groups.google.com/forum/#!forum/aio-libs>? I can do this for you, if you want. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABaiMRcVvZ3CUR1kjc2r97l-kdH-LWK9ks5rZi5kgaJpZM4LppWB> .

[rutsky]

I wrote release e-mail here:
https://groups.google.com/forum/#!topic/aio-libs/eR1pmAwbcTk

[rutsky]

@bgreen-litl just FYI, there is also riprova library that implements backoff, it supports asyncio too.