python-trio / async_generator

Making it easy to write async iterators in Python 3.5

The async_generator library

Python 3.6 added async generators. (What's an async generator? Check out my 5-minute lightning talk demo from PyCon 2016.) Python 3.7 adds some more tools to make them usable, like contextlib.asynccontextmanager.

This library gives you all that back to Python 3.5.

For example, this code only works in Python 3.6+:

async def load_json_lines(stream_reader):
    async for line in stream_reader:
        yield json.loads(line)

But this code does the same thing, and works on Python 3.5+:

from async_generator import async_generator, yield_

@async_generator
async def load_json_lines(stream_reader):
    async for line in stream_reader:
        await yield_(json.loads(line))

Or in Python 3.7, you can write:

from contextlib import asynccontextmanager

@asynccontextmanager
async def background_server():
    async with trio.open_nursery() as nursery:
        value = await nursery.start(my_server)
        try:
            yield value
        finally:
            # Kill the server when the scope exits
            nursery.cancel_scope.cancel()

This is the same, but back to 3.5:

from async_generator import async_generator, yield_, asynccontextmanager

@asynccontextmanager
@async_generator
async def background_server():
    async with trio.open_nursery() as nursery:
        value = await nursery.start(my_server)
        try:
            await yield_(value)
        finally:
            # Kill the server when the scope exits
            nursery.cancel_scope.cancel()

(And if you're on 3.6, you can use @asynccontextmanager with native generators.)

Let's do this

• Install: python3 -m pip install -U async_generator (or on Windows, maybe py -3 -m pip install -U async_generator
• Manual: https://async-generator.readthedocs.io/
• Bug tracker and source code: https://github.com/python-trio/async_generator
• Real-time chat: https://gitter.im/python-trio/general
• License: MIT or Apache 2, your choice
• Contributor guide: https://trio.readthedocs.io/en/latest/contributing.html
• Code of conduct: Contributors are requested to follow our code of conduct in all project spaces.

How come some of those links talk about "trio"?

Trio is a new async concurrency library for Python that's obsessed with usability and correctness – we want to make it easy to get things right. The async_generator library is maintained by the Trio project as part of that mission, and because Trio uses async_generator internally.

You can use async_generator with any async library. It works great with asyncio, or Twisted, or whatever you like. (But we think Trio is pretty sweet.)

async_generator's Issues

ctypes.CDLL(None) doesn't work on Windows

I was asked to raise this here from Fuyukai/Kyoukai#19

I get a TypeError, bad argument type for built-in operation within Kyoukai, at an import statement (within asyncio_extras: from async_generator import yield_.

missing git tag 1.10

Hi,

On pypi the 1.10 is already published. it would be nice to have a corresponding git commit available via a pushed git tag.
Can you please tag and push 1.10 to github?
cheers

Questions

I saw your pycon lightning talk on youtube and thought it was pretty neat. await yield for async for has been something I've wanted since I first read about async for. I have made decorators that try to take care of all the __aiter__/__anext__ boilerplate code in the past, but I have never seen one that used a yield_ function like yours (I have always used queues to pass values). It's a pretty clever idea.

As I was reading through your code and testing differences between your implementation and mine, I had a few questions:

• Why YieldWrapper?

It looks like you are using it as a sentinal value to determine when you've reached the end of the coroutine chain, but why not just yield the value directly and then in send() you can use inspect.isawaitable(result) to determine if you're at the end. I don't know if you would need inspect.isgenerator or inspect.iscoroutine, but for all the tests I've done, inspect.isawaitable(result) has been sufficient.

This would have the added benefit of allowing the decorator to turn any generator into an async iterable (not just ones using yield_).

• Why a new ANextIter for every __anext__ call?

It seems like unnecessary overhead to create a new instance for each __anext__ call, why not just create an instance in __aiter__ (which is called once per async for) and give it a __anext__ that returns self? This also separates them more into an "async iterable" and "async iterator".

• Why comment out yield_from_?

I read your README, and I agree that an asend or athrow seem pretty weird, but I don't understand the problem with yield_from_. The most obvious and useful case I can think of is generator delegation, and as far as I can tell your (commented out) version does that just fine.

For reference, here's the version I was using (updated to adopt your yield_ ideas and be more flexible): https://gist.github.com/bj0/0313ee67766de6a04881

open_cancel_scope → CancelScope

should be de-deprecated ASAP

asynccontextmanager does not support callable classes

I was expecting something like the following example to work, but it does not. This originally came up in: tiangolo/fastapi#1204

from async_generator import asynccontextmanager

class GenClass:
    async def __call__(self):
        yield "hello"


cm = asynccontextmanager(GenClass())

with cm as value:
    print(value)

The result is an exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.6/site-packages/async_generator/_util.py", line 100, in asynccontextmanager
    "must be an async generator (native or from async_generator; "
TypeError: must be an async generator (native or from async_generator; if using @async_generator then @acontextmanager must be on top.

Deprecate async_generator

Now that Python 3.6 reached EOL, I believe this library does more harm than good and we should deprecate it and eventually archive it. What do you think?

`asynccontextmanager` fails in `3.7-dev`

Python version: Python 3.7.6+ (heads/3.7:b0a6ec256b)
OS: macOS 10.14.6

async_generator.asynccontextmanager started to fail in python 3.7-dev(which is also the 3.7-dev in pyenv).
If we change to use contextlib.asynccontextmanager, it works. It's why I'm thinking something wrong in async_generator when an async-generator is being closed. I'm sorry that I haven't dug into the cause.

This issue seems to happen since python/cpython@b76d5e9.

# example_code.py

import asyncio

from async_generator import asynccontextmanager


@asynccontextmanager
async def async_iterator():
    yield 1


async def run():
    async with async_iterator():
        ...


asyncio.run(run())

# Output of the executed code

an error occurred during closing of asynchronous generator <async_generator object async_iterator at 0x108632290>
asyncgen: <async_generator object async_iterator at 0x108632290>
Traceback (most recent call last):
  File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/Users/mhchia/projects/practice/python/async-generator/await-twice.py", line 13, in run
    ...
  File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 84, in __aexit__
    raise
  File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 14, in __aexit__
    await self._aiter.aclose()
RuntimeError: cannot reuse already awaited aclose()/athrow()

During handling of the above exception, another exception occurred:

RuntimeError: cannot reuse already awaited aclose()/athrow()
Traceback (most recent call last):
  File "/Users/mhchia/projects/practice/python/async-generator/await-twice.py", line 16, in <module>
    asyncio.run(run())
  File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/Users/mhchia/projects/practice/python/async-generator/await-twice.py", line 13, in run
    ...
  File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 84, in __aexit__
    raise
  File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 14, in __aexit__
    await self._aiter.aclose()
RuntimeError: cannot reuse already awaited aclose()/athrow()

Issues from Nicolas Boulenguez

In private email, Nicolas Boulenguez reported the following issues they encountered while working on packaging async_generator for Debian:

• The 1.9 tag is missing from Github.

• The source says that the license is "MIT or APACHE2 (your choice)" but the 1.9 tarball on pypi only mentions MIT.

• The 1.9 tarball on pypi contains no manual, neither the source nor the formatted version.

• If this is easy for you to do, please consider removing stuff specific to github like .gitignore from the release tarball. People like me downloading the tarball instead of cloning the repository might have a different workflow with their own .gitignore settings. If removing files from the tarball is convoluted with github, forget this suggestion. This is only a matter of convenience.

appveyor coverage isn't working

should probably figure out why and fix it at some point

Unexpected RuntimeError: partially-exhausted async_generator garbage collected

async_generator is really useful! But I see one surprising behavior. When I run this code:

import trio
from async_generator import async_generator, yield_


@async_generator
async def items():
    for i in range(10):
        await yield_(i)


async def main():
    async for i in items():
        if i > 4:
            break
        print(i)


if __name__ == '__main__':
    trio.run(main)

I get the following exception:

Exception ignored in: <bound method AsyncGenerator.__del__ of <async_generator._impl.AsyncGenerator object at 0x110a9b908>>
Traceback (most recent call last):
  File ".../python3.5/site-packages/async_generator/_impl.py", line 324, in __del__
    .format(self._coroutine.cr_frame.f_code.co_name)
RuntimeError: partially-exhausted async_generator 'items' garbage collected

Is this expected? The same code using the Python 3.6 syntax does not print any exception. Thanks!

New Release

Is it possible to create a new release/tag? The latest one if from January and there have been non-insignificant fixes since.

Thank you.

Abstract async context manager in Python3.6

I have an abstract class that defines an abstract async context manager (Python 3.7 code):

import abc
import contextlib


class Foo(metaclass=abc.ABCMeta):

    @abc.abstractmethod
    @contextlib.asynccontextmanager
    async def bar(self):
        pass

In order to make it compatible with Python 3.6, I would use async_generator.asynccontextmanager

import async_generator


class Foo(metaclass=abc.ABCMeta):

    @abc.abstractmethod
    @async_generator.asynccontextmanager
    async def bar(self):
        pass

However, this raises an error:

TypeError: must be an async generator (native or from async_generator; if using @async_generator then @acontextmanager must be on top.

This can be fixed using async_generator.async_generator, but I think that was intended for Python3.5 (which lacked native async generators) and not for Python3.6.

class Foo(metaclass=abc.ABCMeta):

    @abc.abstractmethod
    @async_generator.asynccontextmanager
    @async_generator.async_generator
    async def bar(self):
        pass

This seems to work on Python3.5 too.

I can extend the class and implement the method without using async_generator.async_generator in both, Python3.6 and Python3.7 (as expected):

class Bar(Foo):

    @async_generator.asynccontextmanager
    async def bar(self):
        print('Before')
        yield 42
        print('After')


bar = Bar()
async with bar.bar() as context:
    print('Context:', context)

Maybe the docs / readme could include an abstract method example.
Or maybe Python3.6 should work without adding @async_generator.

asynccontextmanager decorated functions return False but when using contextlib they return None

I don't know if this is intentional, but contextlib's asynccontextmanager decorated functions return None whereas here they'd return False. The snippet below shows that:

import asyncio
import contextlib
from async_generator import asynccontextmanager


@contextlib.asynccontextmanager
async def f():
    yield


@asynccontextmanager
async def g():
    yield


async def main():
    ctx1 = f()
    ctx2 = g()
    await asyncio.gather(*(ctx.__aenter__() for ctx in [ctx1, ctx2]))
    exits = await asyncio.gather(*(ctx.__aexit__(None, None, None) for ctx in [ctx1, ctx2]))
    print(exits)


l = asyncio.get_event_loop()
l.run_until_complete(main())

1.10: pytest is failing

I'm trying to package your module as an rpm package. So I'm using the typical build, install and test cycle used on building packages from non-root account.

• "setup.py build"
• "setup.py install --root </install/prefix>"
• "pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

May I ask for help because few units are failing:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=2477080430
rootdir: /home/tkloczko/rpmbuild/BUILD/async_generator-1.10
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, trio-0.7.0, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, randomly-3.8.0, Faker-8.12.1, nose2pytest-1.0.8, pyfakefs-4.5.1, tornado-0.8.1, twisted-1.13.3
collected 0 items / 1 error

================================================================================== ERRORS ==================================================================================
______________________________________________________________________ ERROR collecting test session _______________________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/config/__init__.py:570: in _importconftest
    mod = import_path(conftestpath, mode=importmode)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:544: in import_path
    raise ImportPathMismatchError(module_name, module_file, path)
E   _pytest.pathlib.ImportPathMismatchError: ('async_generator._tests.conftest', '/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib/python3.8/site-packages/async_generator/_tests/conftest.py', PosixPath('/home/tkloczko/rpmbuild/BUILD/async_generator-1.10/async_generator/_tests/conftest.py'))
========================================================================= short test summary info ==========================================================================
ERROR  - _pytest.pathlib.ImportPathMismatchError: ('async_generator._tests.conftest', '/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/l...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.40s =============================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.

and the same but with --import-mode=importlib

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --import-mode=importlib
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=1808263463
rootdir: /home/tkloczko/rpmbuild/BUILD/async_generator-1.10
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, trio-0.7.0, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, randomly-3.8.0, Faker-8.12.1, nose2pytest-1.0.8, pyfakefs-4.5.1, tornado-0.8.1, twisted-1.13.3
collected 0 items / 2 errors

================================================================================== ERRORS ==================================================================================
_____________________________________________________ ERROR collecting async_generator/_tests/test_async_generator.py ______________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/async_generator-1.10/async_generator/_tests/test_async_generator.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
async_generator/_tests/test_async_generator.py:9: in <module>
    from .conftest import mock_sleep
E   ImportError: attempted relative import with no known parent package
___________________________________________________________ ERROR collecting async_generator/_tests/test_util.py ___________________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/async_generator-1.10/async_generator/_tests/test_util.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
async_generator/_tests/test_util.py:3: in <module>
    from .. import aclosing, async_generator, yield_, asynccontextmanager
E   ImportError: attempted relative import with no known parent package
========================================================================= short test summary info ==========================================================================
ERROR async_generator/_tests/test_async_generator.py
ERROR async_generator/_tests/test_util.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================ 2 errors in 0.40s =============================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.