tudborg / aiochannel Goto Github PK

A suggestion to add high-level API methods methods that, instead of throwing ChannelClosed, return a boolean on whether operation was successfull vs channel is closed/closing.

Currently, whenever you use put / put_nowait, you always have to wrap it in a try block to catch and handle ChannelClosed exception. It would be a lot more convenient if that detail could be hidden inside a higher-level API and we just got a boolean instead.

The API could look something like this:

async def try_put(self, item) -> bool:
    """Try to put item into the channel, returning boolean whether operation succeeded or if the channel is closed/closing."""
    try:
        await self.put(item)
    except ChannelClosed:
        return False
    return True

(and similar for try_put_nowait)

For example, currently if I have to do something only if operation succeeds, I need to write code like:

try:
    await channel.put(item)
    # do something on success
except ChannelClosed:
    pass

with new API:

if await channel.try_put(item):
    # do something on success

It also makes it especially convenient to use with iterable operators like filter (or its async counterparts).
E.g. looping through a list of channels, putting a message into each one, and filering out closed channels, can all be done in one line:

channels = list(filter(lambda channel: channel.try_put_nowait(item), channels))

Dear Henrik, i found the case, when aiochannel is getting stuck. Could you please have a look?

Long story short: if i remove context switch await asyncio.sleep(0) code works fine:

# used python 3.10.2
import asyncio
from aiochannel import Channel

async def dummy_worker(ch):
    async for _ in ch:
        pass

async def main():
    concurrency = 2
    ch = Channel()
    tasks = [asyncio.create_task(dummy_worker(ch)) for _ in range(concurrency)]

    # Imagine we get items list asynchroniosly from web or moreover,
    # via async generator from stream and we do not know how items are there.
    # if i remove this context switch - everything works fine
    await asyncio.sleep(0)
    items = [1]

    for item in items:
        await ch.put(item)

    ch.close()
    await asyncio.gather(*tasks)


asyncio.run(main())

Two things to reproduce the bug:

1. Make context switch after worker tasks are created before first put
2. Do not make context switch after last put and before calling channel.close() method.

When calling close() method, qsize is still not empty, so one worker is not being cancelled:

import asyncio
import logging

from aiochannel import Channel

async def dummy_worker(ch):
    async for _ in ch:
        pass

async def main():
    concurrency = 3
    ch = Channel()
    tasks = [asyncio.create_task(dummy_worker(ch)) for _ in range(concurrency)]

    # getters 0, qsize 0
    logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())

    await asyncio.sleep(0)

    # getters 3, qsize 0
    logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())
    items = [1]

    for item in items:
        await ch.put(item)

    # getters 2, qsize 1
    logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())

    ch.close()

    # getters 1, qsize 1
    logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())
    await asyncio.gather( *tasks)


logging.basicConfig(level=logging.INFO)
asyncio.run(main())

Would be very nice if the library provided generic type annotations! i.e. define as generic over item type, i.e.

class Channel(Generic[T]):
    ...
    async def put(self, item: T):
    ... etc

Hi, i'm to install aiochannel on my windows 10 64 bits

Unable to upgrade to the 3.10 python. Could you please skip loop parameter for python versions with deprecated loop parameter?

Hi Henrik,

We're running into an issue where InvalidStateError is raised during close().
Tracking it down we find that 49b04cd is needed. However, it was reverted at 61c8f25.

We're currently running an older version for a while now to work around this issue.

I believe the use-case is that a put/get operation is cancelled (e.g. using asyncio.wait(channel.get(some_value), timeout=10)).
In that case, the put()/get() function doesn't remove the future from the deque, but merely marks it cancelled.
_wakeup_next dodges this behavior by checking if the future is done, but close no longer does this.

Let me know if you need more info. I'll be unavailable for 2 weeks, but will respond when I get back.

Best regards,
Stijn

It seems weird that I cancel the getter and then check if it is cancelled right after:

https://github.com/tbug/aiochannel/blob/49b04cded88bb5f2c62a665f44e9e528b78efa27/aiochannel/channel.py#L132-L133

I should check if there is an edge-case or bug in there somewhere.

As Rob Pike once said, "If you don't have select, you don't have real concurrency."

Is it possible to implement select for aiochanels? There is an implementation of select for threading-based channels with API similar to Unix select, where you pass a list of consumers and producers and receive a channel and a value. It gets the job done, but feels a bit clunky maybe.

I'm willing to try to implement it, if you're OK with this approach.