tudborg / aiochannel Goto Github PK
View Code? Open in Web Editor NEWChannels (closable queues) for Python's asyncio
Channels (closable queues) for Python's asyncio
A suggestion to add high-level API methods methods that, instead of throwing ChannelClosed
, return a boolean on whether operation was successfull vs channel is closed/closing.
Currently, whenever you use put
/ put_nowait
, you always have to wrap it in a try
block to catch and handle ChannelClosed
exception. It would be a lot more convenient if that detail could be hidden inside a higher-level API and we just got a boolean instead.
The API could look something like this:
async def try_put(self, item) -> bool:
"""Try to put item into the channel, returning boolean whether operation succeeded or if the channel is closed/closing."""
try:
await self.put(item)
except ChannelClosed:
return False
return True
(and similar for try_put_nowait
)
For example, currently if I have to do something only if operation succeeds, I need to write code like:
try:
await channel.put(item)
# do something on success
except ChannelClosed:
pass
with new API:
if await channel.try_put(item):
# do something on success
It also makes it especially convenient to use with iterable operators like filter
(or its async counterparts).
E.g. looping through a list of channels, putting a message into each one, and filering out closed channels, can all be done in one line:
channels = list(filter(lambda channel: channel.try_put_nowait(item), channels))
Dear Henrik, i found the case, when aiochannel is getting stuck. Could you please have a look?
Long story short: if i remove context switch await asyncio.sleep(0)
code works fine:
# used python 3.10.2
import asyncio
from aiochannel import Channel
async def dummy_worker(ch):
async for _ in ch:
pass
async def main():
concurrency = 2
ch = Channel()
tasks = [asyncio.create_task(dummy_worker(ch)) for _ in range(concurrency)]
# Imagine we get items list asynchroniosly from web or moreover,
# via async generator from stream and we do not know how items are there.
# if i remove this context switch - everything works fine
await asyncio.sleep(0)
items = [1]
for item in items:
await ch.put(item)
ch.close()
await asyncio.gather(*tasks)
asyncio.run(main())
Two things to reproduce the bug:
channel.close()
method.When calling close()
method, qsize is still not empty, so one worker is not being cancelled:
import asyncio
import logging
from aiochannel import Channel
async def dummy_worker(ch):
async for _ in ch:
pass
async def main():
concurrency = 3
ch = Channel()
tasks = [asyncio.create_task(dummy_worker(ch)) for _ in range(concurrency)]
# getters 0, qsize 0
logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())
await asyncio.sleep(0)
# getters 3, qsize 0
logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())
items = [1]
for item in items:
await ch.put(item)
# getters 2, qsize 1
logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())
ch.close()
# getters 1, qsize 1
logging.info('getters %r, qsize %r', len(ch._getters), ch.qsize())
await asyncio.gather( *tasks)
logging.basicConfig(level=logging.INFO)
asyncio.run(main())
Would be very nice if the library provided generic type annotations! i.e. define as generic over item type, i.e.
class Channel(Generic[T]):
...
async def put(self, item: T):
... etc
Unable to upgrade to the 3.10 python. Could you please skip loop parameter for python versions with deprecated loop parameter?
Hi Henrik,
We're running into an issue where InvalidStateError is raised during close().
Tracking it down we find that 49b04cd is needed. However, it was reverted at 61c8f25.
We're currently running an older version for a while now to work around this issue.
I believe the use-case is that a put/get operation is cancelled (e.g. using asyncio.wait(channel.get(some_value), timeout=10)
).
In that case, the put()/get() function doesn't remove the future from the deque, but merely marks it cancelled.
_wakeup_next
dodges this behavior by checking if the future is done, but close
no longer does this.
Let me know if you need more info. I'll be unavailable for 2 weeks, but will respond when I get back.
Best regards,
Stijn
It seems weird that I cancel the getter and then check if it is cancelled right after:
I should check if there is an edge-case or bug in there somewhere.
As Rob Pike once said, "If you don't have select
, you don't have real concurrency."
Is it possible to implement select
for aiochanels? There is an implementation of select for threading-based channels with API similar to Unix select, where you pass a list of consumers and producers and receive a channel and a value. It gets the job done, but feels a bit clunky maybe.
I'm willing to try to implement it, if you're OK with this approach.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.