Comments (8)
Oh I see, I think you need the flatmap or flatten operator. Would this snippet help?
import random
import asyncio
from aiostream import pipe, stream
async def process_async(i):
await asyncio.sleep(random.random())
return [f"{i}a", f"{i}b"]
async def process_one_in_chunk(chunk):
await asyncio.sleep(random.random())
return chunk
async def main():
values = (i for i in range(10))
xs = (
stream.iterate(values)
| pipe.map(process_async)
| pipe.flatmap(stream.iterate)
| pipe.chunks(3)
| pipe.map(process_one_in_chunk, ordered=False)
)
async with xs.stream() as streamer:
async for item in streamer:
print(item)
if __name__ == "__main__":
asyncio.run(main())
It prints:
['1b', '2a', '2b']
['3a', '3b', '4a']
['6a', '6b', '7a']
['4b', '5a', '5b']
['7b', '8a', '8b']
['9a', '9b']
['0a', '0b', '1a']
from aiostream.
how am i supposed to use chain?
chain
is simply the concatenation of several asynchronous sequences.
So stream.chain(a, b, c)
(or equivalently a | pipe.chain(b, c)
) would produce all items from a
then all items from b
then all items from c
.
In the code you wrote, | pipe.chain()
is basically a no-op (as it will simply forwards the items produced by map
).
I would expect this to work, but i get [...]
Could you explain more precisely the result you expected from this code? In any case what you get is the expected behavior:
process_async
returns a list with a single value ([i]
)pipe.map(process_async,ordered=False)
runprocess_async
concurrently and produce the corresponding results as soon as they arrivepipe.list
aggregates the items produced bymap
into a list
So [[7], [9], [0], [2], [4], [6], [5], [8], [1], [3]]
is the expected result.
from aiostream.
that was quick, thanks
I would like to get [3,6,1,8,0,...]
basically i need to flatten the nested list (what itertools.chain
does)
from aiostream.
basically i need to flatten the nested list (what itertools.chain does)
By the way you would get the same result with itertools.chain
:
>>> from itertools import chain
>>> list(chain(map(lambda i: [i], [1, 2, 3])))
[[1], [2], [3]]
from aiostream.
sorry my bad, this is what i'm looking for
from itertools import chain
list(chain.from_iterable(map(lambda i: [i], [1, 2, 3])))
[1, 2, 3]
thanks for your help, i need to go through your example
from aiostream.
Alright, then pipe.concatmap(stream.iterate)
(rather than flatmap
) is probably the best equivalent to chain.from_iterable
.
from aiostream.
thanks a lot, i think that i'm starting to get it. i manage to build a working pipeline. and it ended up quite differently than what i originally thought.
I was thinking to manage the number of concurrent tasks with chunk, but i realized that map can do that on it's own. aiostream is amazing, i can rebuild a lot of thinks using it and the code will be so easy to understand.
thanks a lot for putting aiostream together.
PS: I would suggest you to enable discussions. I think it will be great if people could contribute their recipes there.
from aiostream.
I was thinking to manage the number of concurrent tasks with chunk, but i realized that map can do that on it's own. aiostream is amazing, i can rebuild a lot of thinks using it and the code will be so easy to understand.
I'm glad you found it useful :)
PS: I would suggest you to enable discussions. I think it will be great if people could contribute their recipes there.
Good idea, done! 🎉
from aiostream.
Related Issues (20)
- add support python3.10 HOT 1
- Please upload a wheel release to pypi HOT 2
- `stream.list` implementation is x10 slower than using plain Python built-in functionality (list comprehension) HOT 2
- Idea: Timeouts on chunks HOT 4
- action: support task_limit HOT 5
- Regarding `update_pipe_module` HOT 3
- Aiostream fails to import with a TypeError HOT 1
- CI test pipeline doesn't run in Python 3.12: ModuleNotFoundError: No module named 'setuptools' HOT 6
- License change HOT 3
- 0.5.0 (#84) made backwards incompatible changes HOT 6
- asyncio.Event for graceful/early termination HOT 7
- How to handle stream splitting HOT 2
- Incompatibility with mypy 1.7 and later
- Docstring and prototype not properly shown with pylance HOT 1
- New operators
- No easy to see installation instructions in the docs or README HOT 1
- merge causes anext(): asynchronous generator is already running Exception HOT 2
- Feature suggestion: `strict` parameter for `zip` HOT 3
- How to duplicate streams properly? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiostream.