Comments (13)
Hey Greg,
i think it is too early to make mailing list for aiohttp, i dont want to create another dead mailing list.
regarding big responses. we are using aiohttp for aws s3 integration, it works quite well. here is example of how to download big file:
def coro():
resp = yield from aiohttp.request('get', 'http://...')
with open('f.txt', 'wb') as f:
while True:
try:
chunk = yield from resp.content.read()
f.write(chunk)
except aiohttp.EofStream:
pass
resp.close()
large requests are also supported, you can pass generator as data
parameter:
def coro():
resp = yield from aiohttp.request(
'post', 'http://...', data=send_data(fname))
yield from resp
def send_data(fname):
with open(fname, 'rb') as f:
while True:
chunk = f.read(1024)
if not chunk:
break
yield chunk
you can also chain get request with post request:
def coro():
get_resp = yield from aiohttp.request('get', 'http://...')
resp = yield from aiohttp.request(
'post', 'http://...', data=send_data(get_resp))
yield from resp
def send_data(resp):
while True:
try:
chunk = yield from resp.content.read()
yield chunk
except aiohttp.EofStream:
break
i think main problem right now is documentation
from aiohttp.
i created separate issue for file object problem #20
from aiohttp.
Oh cool, thanks for the info. I figured out the read side out but hadn't caught on to the send side; makes sense though.
from aiohttp.
I tried this technique and it was still gobbling up memory. I wrote a quick test script so maybe you can tell me where I've gone horribly wrong:
from asyncio import coroutine, get_event_loop
from aiohttp import EofStream, request
def send_data():
with open('big1Gfile', 'rb') as fp:
chunk = fp.read(65536)
while chunk:
yield chunk
chunk = fp.read(65536)
@coroutine
def func():
response = yield from request(
'PUT',
'https://host/path',
headers={'x-auth-token': 'blah'},
data=send_data(),
chunked=True)
try:
while True:
chunk = yield from response.content.read()
print(chunk)
except EofStream:
pass
response.close()
get_event_loop().run_until_complete(func())
from aiohttp.
I get similarly large memory usage when not using chunked transfer encoding as well.
When I GET a large file the memory usage is fine, but I wonder if that's just because I can write the response to disk much faster than the network can receive. In other words, if I was chaining the GET to a PUT to another, much slower, host if the memory usage would balloon?
from aiohttp.
Ah yeah, verified that a big GET with a slow reader of the response also uses a lot of memory. I've got to be doing something wrong but I'm not sure how to tell aiohttp how large its buffers may be:
from asyncio import coroutine, get_event_loop, sleep
from aiohttp import EofStream, request
@coroutine
def func():
response = yield from request(
'GET',
'https://host/big1Gfile',
headers={'x-auth-token': 'blah'})
try:
while True:
chunk = yield from response.content.read()
yield from sleep(1)
except EofStream:
pass
response.close()
get_event_loop().run_until_complete(func())
from aiohttp.
Greg,
You pointed to real problems. Client part has to implement flow control subsystem.
asyncio already has flow control system, but aiohttp integration will take some time.
I'll try to come up with something during this week.
from aiohttp.
Ah okay. No rush at all as I'm not trying to use this on a production environment or anything yet. I had read a bit on the flow control and it seems like something you'd want to take your time on to get just right. Thanks for all your work!
from aiohttp.
I've just commited write flow control, could you try latest master for 'put' request.
read flow control is a bit tricky, i need more time for it.
from aiohttp.
Yes, looks good on the write end of things. Uploaded a 1G file and my process never exceeded 21m resident memory now. It figures that read flow control is more difficult and yet less commonly an issue, hah.
I was hitting my request timeout as I had set it to 60 thinking that was the time of no activity before expiring, but I now see it's the overall time of the request so I set that back to None. I don't suppose there's a no-activity timeout? I wonder if that's something I should put into my send_data generator somehow?
from aiohttp.
i've just commited read flow control. read control flow is useful serverside, for example if you use third party http service.
timeout is a timeout for sending request and receiving all headers. then you can use asyncio.wait_for to read response body.
from aiohttp.
Seems to be working like a charm. Thank you much!
from aiohttp.
This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.
If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.
from aiohttp.
Related Issues (20)
- Analyze additional places where there is timer handle churn HOT 7
- Risk of websocket ping tasks being garbage collected before they finish HOT 1
- Remove wait_for_disconnection
- Remove `allowed_protocol_schema_set` backwards compat in `BaseConnector` HOT 1
- Multipart reader fails on Transfer-Encoding: chunked HOT 9
- pytest example in testing.rst throws TypeError: 'async_generator' object is not callable HOT 2
- Response object reference to session returns None HOT 2
- ClientResponse.json() doesn't set status code HOT 3
- [typing] mypy errors when trying to extract headers when type is `LooseHeaders` HOT 1
- Support free-threaded Python 3.13 build HOT 2
- TypeError: _patch_task.<locals>.task_new_init() got an unexpected keyword argument 'eager_start' HOT 5
- Sending default 'Server' header when failing on wrong HTTP method HOT 6
- Websocket ping test has transient failures on loaded systems HOT 1
- Send 100-continue only when handler reads body
- typing error with `timeout` argument of `aiohttp.ClientSession` requests methods HOT 1
- Rename compress parameter HOT 6
- on_request_headers_sent params type was never imported into the top-level file. HOT 2
- AsyncResolver not passing on loop too aiodns.DNSResolver HOT 4
- test_web_sendfile_functional isn't testing sendfile
- Incorrect re-use future in _make_or_get_ssl_context HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiohttp.