spyoungtech / grequests

Requests + Gevent = <3

GRequests: Asynchronous Requests ===============================

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

Installation

Installation is easy with pip:

$ pip install grequests
✨🍰✨

Usage

Usage is simple:

import grequests

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://fakedomain/',
    'http://kennethreitz.com'
]

Create a set of unsent Requests:

>>> rs = (grequests.get(u) for u in urls)

Send them all at the same time using map:

>>> grequests.map(rs)
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, None, <Response [200]>]

The HTTP verb methods in grequests (e.g., grequests.get, grequests.post, etc.) accept all the same keyword arguments as in the requests library.

Error Handling

To handle timeouts or any other exception during the connection of the request, you can add an optional exception handler that will be called with the request and exception inside the main thread. The value returned by your exception handler will be used in the result list returned by map.

>>> def exception_handler(request, exception):
...    print("Request failed")

>>> reqs = [
...    grequests.get('http://httpbin.org/delay/1', timeout=0.001),
...    grequests.get('http://fakedomain/'),
...    grequests.get('http://httpbin.org/status/500')]
>>> grequests.map(reqs, exception_handler=exception_handler)
Request failed
Request failed
[None, None, <Response [500]>]

imap

For some speed/performance gains, you may also want to use imap instead of map. imap returns a generator of responses. Order of these responses does not map to the order of the requests you send out. The API for imap is equivalent to the API for map. You can also adjust the size argument to map or imap to increase the gevent pool size.

for resp in grequests.imap(reqs, size=10):
    print(resp)

There is also an enumerated version of imap, imap_enumerated which yields the index of the request from the original request list and its associated response. However, unlike imap, failed requests and exception handler results that return None will also be yielded (whereas in imap they are ignored). Aditionally, the requests parameter for imap_enumerated must be a sequence. Like in imap, the order in which requests are sent and received should still be considered arbitrary.

>>> rs = [grequests.get(f'https://httpbin.org/status/{code}') for code in range(200, 206)]
>>> for index, response in grequests.imap_enumerated(rs, size=5):
...     print(index, response)
1 <Response [201]>
0 <Response [200]>
4 <Response [204]>
2 <Response [202]>
5 <Response [205]>
3 <Response [203]>

gevent - when things go wrong

Because grequests leverages gevent (which in turn uses monkeypatching for enabling concurrency), you will often need to make sure grequests is imported before other libraries, especially requests, to avoid problems. See grequests gevent issues for additional information.

# GOOD
import grequests
import requests

# BAD
import requests
import grequests

grequests's Issues

Disable error printing on in terminal

Hi, Is there anyway to turn the logging off? Errors like:

Timeout: (<requests.packages.urllib3.connectionpool.HTTPConnectionPool object at 0x1112bafd0>, 'Connection to 163.125.156.220 timed out. (connect timeout=10)')
<Greenlet at 0x11117d9b0: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x1112e7210>>(stream=False)> failed with Timeout

are written in terminal. I found http://stackoverflow.com/questions/11029717/how-do-i-disable-log-messages-from-the-requests-library to silent the log messeges of requests, but it doesn't work for grequests. Any ideas? thanx.

Fails trying to close session on requests 1.0.3

As can be seen if you try and run the example code from the README:

Python 2.6.5 (r265:79063, Oct  1 2012, 22:04:36)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import requests

In [2]: requests.__version__
Out[2]: '1.0.3'

In [3]: import grequests

In [4]: urls = [
   ...:     'http://www.heroku.com',
   ...:     'http://tablib.org',
   ...:     'http://httpbin.org',
   ...:     'http://python-requests.org',
   ...:     'http://kennethreitz.com'
   ...: ]

In [5]: rs = (grequests.get(u) for u in urls)

In [6]: grequests.map(rs)
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))

ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (19, 0))

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/root/requests_lib_upgrade/<ipython console> in <module>()

/usr/local/lib/python2.6/dist-packages/grequests.pyc in map(requests, prefetch, size)
     77     """
     78 
---> 79     requests = list(requests)
     80 
     81     pool = Pool(size) if size else None

/root/requests_lib_upgrade/<ipython console> in <genexpr>((u,))

/usr/local/lib/python2.6/dist-packages/grequests.pyc in wrapped(*args, **kwargs)
     42         kwargs['config'] = config
     43
---> 44         return f(*args, **kwargs)
     45
     46     return wrapped

/usr/local/lib/python2.6/dist-packages/requests/api.pyc in get(url, **kwargs)
     47
     48     kwargs.setdefault('allow_redirects', True)
---> 49     return request('get', url, **kwargs)
     50
     51

/usr/local/lib/python2.6/dist-packages/requests/api.pyc in request(method, url, **kwargs)
     36
     37     session = sessions.Session()
---> 38     return session.request(method=method, url=url, **kwargs)
     39
     40

TypeError: request() got an unexpected keyword argument 'return_response'

Memory consumption issues

Fully described here: http://stackoverflow.com/questions/11815640/memory-leak-with-grequests

Just importing `grequests` breaks `requests.Session.post()`

Just importing grequests breaks requests.Session.post() for me.

 File "venv/lib/python2.7/site-packages/requests/sessions.py", line 508, in post
  return self.request('POST', url, data=data, json=json, **kwargs)
File "venv/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
  resp = self.send(prep, **send_kwargs)
File "venv/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
  r = adapter.send(request, **kwargs)
File "venv/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
  timeout=timeout
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
  body=body, headers=headers)
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
  self._validate_conn(conn)
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
  conn.connect()
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 238, in connect
  ssl_version=resolved_ssl_version)
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 279, in ssl_wrap_socket
  return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 350, in wrap_socket
  _context=self)
TypeError: __init__() got an unexpected keyword argument 'server_hostname'

setup.py should specify install_requires

Mostly for convenience.

It looks like just gevent and request.

suggestion: use concurrent.futures instead of gevent

Hi Kenneth,

May I suggest that you consider using this instead of gevent:

http://docs.python.org/dev/library/concurrent.futures.html

It has a backport to previous versions

http://pypi.python.org/pypi/futures

It has a good interface IMO, it's in the standard library so will be supported indefinitely, it doesn't have external native dependencies like gevent does, and it can help avoid the problems stemming from gevent monkey patching as in issue 1. To be fair, gevent has awesome performance but it's just not as easy a dependency to work with as I'd like (I had to update to 1.0b2 to get pool.imap to work, but that version is not in pypi, etc.)

gevent block error when using requests library

Is it unwise to use both the grequests library and the requests library at the same time?

I ask as I've just seen an issue in my production environment. I have grequests installed for use in one section of my site, and use requests on a number of other pages. One of the simple requests.get() call has failed because it went down along the path of gevent and failed for some reason.

Interestingly, refreshing the page didn't produce this error and succeeded as it usually does.

[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
[Wed Sep 03 00:32:58 2014] [error]     return request('get', url, **kwargs)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
[Wed Sep 03 00:32:58 2014] [error]     return session.request(method=method, url=url, **kwargs)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
[Wed Sep 03 00:32:58 2014] [error]     resp = self.send(prep, **send_kwargs)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
[Wed Sep 03 00:32:58 2014] [error]     r = adapter.send(request, **kwargs)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
[Wed Sep 03 00:32:58 2014] [error]     timeout=timeout
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
[Wed Sep 03 00:32:58 2014] [error]     body=body, headers=headers)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
[Wed Sep 03 00:32:58 2014] [error]     conn.request(method, url, **httplib_request_kw)
[Wed Sep 03 00:32:58 2014] [error]   File "/usr/lib/python2.7/httplib.py", line 958, in request
[Wed Sep 03 00:32:58 2014] [error]     self._send_request(method, url, body, headers)
[Wed Sep 03 00:32:58 2014] [error]   File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
[Wed Sep 03 00:32:58 2014] [error]     self.endheaders(body)
[Wed Sep 03 00:32:58 2014] [error]   File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
[Wed Sep 03 00:32:58 2014] [error]     self._send_output(message_body)
[Wed Sep 03 00:32:58 2014] [error]   File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
[Wed Sep 03 00:32:58 2014] [error]     self.send(msg)
[Wed Sep 03 00:32:58 2014] [error]   File "/usr/lib/python2.7/httplib.py", line 776, in send
[Wed Sep 03 00:32:58 2014] [error]     self.connect()
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 106, in connect
[Wed Sep 03 00:32:58 2014] [error]     conn = self._new_conn()
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 90, in _new_conn
[Wed Sep 03 00:32:58 2014] [error]     (self.host, self.port), self.timeout, *extra_args)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/socket.py", line 570, in create_connection
[Wed Sep 03 00:32:58 2014] [error]     for res in getaddrinfo(host, port, 0 if has_ipv6 else AF_INET, SOCK_STREAM):
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/socket.py", line 621, in getaddrinfo
[Wed Sep 03 00:32:58 2014] [error]     return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
[Wed Sep 03 00:32:58 2014] [error]     return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/threadpool.py", line 222, in apply_e
[Wed Sep 03 00:32:58 2014] [error]     success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/event.py", line 233, in get
[Wed Sep 03 00:32:58 2014] [error]     result = self.hub.switch()
[Wed Sep 03 00:32:58 2014] [error]   File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
[Wed Sep 03 00:32:58 2014] [error]     return greenlet.switch(self)
[Wed Sep 03 00:32:58 2014] [error] LoopExit: This operation would block forever

object has no attribute 'exception'

When i tried to use exception handler i got this exception:
Process Process-1:
Traceback (most recent call last):
File "C:\Develop\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Develop\Python27\lib\multiprocessing\process.py", line 114, in run
self._target(_self._args, *_self._kwargs)
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\nmap\nmap.py", line 149, in scan_progressive
callback(host, scan_data)
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\autoBrowsers.py", line 55, in callback_result
get_async_results('http', host, ports)
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\autoBrowsers.py", line 42, in get_async_results
exception_handler=exception_handler,
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\grequests.py", line 124, in map
exception_handler(request, request.exception)
AttributeError: 'AsyncRequest' object has no attribute 'exception'

Response and request both contains different urls than response due to redirection.

The returned response objects contain the redirected url, and I am unable to find the one that I requested for.
How do I find which returned object corresponds to which requested url?

In [31]: urls = [                             
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

In [32]: rs = (grequests.get(u) for u in urls)

In [33]: for i in grequests.imap(rs):         
    print i, i.url
   ....:     
<Response [200]> https://www.heroku.com/
<Response [200]> http://docs.python-tablib.org/en/latest/
<Response [200]> http://httpbin.org/
<Response [200]> http://docs.python-requests.org/en/latest/
<Response [200]> https://medium.com/@kennethreitz/collections/

neither response.url, nor response.request.url give the requested url.

In [35]: rs = (grequests.get(u) for u in urls)

In [36]: for i in grequests.imap(rs):
    print i, i.request.url
   ....:     
<Response [200]> https://www.heroku.com/
<Response [200]> http://docs.python-tablib.org/en/latest/
<Response [200]> http://httpbin.org/
<Response [200]> http://docs.python-requests.org/en/latest/
<Response [200]> https://medium.com/@kennethreitz/collections/

safe mode doesn't work for async requests

Safe mode doesn't work when making async requests. The reason is catch_exceptions_if_in_safe_mode() decorates request() which calls Session.request() which does not call Request.send() when send_request is False which is the case with async requests.

I guess safe mode should guard two phases; one is the creation of a request, and the second one is sending a request.

see kennethreitz/requests#583

Limit maximum retrieving time

Is there any way to limit maximum time taken to retrieve HTTP response when doing async requests through grequests.map? (no socket connection timeout itself). Monkey patching seems to break socket.setdefaulttimeout() and have played around with gevent.Timeout and similar but without really good results...

Thanks in advance

User data on request and response

I would like to be able to add custom data to a request object and get it back from response object. Primarly as a way to distinguish responses from async requests. Something like:

requests = [grequest.get(url, user_info={'some_id': x}) for url, x in ...]
for response in grequests.map(requests):
    if (response.user_info['some_id'] == ...) ... 

nondeterministic failures on Windows

As reported by @piotr-dobrogost at kennethreitz/requests#605 :

from requests import async

if __name__ == '__main__':

    req = async.get('http://httpbin.org/get')
    res = async.map([req])[0]
    print res.content

produces (about 90% of the time):

Traceback (most recent call last):
  File "C:\python\virtualenv\requests\lib\site-packages\gevent-0.13.7-py2.7-win-amd64.egg\gevent\greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "C:\Users\Piotr\projects\requests\requests\models.py", line 606, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /get
<Greenlet at 0x35e59c8L: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError

None

in the following environment:

Requests 0.12.1
gevent 0.13.7-py2.7-win-amd64
greenlet 0.3.4-py2.7-win-amd64
Python 2.7.3 64bit
Windows Vista 64bit

Mountain lion, unable to install

Unfortunately I cannot make grequests get installed in my 10.8 mac.

I have described my problem in stack overflow:
http://stackoverflow.com/questions/17865198/install-grequests-mac-os-10-8-gcc-error

Any possible workarounds?

"Too many open files" error, regardless of size parameter

I am using grequests to test the performance of my service. Both the service and the machine from which I start the requests are inside Amazon EC2. It seems that I cannot start more requests than ulimit -n, even though I explicitly specify the size parameter for map() / imap()

My setup is roughly like this:

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
params = [json.dumps({'url': urls[i]}) for i in urls]
reqs = [grequests.post(SERVICE_URL, data=params[i % len(params)], headers=headers)
            for i in xrange(no_requests)]
res = grequests.imap(reqs, size=200)
for r in res:
    ...
    r.close()

Doesn't the size parameter here tell grequests that it should issue no_requests, 200 at a time?

RuntimeError importing Gevent

Grequest raise this exception "Gevent is required for grequests." when i use it with virtualenv(python 2.7.10 on linux ubuntu 14.04 LTS).

grequests.imap with empty iterator hangs

The following usage hangs grequests:

>>> for response in grequests.imap(iter([])):
...     print response
...

However, the same occurs with gevent:

for x in Pool().imap_unordered(lambda x: x, iter([])):
...     print x
... 

So maybe this should be reported upstream?

Error from Gevent: cannot switch to another thread

Hi. I am using grequests to send a bunch of requests. After about 2000 requests, error below is reported.
error: cannot switch to a different thread (libev) error creating signal/async pipe: Too many open files
Any idea what happened? I use imap with default pool size.

HEAD follows redirects

Per the plain-old requests documentation, redirects are followed for all requests except HEAD: http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history

However, grequests follows redirects on HEAD unless explicitly told not to:

>>> bad=grequests.map([grequests.head('http://github.com')])[0]
>>> bad.status_code
200
>>> bad.history
(<Response [301]>,)
>>> bad.url
u'https://github.com/'


>>> good=grequests.map([grequests.head('http://github.com', allow
_redirects=False)])[0]
>>> good.status_code
301
>>> good.history
[]
>>> good.url
u'http://github.com/'

gevent's monkeypatching breaks timeouts on linux

See kennethreitz/requests#500 for context, but basically the timeout is no longer respected after importing the async module that monkeypatches socket.

>>> import requests
>>> import time
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 604, in send
    raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 604, in send
    raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> import requests.async
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
4.38320612907 200
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
4.67993688583 200

Gives ConnectionError

Making requests to https gives me [ grequests.map(rs) gives me ] :

ConnectionError: HTTPSConnectionPool(host='www.foobar.com', port=443): Max retries exceeded with url: /index/xx/82
<Greenlet at 0x3d8a2958: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError

All use the same session

Maybe I'm overlooking a easy option here (just started using python again), but how would I fire off 100 requests with them all using the same session? Is it possible with the current API?

Ideally I'd like to be able to do something like

s = requests.session()
grequests.map(rs, session=s)

Anyone needed this before? How did you approach it?

map hangs after RST packets

I'm using the map function to "unshorten" about 5.5 million t.co links. I give 2000 requests a time with a limit of 30. Afterwards I do the same for the unshortened links. Now once in a while t.co will send me back a whole lot of RST packets, each one for a different connection which had already been closed (FIN ACK and ACK) a few seconds earlier and usually my script is then requesting the unshortened links with the map function.

Usually this is not a problem, but sometimes the map function just stops doing anything afterwards and hangs the script.

First request to grequests.map takes 5 extra seconds

The first call to grequests.map takes five seconds longer than subsequent requests on my machine [1].

Example output from test case:

2012-10-15 16:32:03,769 starting...
2012-10-15 16:32:03,776 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:03,776 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:03,777 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,829 "GET / HTTP/1.1" 200 24765
2012-10-15 16:32:08,841 "GET / HTTP/1.1" 200 24834
2012-10-15 16:32:08,850 "GET / HTTP/1.1" 200 25071
2012-10-15 16:32:08,885 took 5.116s
2012-10-15 16:32:08,889 starting...
2012-10-15 16:32:08,891 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,892 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,894 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,968 "GET / HTTP/1.1" 200 25024
2012-10-15 16:32:08,973 "GET / HTTP/1.1" 200 25071
2012-10-15 16:32:08,977 "GET / HTTP/1.1" 200 24731
2012-10-15 16:32:09,005 took 0.116s
2012-10-15 16:32:09,009 starting...
2012-10-15 16:32:09,011 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:09,012 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:09,014 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:09,085 "GET / HTTP/1.1" 200 24988
2012-10-15 16:32:09,089 "GET / HTTP/1.1" 200 24786
2012-10-15 16:32:09,090 "GET / HTTP/1.1" 200 24553
2012-10-15 16:32:09,121 took 0.112s

Test case:

import logging
import time

import grequests

logger = logging.getLogger(__name__)
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.DEBUG)

urls = ['http://www.bbc.co.uk/'] * 3

for i in xrange(3):                                                                                  
    requests = [grequests.get(url) for url in urls]
    logger.info('starting...')
    start = time.time()
    responses = grequests.map(requests)
    logger.info('took %.3fs' % (time.time() - start,))

[1] grequests version 0.1.0 installed via PyPi.

$ uname -a
Linux robson 2.6.35-32-generic-pae #67-Ubuntu SMP Mon Mar 5 21:23:19 UTC 2012 i686 GNU/Linux
$ python --version
Python 2.6.6

Use request in separate thread after using grequest

I am attempting to write a script that performs a bunch of requests asynchronously using grequest, and then spawns some threads that each use the standard python-requests module to perform some additional http requests. It seems that the first call to requests.get() after i've used grequest results in the following error:

NotImplementedError: gevent is only usable from a single thread

Is there something that needs to happen to use the python-requests module from a thread after using grequests?

Gevent failed during installation on MAC OS

Can't find anything that solves my problem, I don't know what else to do.

Installing collected packages: gevent, grequests
  Running setup.py install for gevent
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.10-intel-2.7
    creating build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/__init__.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/_ssl2.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/_sslgte279.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/_threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/backdoor.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/baseserver.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/coros.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/event.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/fileobject.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/greenlet.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/hub.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/local.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/lock.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/monkey.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/os.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/pool.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/pywsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/queue.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/resolver_ares.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/resolver_thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/select.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/server.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/socket.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/ssl.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/subprocess.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/threadpool.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/timeout.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/util.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/win32util.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/wsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
    running build_ext
    Running '/bin/sh /private/tmp/pip-build-09c1Gh/gevent/libev/configure > configure-output.txt' in /private/tmp/pip-build-09c1Gh/gevent/build/temp.macosx-10.10-intel-2.7/libev
    building 'gevent.core' extension
    creating build/temp.macosx-10.10-intel-2.7/gevent
    cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -U__llvm__ -arch x86_64 -arch i386 -pipe -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -Ibuild/temp.macosx-10.10-intel-2.7/libev -Ilibev -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c gevent/gevent.core.c -o build/temp.macosx-10.10-intel-2.7/gevent/gevent.core.o
    In file included from gevent/gevent.core.c:249:
    In file included from gevent/libev.h:2:
    libev/ev.c:483:48: warning: '/*' within block comment [-Wcomment]
    /*#define MIN_INTERVAL  0.00000095367431640625 /* 1/2**20, good till 2200 */
                                                   ^
    libev/ev.c:1029:42: error: '_Noreturn' keyword must precede function declarator
      ecb_inline void ecb_unreachable (void) ecb_noreturn;
                                             ^~~~~~~~~~~~
      _Noreturn
    libev/ev.c:832:26: note: expanded from macro 'ecb_noreturn'
      #define ecb_noreturn   _Noreturn
                             ^
    libev/ev.c:1625:31: warning: 'extern' variable has an initializer [-Wextern-initializer]
      EV_API_DECL struct ev_loop *ev_default_loop_ptr = 0; /* needs to be initialised to make it a definition despite extern */
                                  ^
    libev/ev.c:1796:7: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ANPENDING, pendings [pri], pendingmax [pri], w_->pending, EMPTY2);
          ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:1807:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (W, rfeeds, rfeedmax, rfeedcnt + 1, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:1934:7: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (int, fdchanges, fdchangemax, fdchangecnt, EMPTY2);
          ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    In file included from gevent/gevent.core.c:249:
    In file included from gevent/libev.h:2:
    In file included from libev/ev.c:2484:
    libev/ev_kqueue.c:50:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (struct kevent, kqueue_changes, kqueue_changemax, kqueue_changecnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    In file included from gevent/gevent.core.c:249:
    In file included from gevent/libev.h:2:
    In file included from libev/ev.c:2490:
    libev/ev_poll.c:66:7: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (struct pollfd, polls, pollmax, pollcnt, EMPTY2);
          ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:3648:34: warning: '&' within '|' [-Wbitwise-op-parentheses]
      fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
                           ~~~~~~~~~~^~~~~~~~~~~~~ ~
    libev/ev.c:3648:34: note: place parentheses around the '&' expression to silence this warning
      fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
                                     ^
                           (                      )
    libev/ev.c:3687:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ANHE, timers, timermax, ev_active (w) + 1, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4367:5: warning: unused variable 'ocur_' [-Wunused-variable]
        array_needsize (ev_idle *, idles [ABSPRI (w)], idlemax [ABSPRI (w)], active, EMPTY2);
        ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4407:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_prepare *, prepares, preparemax, preparecnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4445:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_check *, checks, checkmax, checkcnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4592:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_fork *, forks, forkmax, forkcnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4675:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_async *, asyncs, asyncmax, asynccnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    14 warnings and 1 error generated.
    error: command 'cc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-09c1Gh/gevent```

Passing user info through request

Before last commit, there was a way to put some bits of data inside kwargs['config'], that helped when retrieving request details from map() results. What is the preferred way to do that now?

ImportError: No module named queue/Requests stop returning data

I wrote a test in which I use grequests to do a GET on my server 1000 times (throttled back to four simultaneous requests). The first 250 requests work fine, and then they stop returning data (and run very fast). Just before the run ends (after 995 of 1000 GETs), grequests raises this exception:

Traceback (most recent call last):
  File "submit_test.py", line 32, in 
  File "submit_test.py", line 30, in main
  File "/Users/davidp/.virtualenvs/ssk_test/lib/python2.7/site-packages/grequests-0.1.0-py2.7.egg/grequests.py", line 83, in map
  File "/Users/davidp/.virtualenvs/ssk_test/lib/python2.7/site-packages/gevent-0.13.7-py2.7-macosx-10.6-intel.egg/gevent/greenlet.py", line 487, in joinall
ImportError: No module named queue

I saw this same behavior with requests.async. It's not just my server because I get the same response when I use other sites.

My test was run with Python 2.7.3 on Mac OS X 10.7.4. grequests 0.1.0. gevent 0.13.7.

Here is my test code:

import time
import grequests

url = 'http://ssk-dev.appspot.com/test_service/default/search_unique'

def process_page (content, account_id):
    global start
    if content:
        length = len(content)
    else:
        length = 0
    print "In parse_page(): %d %d %f" % (account_id, length, time.time() - start)


def main ():
    global start
    start = time.time()
    
    # Make a list of callback functions 
    callback_list = []
    for i in range (1000):
        def wrapper (id = i):  # Fashion a closure because I need a bit of state
            def callback (response):
                process_page (response.content, id)
            return callback
        func = wrapper (i)
        callback_list.append (func)

    async_request_list = [grequests.get (url, hooks = {'response': func})
                                for func in callback_list]

    grequests.map (async_request_list, size = 4)

main()

Grequests timeout is not honored

When making a request with Grequests, a timeout error is not raised. Here is a failing test:

def test_concurrency_with_timeout(self):
    n = 10
    reqs = []
    reqs.append(grequests.get(httpbin('delay/10'), timeout=1.5))
    resp = grequests.map(reqs)
    self.assertEqual([None], resp)

Cut a new 0.3.0 release :)

test fix for requests hook kwargs [1] is handy for packagers / porters (QA) , along with the test_suite and tests_require changes [2]

[1] f50782a
[2] #47

i need "exception_handler" get back to the main thread

eg.

def map(requests, stream=False, size=None, exception_handler=None):
    ……
    ret = []

    for request in requests:
        if request.response:
            ret.append(request.response)
        elif exception_handler:
            #exception_handlerrequest, request.exception)
            # new code:
            ret.append(exception_handler(request))

    return ret

And then I can use it like this:

from requests import Response
def timeOutException(request):
    if 'ok' in dir(request.response):
        return request.response
    else:
        # build respone for timeout
        response = Response()
        response.status_code = 444
        response.url = request.url
        return response

In [119]: res = grequests.map(apprs, exception_handler=timeOutException)                

In [120]: res
Out[120]: [<Response [200]>, <Response [444]>, <Response [404]>, <Response [200]>]

The same process can also be in imap.

TypeError: request() got an unexpected keyword argument 'return_response'

import grequests as gr
gr.get('http://baidu.com')
Traceback (most recent call last):
File "", line 1, in
File "grequests.py", line 44, in wrapped
return f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests-1.0.4-py2.7.egg/requests/api.py", line 55, in get
return request('get', url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests-1.0.4-py2.7.egg/requests/api.py", line 44, in request
return session.request(method=method, url=url, *_kwargs)
TypeError: request() got an unexpected keyword argument 'return_response'

Requests not being made in same order as list

Although a list of Requests is passed to the map function, the actual HTTP requests are made in arbitrary order.

This is not unexpected, of course, but it would still be nice to have the list's order followed, somehow.

Running gevent.sleep(0) after each spawn (i.e. in the send function) seems to maintain order with no apparent side effects; perhaps you might like to consider adding it?

Documentation missing?

https://crate.io/packages/grequests/ isn't working

map() got an unexpected keyword argument 'exception_handler'

i get this error TypeError:map() got an unexpected keyword argument 'exception_handler'

i am getting it on this line

manresponse =grequests.map(rs,exception_handler = eception_handler)
any help ??

Too many urls in greqests.map causes gevent.hub.LoopExit only on mac

Detailed description is in this SO question: https://stackoverflow.com/questions/23117176/too-many-urls-in-greqests-map-causes-gevent-hub-loopexit-only-on-mac , but I will answer any questions in comments to this issue if needed.

Calling map() with a bunch of requests (over 500) causes timeout error in individual requests, not sure how to catch these timeout exceptions

When I attempt to call map with 500 or more requests, usually a few of my requests timeout (I set it to 7 seconds).

This is ok but the error messages display and i'm not sure how to catch these errors because all the work is done individually in the map() method from grequests.

Can I either catch these timeouts individually somehow, or somehow log all the failed requests and re-attempt them? What is usually the protocol when sending out a bunch of requests asynchronously with the requirement that they all return back.

Associate arbitrary object to each request?

I need to capture the request and an instance of an object that contains meta-data about the request to store in the database, how do you recommend that? You only return the response text.

Importing grequests blocks socket reads forever

I'm importing grequests (which then monkey patches networking libraries) in a multi-threaded Django app run under uWSGI and I get errors like this one:

Traceback (most recent call last):
  File "/.../python2.7/site-packages/redis/connection.py", line 182, in read_response
    buffer = self._sock.recv(4096)
  File "/.../python2.7/site-packages/gevent/socket.py", line 392, in recv
    self._wait(self._read_event)
  File "/.../python2.7/site-packages/gevent/socket.py", line 298, in _wait
    self.hub.wait(watcher)
  File "/.../python2.7/site-packages/gevent/hub.py", line 341, in wait
    result = waiter.get()
  File "/.../python2.7/site-packages/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/.../python2.7/site-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
LoopExit: This operation would block forever

Can this be caused by reading from sockets in multiple threads spawned by uWSGI or is there another explanation for this behaviour?

issue on installation with python 3.4.3 - TypeError:

Try to do a pip3 install grequests with python 3.4.3, and am getting the following Traceback

Installing collected packages: gevent, grequests
  Running setup.py install for gevent
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 316, in <module>
        run_setup(ext_modules)
      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 311, in run_setup
        "Development Status :: 4 - Beta"])
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.4/site-packages/setuptools-12.2-py3.4.egg/setuptools/command/install.py", line 61, in run
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/install.py", line 539, in run
        self.run_command('build')
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build.py", line 126, in run
        self.run_command(cmd_name)
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
        self.build_extension(ext)
      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 234, in build_extension
        self.gevent_prepare(ext)
      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 231, in gevent_prepare
        configure(self, ext)
      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 139, in configure_libev
        make_universal_header('config.h', 'SIZEOF_LONG', 'SIZEOF_SIZE_T', 'SIZEOF_TIME_T')
      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 111, in make_universal_header
        print >>f, line
    TypeError: unsupported operand type(s) for >>: 'builtin_function_or_method' and '_io.TextIOWrapper'
    Running '/bin/sh /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/libev/configure > configure-output.txt' in /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/build/temp.macosx-10.10-x86_64-3.4/libev
    Complete output from command /usr/local/opt/python3/bin/python3.4 -c "import setuptools, tokenize;__file__='/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-2a9m9jp5-record/install-record.txt --single-version-externally-managed --compile:
    running install

    running build

    running build_py

    creating build

    creating build/lib.macosx-10.10-x86_64-3.4

    creating build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/__init__.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/_threading.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/backdoor.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/baseserver.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/coros.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/event.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/fileobject.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/greenlet.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/hub.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/local.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/lock.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/monkey.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/os.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/pool.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/pywsgi.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/queue.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/resolver_ares.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/resolver_thread.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/select.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/server.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/socket.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/ssl.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/subprocess.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/thread.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/threading.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/threadpool.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/timeout.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/util.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/win32util.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    copying gevent/wsgi.py -> build/lib.macosx-10.10-x86_64-3.4/gevent

    running build_ext

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 316, in <module>

        run_setup(ext_modules)

      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 311, in run_setup

        "Development Status :: 4 - Beta"])

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 148, in setup

        dist.run_commands()

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands

        self.run_command(cmd)

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command

        cmd_obj.run()

      File "/usr/local/lib/python3.4/site-packages/setuptools-12.2-py3.4.egg/setuptools/command/install.py", line 61, in run

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/install.py", line 539, in run

        self.run_command('build')

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command

        self.distribution.run_command(command)

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command

        cmd_obj.run()

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build.py", line 126, in run

        self.run_command(cmd_name)

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command

        self.distribution.run_command(command)

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command

        cmd_obj.run()

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 339, in run

        self.build_extensions()

      File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions

        self.build_extension(ext)

      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 234, in build_extension

        self.gevent_prepare(ext)

      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 231, in gevent_prepare

        configure(self, ext)

      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 139, in configure_libev

        make_universal_header('config.h', 'SIZEOF_LONG', 'SIZEOF_SIZE_T', 'SIZEOF_TIME_T')

      File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 111, in make_universal_header

        print >>f, line

    TypeError: unsupported operand type(s) for >>: 'builtin_function_or_method' and '_io.TextIOWrapper'

    Running '/bin/sh /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/libev/configure > configure-output.txt' in /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/build/temp.macosx-10.10-x86_64-3.4/libev

    ----------------------------------------
    Command "/usr/local/opt/python3/bin/python3.4 -c "import setuptools, tokenize;__file__='/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-2a9m9jp5-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent

Incompatibility with multiprocessing.Manager

grequests breaks the multiprocessing.Manager() functionality. To replicate try to execute a script containing the following:

from multiprocessing import Manager
print Manager().Queue()

and then:

import grequests
from multiprocessing import Manager
print Manager().Queue()

:(

error(24, 'Too many open files')

Basically the same issue as with requests
https://github.com/kennethreitz/requests/issues/239

Eventlet support

First at all, thanks for porting requests to the asynchronous world, I loved it!

However I'm using Eventlet instead of Gevent so I adapted the code to make
it work with Eventlet. You can see it at https://github.com/jmg/erequests.

I was wondering if we can make a unified version where the library can
detect or be configured to use either Eventlet or Gevent based on the user
needs.

Many thanks.

No response timeout exception throws AsyncRequest object has no attribute Exception

Replicate:

url="http://www.google.com:81"# (chosen to cause no response)
item=grequests.get(url,hooks={'response':handle_result},timeout=20)
grequests.map(items, exception_handler=oops)

oops is not called but instead, eventually:

Traceback (most recent call last):
File "C:\Users\edwar_000\Desktop\quickloadtest.py", line 54, in
grequests.map(async_list, exception_handler=oops)
File "C:\Python27\lib\site-packages\grequests.py", line 123, in map
exception_handler(request, request.exception)
AttributeError: 'AsyncRequest' object has no attribute 'exception'

Clarify license

Hi there, we would like to update requests to 0.13.0, but we would need to know the exact licensing terms for grequest. 'BSD' isn't enough, as there are several versions (check http://spdx.org/licenses/). Also, the shipped LICENSE file doesn't seem to be BSD. Can you elaborate on that?

gevent patch_all call in grequests breaks pymysql, etc.

I was looking forward to using this awesome package with my app and gevent until I noticed that it accomplishes gevent-compatibility via gevent.monkey.patch_all():

try:
    import gevent
    from gevent import monkey as curious_george
    from gevent.pool import Pool
except ImportError:
    raise RuntimeError('Gevent is required for grequests.')

# Monkey-patch.
curious_george.patch_all(thread=False, select=False)

Unfortunately, patch_all() impacts all other packages/modules used by the app, breaking some of them that were not designed for reentrancy. For example, gevent.monkey.patch_all() broke the combination of pymysql/pooled_db (with sporadic, hard-to-reproduce failures); my app has several greenlets that on occasion make pymysql calls; I isolated the problem to gevent's monkey-patching of the socket module in that case.

A local solution involving monkey-patching of just the requests package to use gevent's variants of blocking API's would greatly enhance compatibility of grequests with apps and eliminate unexpected/undesirable side-effects on other packages. For example, I recently monkey-patched HBase/Thrift via the following code snippet with the desired effect on Thrift and no harm to the rest of the app (pymysql, etc.):

# Monkey-patch Thrift's TSocket to use gevent.socket to make our HBase interface
# gevent-friendly
import gevent.socket
from thrift.transport import TSocket
TSocket.socket = gevent.socket

gevent-zeromq is another example of local monkey-patching, albeit a more complex one: https://github.com/traviscline/gevent-zeromq/blob/master/gevent_zeromq/__init__.py

Thank you,
Vitaly

Grequests use of `monkey.patch_all(...)` breaks my application

The aggressive python stdlib monkey patching originating from https://github.com/kennethreitz/grequests/blob/master/grequests.py#L21 causes a lot of breakage throughout my app. The degree of monkey patching causes the redis client to error out which in turn then breaks all async celery processing.

I really like the grequests lib, but cannot use it if it's going to make things that used to work fine not work anymore.

  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/billiard/pool.py", line 1426, in safe_apply_callback
    fun(*args)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 347, in on_success
    self.acknowledge()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 453, in acknowledge
    self.on_ack(logger, self.connection_errors)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 100, in ack_log_error
    self.ack()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 95, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 511, in basic_ack
    self.qos.ack(delivery_tag)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 127, in ack
    self._remove_from_indices(delivery_tag).execute()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1919, in execute
    return execute(conn, stack, raise_on_error)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1811, in _execute_transaction
    self.parse_response(connection, '_')
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1882, in parse_response
    self, connection, command_name, **options)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 387, in parse_response
    response = connection.read_response()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 307, in read_response
    response = self._parser.read_response()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 105, in read_response
    response = self.read()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 90, in read
    return self._fp.readline()[:-2]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 392, in recv
    self._wait(self._read_event)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 298, in _wait
    self.hub.wait(watcher)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 341, in wait
    result = waiter.get()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
LoopExit: This operation would block forever

Additional references pertaining to gevent monkey patching issues:

• http://stackoverflow.com/questions/13040458/why-is-gevent-spawn-different-than-a-monkeypatched-threading-thread
• jrief/django-websocket-redis#3
• http://bighow.net/5332403-Why_is__gevent_spawn__different_than_a_monkeypatched__threading_Thread____.html
• http://stackoverflow.com/questions/9192539/using-gevent-monkey-patching-with-threading-makes-thread-work-serially
• http://pastebin.com/TPYMvuVr

grequests does not include gevent dependency in the setup.py

I was able to install grequests whitout having gevent. The dependency must be added to the setup.py so pip will try to install the dependency.

imap size.

Great lib!
But: in imap there is a size=2 as default, its very missleading. With size=2 parameter library works similar to blocking version. I would highly recommend changing it to maxint, if in map there is no throttling enabled.

Does grequests work with the tornado framework?

How does gevent and tornado ioloop interact with each other? Will this setup be problematic?