requests / requests-threads

🎭 Twisted Deferred Thread backend for Requests.

Requests

Requests is a simple, yet elegant, HTTP library.

>>> import requests
>>> r = requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
'{"authenticated": true, ...'
>>> r.json()
{'authenticated': True, ...}

Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your PUT & POST data — but nowadays, just use the json method!

Requests is one of the most downloaded Python packages today, pulling in around 30M downloads / week— according to GitHub, Requests is currently depended upon by 1,000,000+ repositories. You may certainly put your trust in this code.

Installing Requests and Supported Versions

Requests is available on PyPI:

$ python -m pip install requests

Requests officially supports Python 3.8+.

Supported Features & Best–Practices

Requests is ready for the demands of building robust and reliable HTTP–speaking applications, for the needs of today.

• Keep-Alive & Connection Pooling
• International Domains and URLs
• Sessions with Cookie Persistence
• Browser-style TLS/SSL Verification
• Basic & Digest Authentication
• Familiar dict–like Cookies
• Automatic Content Decompression and Decoding
• Multi-part File Uploads
• SOCKS Proxy Support
• Connection Timeouts
• Streaming Downloads
• Automatic honoring of .netrc
• Chunked HTTP Requests

API Reference and User Guide available on Read the Docs

Cloning the repository

When cloning the Requests repository, you may need to add the -c fetch.fsck.badTimezone=ignore flag to avoid an error about a bad commit (see this issue for more background):

git clone -c fetch.fsck.badTimezone=ignore https://github.com/psf/requests.git

You can also apply this setting to your global Git config:

git config --global fetch.fsck.badTimezone ignore

---

requests-threads's Issues

Exit after Run

Hello,

I'm just learning async, so this might just be dumb, and it seems my piece here is working fine, other than it is returning Process finished with exit code 0 after I run the class.

`class foo:

def __init__(self):
    self.session=AsyncSession(n=100)
    self.main_out=self.session.run(self.main)
    print("still alive")

async def main(self):
    rs = []
    for _ in range(100):
        rs.append(await self.sub('bar'))
    return[ (x[0].json(),x[1]) for x in rs]

async def sub(self,key):
    return  await self.session.get('http://httpbin.org/get'),key`

readme example not async?

The example in the readme file reads:

from requests_threads import AsyncSession

session = AsyncSession(n=100)

async def _main():
    rs = []
    for _ in range(100):
        rs.append(await session.get('http://httpbin.org/get'))
    print(rs)

if __name__ == '__main__':
    session.run(_main)

shouldn't it be

from requests_threads import AsyncSession

session = AsyncSession(n=100)

async def _main():
    rs = []
    for _ in range(100):
        rs.append(session.get('http://httpbin.org/get'))
    for i in range(100):
        rs[i] = await rs[i]
    print(rs)

if __name__ == '__main__':
    session.run(_main)

to be run async?

POST METHOD ?

Can we use the post method here?

async/await example question

I'm not that familiar with this new AsyncSession object, but it looks to me like the async/await example doesn't actually perform those GET requests concurrently as you are awaiting for each one to return one by one. Should that example be this instead?

async def _main():
    for future in asyncio.as_completed([session.get('http://httpbin.org/get') for _ in range(100)]):
        print(await future)

Add example for POST request

Currently the documentation provides an example on for GET request. Can an example for POST request that demonstrates sending a payload and setting headers be add to the documentation?

Would it be something like this

from requests_threads import AsyncSession

session = AsyncSession(n=100)
async def _main():
  url = u"https://api.github.com/repos/psf/requests/issues/482/comments"
  body = json.dumps({u"body": u"Sounds great! I'll get right on it!"})
  response = await session.post(url=url, data=body)

Thanks

Can't pass in a deferred to session.run under Python 3

The session.run() function has no path for a non-coroutine function being passed in, it'll simply return without calling twisted.react():

requests-threads/requests_threads.py

Lines 61 to 72 in b4d4384

	def run(self, f):
	# Python 3 only.
	if hasattr(inspect, 'iscoroutinefunction'):
	# Is this a coroutine?
	if inspect.iscoroutinefunction(f):
	def w(reactor):
	return self.wrap(f())
	# If so, convert coroutine to Deferred automatically.
	return task.react(w)
	else:
	# Otherwise, run the Deferred.
	return task.react(f)

Under Python 3, line 63 is True, but if line 65 is False (so f is not a coroutine function) , the function ends. Should the else on line 70 be dropped, and lines 71 and 72 unindented?

ProactorEventLoop is not supported

I get this error:

Traceback (most recent call last):
  File "C:\Users\Menna\Desktop\requests-futures\twayback.py", line 3, in <module>
    session = AsyncSession(n=100)
  File "C:\Users\Menna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests_threads.py", line 37, in __init__
    asyncioreactor.install(loop)
  File "C:\Users\Menna\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\asyncioreactor.py", line 308, in install
    reactor = AsyncioSelectorReactor(eventloop)
  File "C:\Users\Menna\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\asyncioreactor.py", line 63, in __init__
    raise TypeError(
TypeError: ProactorEventLoop is not supported, got: <ProactorEventLoop running=False closed=False debug=False>

upon running the following Python script:
https://pastebin.com/raw/xrujJmSu

Use the following example command if you'd like to run the script:
script.py -u jack -from 20220205 -to 20220206

Do multiple http_get url and get all responses, then return all responses to main()

I am a newbie for using this library.
I want to call an sub-function in main() to do http_get multiple urls, and return all responses by dict().
But I don't how to use this python 2.7 sample code to finish it.
I try to move code from main() to other sub-function and use returnValue(dict()) to return to main(), but the program exits when inlineCallbacks sub-function is done, so main function can not do anything after calling sub-function.
Could anyone can give some idea or direction? Thanks a lot.

from twisted.internet.defer import inlineCallbacks
from twisted.internet.task import react
from requests_threads import AsyncSession

session = AsyncSession(n=100)

@inlineCallbacks
def main(reactor):
    responses = []
    for i in range(100):
        responses.append(session.get('http://httpbin.org/get'))

    for response in responses:
        r = yield response
        print(r)

if __name__ == '__main__':
    react(main)