dcoles / pycurl-requests

A Requests-compatible interface for PycURL.

PycURL Requests <pycurl://☤>

PycURL Requests is a Requests-compatible interface for PycURL.

Requirements

• Python 3.6+
• PycURL
• chardet

Installation

Latest release via pip:

pip install pycurl-requests [--user]

via Git:

git clone https://github.com/dcoles/pycurl-requests.git; cd pycurl-requests
python3 setup.py install [--user]

Quick-start

>>> import pycurl_requests as requests
>>> r = requests.get('https://api.github.com/repos/dcoles/pycurl-requests')
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf-8'
>>> r.encoding
'utf-8'
>>> r.text
'{\n  "id": 236427187,\n...'
>>> data = r.json()
>>> data['name']
'pycurl-requests'
>>> data['html_url']
'https://github.com/dcoles/pycurl-requests'
>>> data['description']
'A Requests-compatible interface for pycURL'

The library can also be used to run existing Python scripts that import the requests module. By running the script through the pycurl_requests helper, any use of the requests module will be automatically redirected to pycurl_requests.

python3 -m pycurl_requests -- script.py arg arg...

request tool

A basic curl-like command-line utility is included:

usage: request.py [-h] [-d DATA] [-H HEADER] [--json JSON] [-L] [-o OUTPUT]
                  [-X REQUEST] [-v]
                  url

A basic `curl`-like command-line HTTP utility

positional arguments:
  url                   URL of resource to connect to

optional arguments:
  -h, --help            show this help message and exit
  -d DATA, --data DATA  Add POST data
  -H HEADER, --header HEADER
                        Add custom request header (format: `Header: Value`)
  --json JSON           Add JSON POST data
  -L, --location        Follow redirects
  -o OUTPUT, --output OUTPUT
                        Write to file instead of stdout
  -X REQUEST, --request REQUEST
                        Request command to use (e.g. HTTP method)
  -v, --verbose         Verbose logging

This can also be used with the Requests library if PYCURLREQUESTS_REQUESTS environment variable is set to a non-null value.

Documentation

This library aims to be API compatible with Requests, thus the Requests documentation should be mostly applicable.

Adapters

PycURL support is implemented as a transport adapter. This means it's possible to use PycURL with the Requests library itself!

import pycurl
import requests
from pycurl_requests.adapters import PyCurlHttpAdapter

with requests.Session() as session:
    curl = pycurl.Curl()
    session.mount('https://', PyCurlHttpAdapter(curl))
    session.mount('http://', PyCurlHttpAdapter(curl))

    response = session.get('http://example.com')

cURL options

It is possible customize cURL's behaviour using the curl attribute on a Session object.

For example, to make a request without requesting the body:

import pycurl
import pycurl_requests as requests

with requests.Session() as session:
    session.curl.setopt(pycurl.NOBODY, 1)
    response = session.get('http://example.com')

See the pycurl.Curl object documentation for all possible curl attribute methods.

cURL exceptions

All pycurl.error exceptions are mapped to a requests.RequestException (or one of its subclasses).

For convenience, the original pycurl.error error message and cURL error code will be set on the exception object as the curl_message and curl_code attributes.

import pycurl_requests as requests

try:
    requests.get('http://connect_error')
except requests.RequestException as e:
    print('ERROR: {} (cURL error: {})'.format(e.curl_message, e.curl_code))

It is also possible to obtain the original pycurl.error using the __cause__ attribute.

Logging

Detailed log records from libcurl, including informational text and HTTP headers, can be shown by setting the curl logger (or sub-loggers) to DEBUG level:

import logging

logging.getLogger('curl').setLevel(logging.DEBUG)

Log records are split into dedicated sub-loggers for each type of record:

• curl.text — Informational text
• curl.header_in — Header data received from the peer
• curl.header_out — Header data sent to the peer

Known limitations

• No support for reading Cookies
• No support for client-side certificates
• No support for proxies
• No support for link headers (e.g. Response.links)
• No support for sending multi-part encoded files
• Basic support for Session objects (e.g. requests.Session)

License

Licensed under the MIT License.

pycurl-requests's Issues

See PR #1725 from asciimoo/searx

See searx/searx#1725

The package searx.httpclient is nearly autonomous and implements the requests API over curl.

I've encountered a segfault using HTTP/2 on Ubuntu 18.04: https://github.com/asciimoo/searx/blob/6abf4213d2c05c505836170dbf19483b06030799/searx/httpclient/misc.py

Remove pytest from the reuirements

Please remove pytest from the requirements

FEATURE SUGGEST: Update sessions headers

I didn't find that if the session is able to update headers like in requests api:
session = requests.Session()

session.headers.update({'x-test': 'true'})

This is the requests documentation for it:
https://requests.readthedocs.io/en/latest/user/advanced/#session-objects

curl options error

TypeError: request() got an unexpected keyword argument 'curl'

Multiple Set-Cookie headers merged incorrectly

what i got from POSTMAN

Set-Cookie: test!
Set-Cookie: PHPSESSID=8536fl1c5igh89aqsjuf3l40jm; path=/
Set-Cookie: TestCookie=%3A%202595; expires=Thu, 01-Jul-2021 07:56:03 GMT; Max-Age=3600

what i got from pycurl-requests (resp.headers.get("Set-Cookie", None))

Set-Cookie: test!, PHPSESSID=8536fl1c5igh89aqsjuf3l40jm; path=/, TestCookie=%3A%202595; expires=Thu, 01-Jul-2021 07:56:03 GMT; Max-Age=3600

this behaviour does not confirm to rfc.

Byte-type headers are not encoded correctly

Setting headers = {'Header': b'Value'} results in an incorrect header:

Header: b'Value'

The byte-string should be passed through unchanged.