GithubHelp home page GithubHelp logo

Error in requests version 2.31.0 with urllib3 2.2.1 and 2.10 raise IncompleteRead(self._fp_bytes_read, self.length_remaining) about requests HOT 2 CLOSED

pcor avatar pcor commented on September 25, 2024
Error in requests version 2.31.0 with urllib3 2.2.1 and 2.10 raise IncompleteRead(self._fp_bytes_read, self.length_remaining)

from requests.

Comments (2)

pcor avatar pcor commented on September 25, 2024

I have installed an empty enviroment with conda and requests:

python 3.12.2
requests 2.31.0
urllib3 1.26.18

And we have no errors, when I install urllib3 2.1.0, the program crashes, Below is a simple test program that crashes.

python 3.12.2
requests 2.31.0
urllib3 2.1.0

import requests
from io import BytesIO


class Test():
    def __init__(self):
        self.proxy = None
        self.headers = {}
        self.headers['Connection'] = 'close'
        self.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36'
        self.headers['Accept'] = '*/*'
        self.headers['Accept-Encoding'] = 'gzip, deflate'
        self.session = requests.Session()

    def get(self, url):
        self.session.headers = self.headers
        r = self.session.get(url, proxies=self.proxy, allow_redirects=True, stream=True)
        raw_content = r.raw.read()
        content_length = len(raw_content)
        r.raw._fp = BytesIO(raw_content)
        content=r.text
        with open('test.html', 'w', encoding='utf-8') as fo:
            fo.write(content)

lurl = 'https://www.nolon.es/es/es/comprar?MainUseId=&Invest=False&ClassId=&HighlightTypeId=&PropertyTypeId=&DistrictId=30&CountyId=&AreaRange.MinValue=&AreaRange.MaxValue=&PriceRange.MinValue=0&PriceRange.MaxValue=&CampaignId=&PropertyId=&ResultsPerPage=27&SortOrderBy=DescendingDate&SortOrderBy=DescendingDate'
Test().get(lurl)`



This is the error I get:

traceback (most recent call last):
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 712, in _error_catcher
yield
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 833, in _raw_read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(42084 bytes read, -21042 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 816, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 934, in stream
data = self.read(amt=amt, decode_content=decode_content)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 877, in read
data = self._raw_read(amt)
^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 811, in _raw_read
with self._error_catcher():
File "/opt/miniconda3/envs/test/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 729, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(42084 bytes read, -21042 more expected)', IncompleteRead(42084 bytes read, -21042 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/datos1/test/test.py", line 28, in
Test().get(lurl)
File "/home/datos1/test/test.py", line 23, in get
content=r.text
^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 923, in text
if not self.content:
^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(42084 bytes read, -21042 more expected)', IncompleteRead(42084 bytes read, -21042 more expected))

from requests.

sigmavirus24 avatar sigmavirus24 commented on September 25, 2024

You must never call r.text or r.content after consuming the underlying response data. That's you're actual bug. I suspect requests and urllib3 were silently returning nothing but you're code has always been the issue

from requests.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.