Problems with streamed responses about requests-cache

[JWCook]

Thanks for looking into this. So the more general case of this is when a request results in a cache miss, but the same request with updated headers results in a cache hit. That isn't something I expected, but it makes sense after looking at your example with the GitHub API.

Specifically, this is a corner case with:

• streaming requests
• conditional requests
• response redirects that change based on validation status

In this case, when the remote content has changed (200 response instead of 304) GitHub will redirect the request from api.github.com to codeload.github.com.

[JWCook]

Thanks for the bug report. That attribute is supposed to have been set in requests.Response.__init__() before being wrapped by OriginalResponse.

I'm not sure yet why that's happening, but it should be easy enough to fix.

[netomi]

I am not 100 percent sure why it happens, will try to compile a test case. It happens when accessing the api from GitHub to download an archive of a repository with stream enabled.

Atm I use force refresh to circumvent the problem. I have not noticed with a previous version of the cache, only since I updated to the latest version but that might be a concidence.

[netomi]

Ok so this test fails after the response has been cached in a filesystem cache.
Strangely, if I run the test a second time, it still works, but from the third time on it fails consistently with the error as above. So something weird is going on that I will need to dig further into it.

from requests_cache import CachedSession
from tempfile import TemporaryFile


def test():
    session = CachedSession(
            "test",
            backend="filesystem",
            use_cache_dir=False,
            cache_control=True,
            allowable_methods=["GET"],
        )

    headers = {
        "Accept": "application/vnd.github+json",
        "X-GitHub-Api-Version": "2022-11-28",
        "X-Github-Next-Global-ID": "1",
    }

    response = session.request(
            "GET",
            url="http://api.github.com/repos/requests-cache/requests-cache/zipball/main",
            headers=headers,
            refresh=True,
            stream=True,
        )

    with TemporaryFile() as file:
        for chunk in response.iter_content(chunk_size=8192):
            file.write(chunk)

[netomi]

I do believe its due to redirects. The url that is requested is redirected to some other url.
At some point in the _send_and_cache method we receive a CachedResponse that we try to wrap in an OriginalResponse class which then leads to the problem. If I change the code to the following:

        if isinstance(response, CachedResponse):
            return response
        else:
            return OriginalResponse.wrap_response(response, actions)

the problem goes away. or this could also happen in the wrap_response method itself. Though I do not fully understand if the execution flow is correct in case of redirects and this would require a fix at a different place to make it clean.

[netomi]

Initially I thought its related to an update of the requests-cache library but I guess it was just a concidence and I tested it with older versions and triggered the same behavior. GitHub must have added a redirect for that API call that resulted in that behavior as it was working perfectly for a couple of months before.

Ty for fixing this.