fetch_url doesn't return RawResponse and doesn't provide access to response code about trafilatura

[adbar]

Hi @edkrueger, thanks for your feedback. I agree there is something missing here but in my opinion it's more a documentation issue.

Rather than risking to break existing code I'd prefer to explain how to use RawResponse in order to preserve necessary information. To sum up, we have two different modes:

• expert: RawResponse
• seamless with decode=True (it may have been a debatable design decision that the return values are not consistent though)

We could break things in expert mode (i.e. change the class and its attributes) and extend the docs, does that work for you and do you see a way to move on?

[edkrueger]

Hey @adbar, I think the mixed returns add some complexity to using the tool. For example, consider someone has written code in the "expert mode" but then decides to use decode=True. Changing that flag would break their code.

Also, getting the status code along with the HTML seems essential to most analyses. For example, you want to filter out cases where you get the HTML for a 404 page.

In order to avoid breaking anything in the external interface, here is an idea:

• let's leave fetch_url function's external interface alone, document it and add a deprecation warning.
• Make a new "easy mode" function that returns the HTML and status code as a tuple, so you can do html, status_code = easy_fetch(url).

[adbar]

The problem is that fetch_url() is mentioned in various places on the Internet (not just in the docs) with its current behavior and people are going to wonder why it doesn't work anymore (especially beginners). Even if a deprecation is nice we could maybe work on the more advanced functionality first.

I believe it's easier to make construction work on parts for which developers are likely to read the docs or are ready for a bit of tinkering:

1. Let's assume your easy_fetch suggestion is actually the current fetch_url(decode=True), albeit without status code
2. From there we could start to deprecate fetch_url(decode=False) and create a new function like expert_fetch() (there might be a better name)
3. The latter would offer the functionality you mention: Response object with data, status, url, and html (conditional on value of decode argument), as well as as_dict method or argument, and even more since we're at it
4. Since _handle_response is clearly designed for internal use it shouldn't be troublesome to update it
5. Anybody able to catch and understand the deprecation notice in (2) will likely be happy about the new, more configurable function

What do you think?

[adbar]

@edkrueger What do you think and do you have time to work on the PR?