Comments (4)
Hi @edkrueger, thanks for your feedback. I agree there is something missing here but in my opinion it's more a documentation issue.
Rather than risking to break existing code I'd prefer to explain how to use RawResponse
in order to preserve necessary information. To sum up, we have two different modes:
- expert:
RawResponse
- seamless with
decode=True
(it may have been a debatable design decision that the return values are not consistent though)
We could break things in expert mode (i.e. change the class and its attributes) and extend the docs, does that work for you and do you see a way to move on?
from trafilatura.
Hey @adbar, I think the mixed returns add some complexity to using the tool. For example, consider someone has written code in the "expert mode" but then decides to use decode=True
. Changing that flag would break their code.
Also, getting the status code along with the HTML seems essential to most analyses. For example, you want to filter out cases where you get the HTML for a 404 page.
In order to avoid breaking anything in the external interface, here is an idea:
- let's leave
fetch_url
function's external interface alone, document it and add a deprecation warning. - Make a new "easy mode" function that returns the HTML and status code as a tuple, so you can do
html, status_code = easy_fetch(url)
.
from trafilatura.
The problem is that fetch_url()
is mentioned in various places on the Internet (not just in the docs) with its current behavior and people are going to wonder why it doesn't work anymore (especially beginners). Even if a deprecation is nice we could maybe work on the more advanced functionality first.
I believe it's easier to make construction work on parts for which developers are likely to read the docs or are ready for a bit of tinkering:
- Let's assume your
easy_fetch
suggestion is actually the currentfetch_url(decode=True)
, albeit without status code - From there we could start to deprecate
fetch_url(decode=False)
and create a new function likeexpert_fetch()
(there might be a better name) - The latter would offer the functionality you mention:
Response
object with data, status, url, and html (conditional on value ofdecode
argument), as well asas_dict
method or argument, and even more since we're at it - Since
_handle_response
is clearly designed for internal use it shouldn't be troublesome to update it - Anybody able to catch and understand the deprecation notice in (2) will likely be happy about the new, more configurable function
What do you think?
from trafilatura.
@edkrueger What do you think and do you have time to work on the PR?
from trafilatura.
Related Issues (20)
- Extract page type with og:type or jd+json HOT 4
- What is the recommended approach to outputting a more readable article HOT 1
- Cannot extract heading correctly in a list HOT 5
- probe_alternative_homepage no_ssl arg from fetch_url HOT 1
- Doesnt extract li tags content with an id HOT 5
- 'lxml.etree._Element' object has no attribute 'text_content' HOT 17
- Getting warnings while testing extract() function with with pytest
- feat: add basic auth support HOT 4
- ValueError: signal only works in main thread HOT 2
- Proxy support to Trafilatura HOT 4
- Headers with classes dont get detected HOT 1
- Paras get broken up into fragments HOT 3
- Doesn't detect bullet points within tables HOT 5
- Simple HTML processing issue? HOT 4
- Unnecessary Comments LOG INFO HOT 1
- Codeblock Markdown formatting is missing HOT 3
- Cannot extract Heading tags HOT 26
- [1.6.0] New Content Hashes HOT 2
- Restrictions on Web Crawling HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trafilatura.