Comments (4)
I can indeed reproduce the bug. Images are not my priority, the corresponding code mostly consists of a series of contributions and it's not perfect. Let's see if someone can improve on this.
from trafilatura.
I can indeed reproduce the bug. Images are not my priority, the corresponding code mostly consists of a series of contributions and it's not perfect. Let's see if someone can improve on this.我确实可以重现这个错误。图片不是我的重点,相应的代码主要由一系列贡献组成,而且并不完美。让我们看看是否有人能对此加以改进。
Thank you very much, I found that most of the sites can't get pictures in the process, and this is just one of the cases
from trafilatura.
That's my code: `from trafilatura import fetch_url, extract
url = 'https://shumeipai.nxez.com/2020/06/11/stanford-pupper-assembly-tutorial.html' downloaded = fetch_url(url) result = extract(downloaded, output_format='markdown', favor_recall=True, include_images=True, include_links=True)`
Try it:
from trafilatura import fetch_url, extract
import re
url = 'https://shumeipai.nxez.com/2020/06/11/stanford-pupper-assembly-tutorial.html'
downloaded = fetch_url(url)
img_src_regex = r'<img[^>]+src="([^"]+)"[^>]*>'
def replace_img_tags(match):
src = match.group(1)
return f'111222333000-{src}-000333222111'
downloaded = re.sub(img_src_regex, replace_img_tags, downloaded)
result = extract(downloaded, output_format='markdown', favor_recall=True, include_images=True, include_links=True)
result = re.sub("111222333000-","<img src=\"", result)
result = re.sub("-000333222111","\">", result)
print(result)
from trafilatura.
That's my code:
from trafilatura import fetch_url, extract url = 'https://shumeipai.nxez.com/2020/06/11/stanford-pupper-assembly-tutorial.html' downloaded = fetch_url(url) result = extract(downloaded, output_format='markdown', favor_recall=True, include_images=True, include_links=True)
Try it:
from trafilatura import fetch_url, extract import re url = 'https://shumeipai.nxez.com/2020/06/11/stanford-pupper-assembly-tutorial.html' downloaded = fetch_url(url) img_src_regex = r'<img[^>]+src="([^"]+)"[^>]*>' def replace_img_tags(match): src = match.group(1) return f'111222333000-{src}-000333222111' downloaded = re.sub(img_src_regex, replace_img_tags, downloaded) result = extract(downloaded, output_format='markdown', favor_recall=True, include_images=True, include_links=True) result = re.sub("111222333000-","<img src=\"", result) result = re.sub("-000333222111","\">", result) print(result)
Thanks, it worked, I modified the source code of trafilatura and was able to solve part of the problem, but as I was using it I realized that most of the url's didn't work perfectly, there were too many adaptations needed, gave up!
from trafilatura.
Related Issues (20)
- Image/Video caption and credits removal HOT 3
- Footer removal HOT 1
- trafilatura.fetch_url Timeout is set but does not work HOT 2
- Parts of article block are sometimes not being extracted HOT 1
- Error parsing non-English web pages HOT 2
- Sometimes, html tags remain on the string HOT 2
- Deprecate GUI in its current form (Gooey)
- Deprecate Python 3.6 & 3.7
- weird xml extraction HOT 2
- Account for empty cells in table extraction (xml) HOT 3
- some extraction duplicated in xml HOT 3
- links/urls are not apprearing using extract HOT 1
- Can I get an extracted element's CSS selector? HOT 2
- AttributeError in baseline extraction of JSON text
- Impossible to extract Ryan Reynolds website HOT 1
- Missing h1 heading if <header> outside of <article> HOT 1
- Validate value of `output_format` in `extract()` and `bare_extraction()`
- CLI: Trigger formatting parameter when the output is in Markdown format
- Add magic_html to benchmarks HOT 1
- Extraction with `include_images=True` takes too much time HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trafilatura.