domoench / crawler Goto Github PK
View Code? Open in Web Editor NEWA website crawler
A website crawler
www.site.com/users/
if you don't care to crawl every user page in the domain.Seems to happen more with large sites - after crawling has been progressing for a while.
The result is the program eventually deadlocks. Perhaps related to the GIL?
Add a stringIO parameter to the OutputThread constructor so its more customizable and testable.
For example:
On page http://cholula.com/recipes/drinks/
there is a link with a relative href of monterrey_martini.php
.
Currently the conversion from relative to absolute produces http://cholula.com/monterrey_martini.php
It should produce http://cholula.com/recipes/drinks/monterrey_martini.php
Make it multithreaded so you aren't blocking on network I/O for such a large fraction of runtime.
Issue can be seen by crawling http://www.hrc.utexas.edu/. Is thrown when trying to iterate through the etree, not during the etree's construction.
The reason the etree.HTML() actually returns None if it fails. In this instance if failed because it tried to parse a .qtl file as an HTML page.
Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/david/Desktop/env/webcrawler/crawlthread.py", line 37, in run
page_data = self.processPage(url)
File "/Users/david/Desktop/env/webcrawler/crawlthread.py", line 70, in processPage
p = ParseHtml(url)
File "/Users/david/Desktop/env/webcrawler/parseHtml.py", line 38, in __init__
for e in self.root.iter('a'):
AttributeError: 'NoneType' object has no attribute 'iter'
One site that exposes the problem is thunderbirdcoffee.com.
ERROR: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
Still need to call crawlqueue.task_done() in this case.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.