domoench / crawler Goto Github PK

Issue can be seen by crawling http://www.hrc.utexas.edu/. Is thrown when trying to iterate through the etree, not during the etree's construction.

The reason the etree.HTML() actually returns None if it fails. In this instance if failed because it tried to parse a .qtl file as an HTML page.

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/david/Desktop/env/webcrawler/crawlthread.py", line 37, in run
    page_data = self.processPage(url)
  File "/Users/david/Desktop/env/webcrawler/crawlthread.py", line 70, in processPage
    p = ParseHtml(url)
  File "/Users/david/Desktop/env/webcrawler/parseHtml.py", line 38, in __init__
    for e in self.root.iter('a'):
AttributeError: 'NoneType' object has no attribute 'iter'

Seems to happen more with large sites - after crawling has been progressing for a while.

The result is the program eventually deadlocks. Perhaps related to the GIL?

Still need to call crawlqueue.task_done() in this case.

Make it multithreaded so you aren't blocking on network I/O for such a large fraction of runtime.

One site that exposes the problem is thunderbirdcoffee.com.

ERROR: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

• Ability to specify start point of crawling
• Ability to specify a config file denoting subpaths to ignore. For example, maybe the path www.site.com/users/ if you don't care to crawl every user page in the domain.

For example:

On page http://cholula.com/recipes/drinks/ there is a link with a relative href of monterrey_martini.php.
Currently the conversion from relative to absolute produces http://cholula.com/monterrey_martini.php
It should produce http://cholula.com/recipes/drinks/monterrey_martini.php

Add a stringIO parameter to the OutputThread constructor so its more customizable and testable.