Another concurrency spider, powered by gevent,requests,pyquery
- The concurrency foundation on gevent
- The spider strategy highly configurable:
- Max depth
- Sum totals of urls
- Max concurrency of http request,avoid dos
- Request headers and cookies
- Only crawl same host url
- Only crawl same domain url
- Max running time
- python 2.7
- gevent 1.0dev
- requests 1.0.3
- pyquery 1.2.4
spider = Spider()
spider.setRootUrl("http://www.sina.com.cn")
spider.run()
- Support Distributed , update
gevent.Queue
->redis.Queue
- Storage system highly configurable
- Support Ajax url (webkit etc..)
Copyright © 2013 by kenshin
Under MIT license : rem.mit-license.org