C:\Users\dell\AppData\Local\Programs\Python\Python35-32\python.exe D:/scrapyspider/tutorial/main.py
2016-07-17 01:04:49 [scrapy] INFO: Scrapy 1.2.0dev2 started (bot: tutorial)
2016-07-17 01:04:49 [scrapy] INFO: Overridden settings: {'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2774.3 Safari/537.36', 'SPIDER_MODULES': ['tutorial.spiders'], 'NEWSPIDER_MODULE': 'tutorial.spiders', 'DOWNLOAD_DELAY': 1, 'BOT_NAME': 'tutorial', 'SCHEDULER': 'scrapy_redis.scheduler.Scheduler'}
2016-07-17 01:04:49 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats']
2016-07-17 01:04:49 [dmoz] INFO: Reading start URLs from redis key 'dmoz:start_urls' (batch size: 16)
2016-07-17 01:04:49 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-07-17 01:04:49 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-07-17 01:04:50 [scrapy] INFO: Enabled item pipelines:
['tutorial.pipelines.DmozPipeline']
2016-07-17 01:04:50 [scrapy] INFO: Spider opened
2016-07-17 01:04:50 [scrapy] INFO: Closing spider (shutdown)
Unhandled error in Deferred:
2016-07-17 01:04:50 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\commands\crawl.py", line 57, in run
self.crawler_process.crawl(spname, **opts.spargs)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\crawler.py", line 163, in crawl
return self._crawl(crawler, *args, **kwargs)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\crawler.py", line 167, in _crawl
d = crawler.crawl(*args, **kwargs)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\internet\defer.py", line 1273, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\internet\defer.py", line 1125, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\crawler.py", line 87, in crawl
yield self.engine.close()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 100, in close
return self._close_all_spiders()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 340, in _close_all_spiders
dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 340, in <listcomp>
dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 298, in close_spider
dfd = slot.close()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 44, in close
self._maybe_fire_closing()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 51, in _maybe_fire_closing
self.heartbeat.stop()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\internet\task.py", line 202, in stop
assert self.running, ("Tried to stop a LoopingCall that was "
builtins.AssertionError: Tried to stop a LoopingCall that was not running.
2016-07-17 01:04:50 [twisted] CRITICAL:
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy_redis\scheduler.py", line 120, in open
debug=spider.settings.getbool('DUPEFILTER_DEBUG'),
TypeError: __init__() got an unexpected keyword argument 'server'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\crawler.py", line 74, in crawl
yield self.engine.open_spider(self.spider, start_requests)
ValueError: ("Failed to instantiate dupefilter class '%s': %s", 'scrapy.dupefilters.RFPDupeFilter', TypeError("__init__() got an unexpected keyword argument 'server'",))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\internet\defer.py", line 1125, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\crawler.py", line 87, in crawl
yield self.engine.close()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 100, in close
return self._close_all_spiders()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 340, in _close_all_spiders
dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 340, in <listcomp>
dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 298, in close_spider
dfd = slot.close()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 44, in close
self._maybe_fire_closing()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\scrapy\core\engine.py", line 51, in _maybe_fire_closing
self.heartbeat.stop()
File "C:\Users\dell\AppData\Local\Programs\Python\Python35-32\lib\site-packages\twisted\internet\task.py", line 202, in stop
assert self.running, ("Tried to stop a LoopingCall that was "
AssertionError: Tried to stop a LoopingCall that was not running.
Process finished with exit code 0
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_ORDER = 'BFO'
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIS_URL = None
REDIS_HOST = '127.0.0.1'
REDIS_PORT = 6379