关于win10上的使用？ about tieba_spider

[barnett2010]

Crawling page 2023...
Crawling page 2024...
2020-03-11 21:32:41 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying
<GET https://tieba.baidu.com/p/3564343563?pn=5098> (failed 3 times): User timeo
ut caused connection failure: Getting https://tieba.baidu.com/p/3564343563?pn=50
98 took longer than 180.0 seconds..
2020-03-11 21:32:42 [scrapy.core.scraper] ERROR: Error downloading <GET https://
tieba.baidu.com/p/3564343563?pn=5098>
Traceback (most recent call last):
File "c:\python37\lib\site-packages\twisted\internet\defer.py", line 1416, in
_inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "c:\python37\lib\site-packages\twisted\python\failure.py", line 512, in t
hrowExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "c:\python37\lib\site-packages\scrapy\core\downloader\middleware.py", lin
e 42, in process_request
defer.returnValue((yield download_func(request=request, spider=spider)))
File "c:\python37\lib\site-packages\twisted\internet\defer.py", line 654, in _
runCallbacks
current.result = callback(current.result, *args, **kw)
File "c:\python37\lib\site-packages\scrapy\core\downloader\handlers\http11.py"
, line 377, in _cb_timeout
raise TimeoutError("Getting %s took longer than %s seconds." % (url, timeout
))
twisted.internet.error.TimeoutError: User timeout caused connection failure: Get
ting https://tieba.baidu.com/p/3564343563?pn=5098 took longer than 180.0 seconds
..

[barnett2010]

会有这样的提示，然后脚本停止运行

[Aqua-Dream]

这是你网断了吧，错误提示写的是网络180秒无响应

[barnett2010]

@Aqua-Dream
感谢大佬的回复。我又测试了几个贴吧，总页数有1万多页，基本都是下载到2000多时，就会出现上面的提示。

我猜测有个可能，现在贴吧限制了，17年前的帖子都不让看了。是不是这个引起的出错。。

---

还有一个，想要一个这样的功能。
如果有几十个贴吧想备份，
能不能同时写进config里，然后一次性下载完成。
或者写一个贴吧list.txt。config调用这个文件。
list里
按 吧名 数据库名 一行一个。

[Aqua-Dream]

你写个batch后台运行不就行了