Comments (5)
`
while True:
proxy = await proxies.get()
if proxy is None:
break
print(proxy)
if "HTTP" not in proxy.types:
continue
if "High" == proxy.types["HTTP"]:
row = '%s://%s:%d' % ("http", proxy.host, proxy.port)
r.set(row, 0, ex=60 * 60 * 24)
`
看到逻辑了,但是奇怪这个逻辑没有正常退出
from proxypool.
这里好像就并没有像书中说的那样,逻辑放在客户端,这里还是有些逻辑
from proxypool.
由于很多代理资源在**无法访问的网站,部署在国内的服务器上会影响资源的获取,所以推荐将服务器部署到国外的服务器。进程没有退出是因为有很多国内网站访问不了但是没有超时导致,多等段时间就可以了。
如果你需要连续运行可以加个while True循环
from proxypool.
其实我是用美国的vps,跑的,但是还等了一段时间,看代码逻辑理论上应该要正常退出,还有一个问题,就是proxybroker的爬取的资源其实好像不是特别多?我爬到就几百个,然后就卡住了(控制台没有输出)
关于是否需要连续运行,我觉得完全没必要,配合有效时间,定时增量跑即可
然后我发现,有些代理,ping得通,但是其实发出http request,目的地返回500,怀疑是代理有什么处理,因为其它同区域的代理是可以通的
from proxypool.
我用的是DigitalOcean,每天大概有1万左右的IP资源,可用的有好几千。
由于每个人的VPS不一样,网络情况不一样,所以得到的代理数量可能不一样。
from proxypool.
Related Issues (6)
- 在SFO2创建了一个,但还是爬着就停住了 HOT 17
- CONNECT:80 CONNECT:25 是什么代理? HOT 1
- PROVIDERS里的先后顺序是否和爬取顺序有关? HOT 2
- xicidaili.com 访问太快就503了 HOT 1
- 非Python程序猿 问个代码上的问题 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from proxypool.