GithubHelp home page GithubHelp logo

关于该包的使用 about proxypool HOT 5 CLOSED

derekhe avatar derekhe commented on September 18, 2024
关于该包的使用

from proxypool.

Comments (5)

zplzpl avatar zplzpl commented on September 18, 2024

`
while True:
proxy = await proxies.get()
if proxy is None:
break
print(proxy)
if "HTTP" not in proxy.types:
continue
if "High" == proxy.types["HTTP"]:
row = '%s://%s:%d' % ("http", proxy.host, proxy.port)
r.set(row, 0, ex=60 * 60 * 24)

`

看到逻辑了,但是奇怪这个逻辑没有正常退出

from proxypool.

zplzpl avatar zplzpl commented on September 18, 2024

这里好像就并没有像书中说的那样,逻辑放在客户端,这里还是有些逻辑

from proxypool.

derekhe avatar derekhe commented on September 18, 2024

由于很多代理资源在**无法访问的网站,部署在国内的服务器上会影响资源的获取,所以推荐将服务器部署到国外的服务器。进程没有退出是因为有很多国内网站访问不了但是没有超时导致,多等段时间就可以了。

如果你需要连续运行可以加个while True循环

from proxypool.

zplzpl avatar zplzpl commented on September 18, 2024

其实我是用美国的vps,跑的,但是还等了一段时间,看代码逻辑理论上应该要正常退出,还有一个问题,就是proxybroker的爬取的资源其实好像不是特别多?我爬到就几百个,然后就卡住了(控制台没有输出)

关于是否需要连续运行,我觉得完全没必要,配合有效时间,定时增量跑即可

然后我发现,有些代理,ping得通,但是其实发出http request,目的地返回500,怀疑是代理有什么处理,因为其它同区域的代理是可以通的

from proxypool.

derekhe avatar derekhe commented on September 18, 2024

我用的是DigitalOcean,每天大概有1万左右的IP资源,可用的有好几千。
由于每个人的VPS不一样,网络情况不一样,所以得到的代理数量可能不一样。

from proxypool.

Related Issues (6)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.