mazzzystar / proxy

A simple tool for fetching usable proxies from several websites.

Proxy

A tiny tool for crawling, assessing, storing some useful proxies.中文版

Construct your ip pool

Install mysql:

pip install pymysql requests

Modify db connection information in config.py.

# crawl, assess and store proxies
python ip_pool.py

# assess proxies quality in db periodically.
python assess_quality.py

Demo on how to use these proxies.

Please first construct your ip pool.

Crawl github homepage data:

# visit database to get all proxies
ip_list = []
try:
    cursor.execute('SELECT content FROM %s' % cfg.TABLE_NAME)
    result = cursor.fetchall()
    for i in result:
        ip_list.append(i[0])
except Exception as e:
    print e
finally:
    cursor.close()
    conn.close()

# use this proxies to crawl website
for i in ip_list:
    proxy = {'http': 'http://'+i}
    url = "https://www.github.com/"
    r = requests.get(url, proxies=proxy, timeout=4)
    print r.text

More detail in crawl_demo.py。

Contact

[email protected]

proxy's Issues

觉得爬到的HTTP代理很奇怪

你好，我使用此程序爬到了不少代理，但是使用这些代理，只能访问百度，却无法访问别的网站，(即如果不是用百度作为筛选网站，几乎得不到可用代理)，请问这是怎么回事？

可以请教一下数据库怎么创建吗？小白一枚

希望作者有时间能写个数据库创建教程，之前没接触过这些，有点茫然，但确实对您的成果感兴趣