Extends the Python requests library by adding additional options for proxy/user agent rotation as well as tracking.
- Requires the requests library
- Requires two .txt files: one which contains a list of webpages with proxies and the other with a list of user agents
-
Create a virtual environment for your project and activate
virtualenv -p python3 [projectname]
-
Clone repository
git clone https://github.com/hd9319/http-proxy
-
Install dependencies using requirements.txt files
pip install -r requirements.txt
- Import ProxyRequest class into any project and use in place of requests library.
- Instantiate class and pass in parameters for retrieving list of user agents and proxies
- Refresh proxies before running any type of requests
- request() method returns a dictionary containing a response object, headers and proxy ip used for making the call.
from proxy import ProxyRequest
proxy_urls_path = 'C/.../*.txt'
user_agents_path = 'C/.../*.txt'
requests = ProxyRequest(proxy_urls_path, user_agents_path, timer=0)
acceptable_status_codes=[200])
_ = requests.refresh_proxies()
response = requests.request(request, headers)