darthbear / scrapy-proxynova

Use scrapy with a list of proxies generated from proxynova.com

scrapy-proxynova

Use scrapy with a list of proxies generated from proxynova.com

The first run will generate the list of proxies from http://proxynova.com and store it in the cache.

It will individually check each proxy to see if they work and remove the ones that timed out or cannot connect to.

Example:

./run_example.sh

To regenerate the proxy list, run: python proxies.py

In settings.py add the following line: DOWNLOADER_MIDDLEWARES = { 'scrapy_proxynova.middleware.HttpProxyMiddleware': 543 }

Options

Set these options in the settings.py.

• PROXY_SERVER_LIST_CACHE_FILE — a file to store proxies list. Default: proxies.txt.
• PROXY_BYPASS_PERCENT — probability for a connection to use a direct connection and not use a proxy

scrapy-proxynova's Issues

Fix parsing because Proxynova changed the way they offuscate the IP address

Can not found proxies.py file to generate the proxy list

[scrapy-proxynova] ~> ls
README.md           run_example.sh
build               scrapy_proxynova
dist                scrapy_proxynova.egg-info
example-project         setup.py

Track how fast each proxy responds and prefer the faster proxies.

We could track last 10 response times in deque for each proxy. And from time to time resort them according to the average response time.

Selection should use gaussian distrubution, like that:

from random import gauss

proxies = [some list where fastest proxies go first]

index = int(round(gauss(0, len(proxies) / 3.0)))
index = min(index, len(proxies) - 1)

return proxies[index]

Error out if proxynova is down or has no proxies available

Multi thread the process to check connections when generating proxy list

Have a lock mechanism to prevent multiple scrapy instances to generate proxy lists at the same time

If scrapy is run multiple times in separate processes, we want to prevent them to all do the same job and try to generate a proxy list. Rather the first one should generate the list and the other ones wait for it to complete before starting crawling.

Use proxy_list.txt instead of fetching the proxy list from the different pages in proxynova

http://www.proxynova.com/proxy_list.txt?country=us

It's not working

Not working response 404 not found

Got an error

I got this error while running the example project
Error: Cannot find any available proxies
I tried accessing the proxy page from my browser but proxy nova said that this list is currently offline so i think it's proxynova issue rather than scrapy-proxynova.