Comments (3)
Hot to reproduce.
from scrapy-proxynova.
Yeah, errors are different every time. Sometime the server is clearly down, sometimes it times out, sometimes the db is down so the page loads but without any proxy. The latter happened to me this morning and the spider started with an empty list of proxies resulting in a lot of errors before I realized what happened.
One thing we can do maybe is if we cannot connect to the site (timeout or error 500) or list of proxies is empty then throw an error?
from scrapy-proxynova.
Well, this is a good solution.
from scrapy-proxynova.
Related Issues (9)
- It's not working
- Have a lock mechanism to prevent multiple scrapy instances to generate proxy lists at the same time HOT 3
- Multi thread the process to check connections when generating proxy list HOT 2
- Track how fast each proxy responds and prefer the faster proxies. HOT 2
- Fix parsing because Proxynova changed the way they offuscate the IP address HOT 1
- Use proxy_list.txt instead of fetching the proxy list from the different pages in proxynova HOT 1
- Got an error HOT 1
- Can not found proxies.py file to generate the proxy list
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapy-proxynova.