Default Delay causes Service Unavailable about pagodo

[opsdisk]

I tried adding some logic to detect an HTTP 503 error and do some backoff throttling, but the underlying request is made in the googlesearch module. You may be able to spread the load between a couple VPS servers or I could add an option to round-robin through a list of HTTP proxies, but again the underlying request is made in the googlesearch module (https://github.com/MarioVilas/googlesearch/blob/master/googlesearch/__init__.py#L124)

[cr4zyd3v]

I tried to bypass google bot detection system with random countries like google.cz, google.pl, google.com.br, using rondomized proxies, with random user agents and random sleep time between requests but I got banned.. there are some dorks that will trigger google detection like inurl:".php" or "site:xxx.com" but if you try simple requests like "?id=foo" (with out file type) google will not consider it as a bot.. I did no tried too much so i cant confirm that.. I heard that v3n0m project has a captcha solver but it would be hard to implement in pagodo i guess :d

[opsdisk]

Just pushed some updates to master. I've had success running this lately with the new default values...it may take around 4 days to complete though. I'm going to close this for now, unless you have some new data I can work with.