TimeoutError is thrown on retrieval of own ip address about proxybroker

[th3l0b3]

same issue here as well.

[Sebisnow]

The funny thing is, that the error occurs very often but not all the time. Today proxybroker worked fine for once. On the next try the error reoccurred!

[gabrielgradinaru]

I have managed to reproduce this issue by running the following command multiple times:
proxybroker --log DEBUG grab --countries US --limit 10
And it seems to reproduce 10-20% of the time.

The root cause is:
python/asyncio#258
An easy workaround:
https://stackoverflow.com/questions/37778019/aiohttp-asyncio-runtimeerror-event-loop-is-closed

[constverum]

@gabrielgradinaru it seems the ProxyBroker is being run from a script, not via CLI, so the fix for the CLI is unlikely could help here.

  File "/home/sebastian/PycharmProjects/testing/test/test_prox2.py", line 19, in <module>
    loop.run_until_complete(tasks)
...
  File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/api.py", line 98, in find
    await set_my_ip(self._timeout, self._loop)

And as I can see was used the old 0.1.4 version.
@Sebisnow if the error still occurs could you please provide full content of the test_prox2.py file?

[gabrielgradinaru]

@constverum indeed, the current fix/workaround will only work for the CLI.

Looking a bit deeper it looks like in api.py line 373, cancel() is called on a Task but as you can see in the docs:
https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel

this does not guarantee that the task will be cancelled

I guess a better fix would be to wait until the Task is actually finished, I will try to find a way to do this when I will have time.

[constverum]

I guess a better fix would be to wait until the Task is actually finished, I will try to find a way to do this when I will have time.

ProxyBroker/proxybroker/cli.py

Lines 313 to 324 in 2612b58

	try:
	if tasks:
	loop.run_until_complete(asyncio.gather(*tasks, loop=loop))
	if ns.show_stats:
	broker.show_stats(verbose=True)
	else:
	loop.run_forever()
	except KeyboardInterrupt:
	broker.stop()
	finally:
	loop.stop()
	loop.close()

Basically, we could call broker.stop() before stopping the loop. It will cancel the tasks correctly.
But anyway to solve this issue need to add alternative way to get external IP (in addition to httpbin.org).

[gabrielgradinaru]

I agree that an alternative way do get external IP is useful here, but calling broker.stop() before stopping the loop is not going to help in this case. This is because broker._done() has already been called before in _grab:

ProxyBroker/proxybroker/api.py

Line 295 in 2a09c0f

self._done()

and the tasks have been popped from the _all_tasks queue:

ProxyBroker/proxybroker/api.py

Lines 368 to 374 in 2a09c0f

	def _done(self):
	log.debug('called done')
	while self._all_tasks:
	task = self._all_tasks.pop()
	if not task.done():
	task.cancel()
	self._push_to_result(None)

Please correct me if I'm wrong and this is not the case.
I have also tried to find a better fix than eliminating the loop.close() call but could not find one 😢

[Yuraz71]

Can I reopen this issue ?
In last week I be in struggle with "Could not get the external IP" exception.

Traceback (most recent call last):
File "refresh_proxy.py", line 50, in
prox = Proxies(loop)
File "refresh_proxy.py", line 13, in init
self._init_proxies(loop)
File "refresh_proxy.py", line 24, in _init_proxies
loop.run_until_complete(asyncio.wait_for(tasks, 120))
File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
return fut.result()
File "/usr/local/lib/python3.7/dist-packages/proxybroker/api.py", line 173, in find
ip = await self._resolver.get_real_ext_ip()
File "/usr/local/lib/python3.7/dist-packages/proxybroker/resolver.py", line 109, in get_real_ext_ip
raise RuntimeError('Could not get the external IP')
RuntimeError: Could not get the external IP

I found some incomprehensibility in resolver.py.
Maybe you can explain this problem for me, and I will leave everything as it is?

So, in resolver.py I found:
_ip_hosts = [
'https://wtfismyip.com/text',
'http://api.ipify.org/',
'http://ipinfo.io/ip',
'http://ipv4.icanhazip.com/',
'http://myexternalip.com/raw',
'http://ipinfo.io/ip',
'http://ifconfig.io/ip',
]
After that:
host = random.choice(self._ip_hosts)
self._ip_hosts.remove(host)

So, when we remove last item from this list we get exception.
And this is confirmed in practice.
I try refresh proxy list every 5 minutes. After 7th times I always get exception.
I try comment this line in local machine:
self._ip_hosts.remove(host)
and all works is fine.

@constverum, could you explain what is wrong ?