Poll Result error causes celeryd to crash about billiard

[ask]

The exception that is raised is not pickleable:

>>> from requests.packages.urllib3.exceptions import MaxRetryError
>>> x = MaxRetryError(None, "foo")
>>> import pickle
>>> pickle.loads(pickle.dumps(x))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
TypeError: __init__() takes exactly 3 arguments (2 given)

Celery have some elaborate mechanisms to deal with these non-standard exceptions though,
and it works fine for me:

@task
def unpick():
    from requests.packages.urllib3.exceptions import MaxRetryError
    raise MaxRetryError(None, "http://foo")

Does that not work for you? If it does, then at what point does it raise this exception?

[myusuf3]

I can confirm by upgrading to django-celery 3.0.11 and billard 2.7.3.19 that comes with it; stops the crashing of celery when a Exception that cannot be pickled is returned.

[ask]

Closing

[pjwerneck]

I'm still having the exact same problem with celery 3.0.18 and billard 2.7.3.19:

[2013-04-15 20:23:59,174: ERROR/MainProcess] Unrecoverable error: TypeError('__init__() takes exactly 3 arguments (2 given)', <class 'requests.packages.urllib3.exceptions.MaxRetryError'>, ("HTTPConnectionPool(hos
t='xxx', port=80): Max retries exceeded with url: xxx,))
Traceback (most recent call last):
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/celery/worker/__init__.py", line 347, in start
    component.start()
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/celery/worker/consumer.py", line 390, in start
    self.consume_messages()
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/celery/worker/consumer.py", line 474, in consume_messages
    readers[fileno](fileno, event)
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/billiard/pool.py", line 672, in handle_event
    self._it.next()
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/billiard/pool.py", line 645, in _process_result
    ready, task = poll(timeout)
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/billiard/pool.py", line 1062, in _poll_result
    return True, self._quick_get()
TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'requests.packages.urllib3.exceptions.MaxRetryError'>, ("HTTPConnectionPool(host='x', port=80): Max retries exceeded with url: x",))