dealertrack / celery-redis-sentinel Goto Github PK

File "/lib/python3.5/site-packages/redis/client.py", line 2626, in execute
return execute(conn, stack, raise_on_error)
File "/lib/python3.5/site-packages/redis/client.py", line 2540, in _execute_transaction
self.raise_first_error(commands, response)
File "/lib/python3.5/site-packages/redis/client.py", line 2574, in raise_first_error
raise r
redis.exceptions.ResponseError: Command # 1 (SETEX b'celery-task-meta-84b7aefb-0487-45bf-875b-49cd5087e7e6' {"traceback": null, "status": "SUCCESS", "children": [], "task_id": "84b7aefb-0487-45bf-875b-49cd5087e7e6", "result": 11} 86400) of pipeline caused error: value is not an integer or out of range

exc, exc_info.traceback)))

Above exception is raised when I used this module to configure result backend. The SETEX looks a bit strange that timeout value is not at the 2nd position of the args.

If this module is not used, but use Celery 4.0.2 directly with Redis result backend. The SETEX looks like this:
[(('SETEX', b'celery-task-meta-a11b3664-320d-4b48-a25b-154c61ea9bc7', 86400, '{"task_id": "a11
b3664-320d-4b48-a25b-154c61ea9bc7", "traceback": null, "result": 11, "status": "SUCCESS", "children": []}'), {}), (('PUBLISH', b'celery-task-meta-a11b3664-320d-4b48-a25b-154c61ea9bc7', '{"task_id": "a11b3664-320d-4b48-a25b-154c61ea9bc7",
"traceback": null, "result": 11, "status": "SUCCESS", "children": []}'), {})]

Hi,
I'm trying to use celery-redis-sentinel in my django project, but I always get a :

DecodeError at /fr/
('string indices must be integers, not unicode', <function unpickle_backend at 0x7f4c62dac938>, (<class 'celery_redis_sentinel.backend.RedisSentinelBackend'>, ('redis-sentinel://localhost:7000',), {'expires': 86400}))

I have the following setup :

• 5 redis-servers (1 master and 4 slaves) on 127.0.01:7000-7005
• 4 sentinels on 127.0.0.1:9000-9003
• Celery broker is rabbitmq 3.6.9
• Celery backend is redis 4.0.1

class CeleryConfig:

    CELERY_RESULT_BACKEND = 'redis-sentinel://localhost:9000'
    CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {
        'sentinels': [('127.0.0.1', 9000),
                      ('127.0.0.1', 9001),
                      ('127.0.0.1', 9002)],
        'service_name': 'blookup',
        'socket_timeout': 0.1,
    }

celery_redis_sentinel.register()
celery_app = Celery('myapp', backend='redis-sentinel://127.0.0.1:9000')

The error seems to be on backend.py@35 :

 self.sentinels = self.transport_options['sentinels']

self.transport_options value is 'redis-sentinel://localhost:7000' instead of a dict.

Even thought redis sentinel is used, currently if locally redis is not running, celery will not start. It is just required to start celery and is actually not used when launched.

This is obviously not a desired behavior and should be fixed.

Configuration ->
. broker -> redis-sentinel://localhost:26379/0
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule --> this file to redis
. logfile -> [stderr]@%INFO

I can use redis-sentinel to message broker ok, but how can I use db to redis-sentinel

Hi，is there any plan to support for Python3.X?

Hi, I would like to use this package but I get the following error:

>>> from celery_redis_sentinel import register
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/celery_redis_sentinel/register.py", line 4, in <module>
    from celery.backends import BACKEND_ALIASES
ImportError: cannot import name BACKEND_ALIASES

I thought there was an issue with $PYTHONPATH but from celery import backends works fine. Do you have any advice?

I am running Mac OS X, Python 2.7.12. Thanks for taking a look!

Hi, when i use celery==4.0.2, it got an error:

      ......
      self._avail_channels.append(self.create_channel(self))
      File "/Users/simon/.venv/test/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 923, in create_channel
        channel = self.Channel(connection)
      File "/Users/simon/.venv/test/lib/python2.7/site-packages/kombu/transport/redis.py", line 501, in __init__
        self.client.ping()
      File "/Users/simon/.venv/test/lib/python2.7/site-packages/redis/client.py", line 682, in ping
        return self.execute_command('PING')
      File "/Users/simon/.venv/test/lib/python2.7/site-packages/redis/client.py", line 570, in execute_command
        connection = pool.get_connection(command_name, **options)
      File "/Users/simon/.venv/test/lib/python2.7/site-packages/redis/connection.py", line 898, in get_connection
        connection = self.make_connection()
      File "/Users/simon/.venv/test/lib/python2.7/site-packages/redis/connection.py", line 907, in make_connection
        return self.connection_class(**self.connection_kwargs)
    TypeError: __init__() got an unexpected keyword argument 'connection_pool'

And self.connection_class is <class 'redis.connection.Connection'>.

I'm running celery-redis-sentinel in a Kubernetes cluster, which means that if my Redis container goes down it often comes back up with a new IP. When this happens, celery seems to connect successfully then throws an error pointing to the original IP:

[2017-08-01 19:00:34,182: INFO/MainProcess] Connected to redis-sentinel://<new redis host>:6379/<db>
[2017-08-01 19:00:34,681: INFO/MainProcess] mingle: searching for neighbors
[2017-08-01 19:00:39,780: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 439, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 494, in _connect
    raise err
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 482, in _connect
    sock.connect(socket_address)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 572, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/usr/local/lib/python3.5/dist-packages/redis/sentinel.py", line 44, in connect
    self.connect_to(self.connection_pool.get_master_address())
  File "/usr/local/lib/python3.5/dist-packages/redis/sentinel.py", line 34, in connect_to
    super(SentinelManagedConnection, self).connect()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 113 connecting to <old redis host>:6379. No route to host.

Not sure why this behavior is occurring if Celery is aware of the new IP. Note that this same string of messages is repeated every few seconds.

Hi,

we've come across this project https://github.com/exponea/redis-sentinel-url#url-scheme-for-connecting-via-sentinel which uses the following syntax for redis with sentinel :

redis+sentinel://[:password@]host:port[,host2:port2,...][/service_name[/db]][?param1=value1[&param2=value=2&...]]

Any thoughts ? maybe we could converge towards that ? or suggest the redis-sentinel to that project ?

We are looking to use Redis Sentinels with Airflow / Celery. Is this plugin still maintained? If not, is there a blessed way to use Sentinels with Airflow / Celery?

I saw the last release was 2 years ago and the last commit was 1 year ago…so I just thought I'd ask. 😄