When starting, takes a long time for initial beat start. Hangs at "Acquiring lock..." about redbeat

[mbarszcz]

I think for critical applications reducing the timeout is a must anyway -- because even if celery was cleaning up the lock, the instance may still go down ungracefully and the lock won't be released fast enough.

[sibson]

I believe upstream celery should be setting up the proper signal handlers and then calling the close() method. If that is all working properly then I don't think there is much redbeat can do. Reducing the loop interval would make the CPU and Redis busier, which may or may not be a concern. You could also adjust the redbeat_lock_timeout. Changing the lock timeout either way is a workaround rather than a proper solution.

[sibson]

I suspect there is an old lock that needs to be cleaned up, this PR seems like it might be related, https://github.com/sibson/redbeat/pull/84/files

[metachris]

Thanks for the hint. What do you think of this PR? Not sure how i'd be able to get this into a production setup without having it be part of a PyPI package (would prefer to avoid pulling code from Github and using a development branch to provision the production setup).

But actually, i have the up to 25 minute delay before it says beat: Extending lock..., so I'm not sure that this PR could impact that, because it only takes over at that point, at which the delay already happened if i understand correctly.

[sibson]

hmm, I think you need is a non-blocking call to aquire, additional logging and retry on failure in https://github.com/sibson/redbeat/blob/master/redbeat/schedulers.py#L419-L426. The quick answer would be to redis-cli into the server and inspect the lock values to confirm it hasn't been cleaned up. The other piece to confirm is the cleanup code is being run when you shutdown the old version before deploy, https://github.com/sibson/redbeat/blob/master/redbeat/schedulers.py#L390-L393

[metachris]

Yeah I'll check if the lock is set. I'm using Beanstalk with a multi-docker setup, not sure if Beanstalk does a graceful shutdown or just kills the celerybeat docker container. Seems to me it just kills it and the lock would be left over, which would be consistent with the delays of around 25m i've been seeing.

[metachris]

Just confirmed that Elastic Beanstalk is not doing a graceful shutdown and the lock is not released when the instance is removed.

[metachris]

What do you think could be approaches for redbeat not being shutdown gracefully, and having the lock left over?

[metachris]

Would there be any downsides to reducing beat_max_loop_interval to something like 20 seconds, so the default lock period would be 100 seconds?

[metachris]

Update: even if I open a shell into the Docker container and kill the celerybeat process, it does not do a graceful shutdown, still leaves the lock in redis. btw, using latest redbeat 0.11.1