session failure while holding lock about kazoo

[bbangert]

If the session disconnects, then the nodes are lost and the lock is lost. A listener should be added with client.add_listener and listen for the LOST state to mark a variable you have access to indicating the session was lost (or just bailing the process entirely, which is recommended in the Zookeeper wiki docs for handling a LOST session).

It'd prolly be good to have the lock itself reg a listener so it could be aware of session lost and set a flag on itself indicating it was lost that can be checked. But depending on how much state, locks, etc. have been registered with Zookeeper, its generally easier to bail the process and start fresh than attempt to recover (thus the Zookeeper doc recommendation).

[nekto0n]

So recipe for now is to manually register a listener with hope that it will be implemented in future?

[bbangert]

Well, not "with hope", it will be added. :)

But I'd generally recommend following the Zookeeper guide docs and registering a listener that shuts down your program as gracefully as possible if the session state goes to LOST. If you're extremely familiar with Zookeeper and up for trying to re-establish all the state needed, backing up as appropriate to re-establish locks, etc, then you could try and recover yourself.

[nekto0n]

Session won't go to LOST until reconnected according to my tests and docs:

When a ZooKeeper client disconnects from a server, you will not receive notification of changes until reconnected.

So isolated/disconnected master will assume being master... creepy behaviour. All I get now is SUSPENDED right after disconnect.

[nekto0n]

Curator Lock suggests listening for both SUSPENDED and LOST and react accordingly.

[bbangert]

Yep, that applies with kazoo as well, im still in the process of writing up more of the general usage docs that will advise on these issues.