Exception during manual topology recovery about langohr

[michaelklishin]

As you can see Langohr does recover your connection but then something goes wrong when it recovers you topology (queue, bindings, etc). I can't say what it may be without seeing your code. You can disable topology recovery and use recovery hooks (callbacks), there should be examples in the docs or the examples repo.

RabbitMQ restart is not really different from a network issue except that RabbitMQ notifies clients first with a connection.close.

[eigenhombre]

The code is posted in the Gist mentioned above: https://gist.github.com/msszczep/11266360 -- it's basically the same as your example, with the publish removed and a few queue parameters changed.

Can you suggest something else to try?

Thanks!

[michaelklishin]

@eigenhombre what's in RabbitMQ log? have you tried inspecting the cause exception? Have you tried adding a recovery test that does what you want?

[michaelklishin]

Besides getting to the root cause, I recommend trying automatic recovery mode (for topology). Why did you choose to recover queues and consumers manually?

[michaelklishin]

I cannot reproduce your issue with both a test and example1 and manual RabbitMQ restarts. You have something in your system
that prevents queue.declare from succeeding.

Looking at the code, there are commented out
options to queue.declare. Most likely you ran this example multiple times and now queue
declaration fails because existing queue has different attributes (406 Precondition Failed).
Inspecting RabbitMQ log should easily confirm or deny this hypothesis. If that's the case, delete the queue and try again.

[eigenhombre]

Hi @michaelklishin , thanks for looking into this.

Rather than chase the differences between your example and ours, I just ran your example1 from langohr.examples. It fails as well, intermittently with the same sort of exception. The RMQ logs from the server are in this gist. RMQ version on server is 3.1.5.

It seems that something is not quite reliable with the restart logic. Can I help you reproduce this on your end in any way? You might have to beat on it for awhile (sudo service rabbitmq-server restart several times).

Thanks!

[michaelklishin]

@eigenhombre I can't (I don't use upstart or Linux but it should not matter). Catch the exception in your recovery handler and run

(.. e getCause printStackTrace)
(println (.getCause e))

Again, have you tried automatic topology recovery? Why do you want to do it manually?

[michaelklishin]

One guess I have is that RabbitMQ sends clients a connection.close, it triggers a reconnection but then RabbitMQ server itself is restarted with a delay and by the time the client reconnects, the server shuts down, so you get a socket exception in connection negotiation. Although if RabbitMQ is not available on reconnection attempts, Langohr will retry indefinitely, and this case should be covered, too.

Try stopping RabbitMQ and starting it in 15 seconds instead of doing a restart.

[michaelklishin]

You can also bump :network-recovery-delay to 10000 ms or so. It's unlikely that restarting Rabbit takes ~ 5 seconds but if you have a lot of queues it might.

[eigenhombre]

Hi @michaelklishin , I tried doing the stop ... wait ... start combo you suggested, instead of service rabbitmq-server restart and saw the same behavior.

Regarding have you tried automatic topology recovery? Why do you want to do it manually? -- I was just following your example code. If I change to :automatically-recover-topology true, the exception still occurs.

Regarding :network-recovery-delay 10000 - that does seem to help survive the restart, but if rabbitmq is down > 10 seconds, the same stacktrace is thrown.

Note that my production code only fails this way on my laptop setup - Mac for the Clojure code, Vagrant VM for RabbitMQ. We have not seen this fail in production (yet).

Cheers!

[michaelklishin]

@eigenhombre then it's certainly something in your local environment, most likely different queue attributes. RabbitMQ log can give you some insights.