Consumers crash if they panic about amqprs

[gftea]

I see, now.
For current library, I think there are different ways

1. as you mentioned, user may have to use catch_unwind to do that when using basic_consume and basic_consume_blocking which spawn consumer task internally
2. instead, there is basic_consume_rx which return the consumer channel receiver, then you can spawn your own task, and get a JoinHandle for the task, and you can use the handle to check if panic in that task.

possible improvement, in amqprs library,

1. we provide api for user to get the consumer task handle, so user can check if task panic and do basic_cancel when panic
2. we implicitly cancel consumer if the consumer task panic

I do not think alt.2 is good idea due to implicity and hide the fault to be detected later phase, I prefer alt.1 so that user knows they are dealing with possible panic of 3rd party code.

[gftea]

Hi,

I think the consumer implementation is on user, so if it is reponsibility of user to cancel the consumer by using basic_cancel

[cdbennett]

Where would one all basic_cancel() from? The panic happens in some code called by our application's consumer handler, and then the consumer just vanishes.

It seems like, as a user, if I want my application to be robust, I would have to wrap the entire body of my consumer handler method with a std::panic::catch_unwind() call.

And then after catching the panic, either log and ignore the panic, abort the application, cancel and recreate a new consumer, etc. Does this sound right?

[gftea]

What did you meant consumer handlers? User implements consumer trait, where they can do error handling explicitly in their application logic , when there is error, they can decide to call basic cancel on current channel.

What use cases we cannot handle?

[cdbennett]

Here's an example. In my amqprs consumer implementation, to handle a received message, it calls out to code that does network communications and data parsing. In that parsing it could cause a panic by trying to access a Vec with an index out of range. This results in bad behavior of the app because the consumer is sort of a "zombie".

I am still new to Rust, but it seems like the only way to make this robust from a user perspective is to use std::panic::catch_unwind() around the entire body of the consumer handler, in case there are bugs that could cause a panic. Does that make sense?

[cdbennett]

If you have potential library improvements that could help detect a consumer that was killed by a panic, and clean up or recover, that would be a welcome change, it seems like catching the panic directly is not so easy.

I tried to use std::panic::catch_unwind() in my consumer but it's not working since the consumer and the other library code that the consumer handler calls is also async.

Looks like catching panics in async code is not easy. E.g. this StackOverflow question suggests FuturesExt as a solution but it has shortcomings.

[gftea]

can you try use basic_consume_rx to see if it works in your case, otherwise it would be good that you provide a minimal example so that I can explore it

[cdbennett]

I will look into basic_consume_rx more at some point. It seems like it would allow managing a worker task that could be restarted if it panics, or else as you say, the consumer could at least be cleaned up.

For now, I do have a test case.

Consumer test program is at https://github.com/cdbennett/amqprs-reconnect-example/blob/main/src/bin/consumer.rs

To do the test, run the consumer with cargo run --bin consumer and then in another terminal run the sending cargo run --bin sender several times. The 3rd time the consumer will intentionally panic, and then later messages will not be received.

[gftea]

tokio support catching panic from task, this is working example for me

    let args = BasicConsumeArguments::new(&queue_name, "panic_consumer");

    let (ctag, mut rx) = channel.basic_consume_rx(args).await.unwrap();

    let jh = tokio::spawn(async move {
        while let Some(msg) = rx.recv().await {
            if "panic" == std::str::from_utf8(&msg.content.unwrap()).unwrap() {
                panic!("panic consumer");
            }
        }
    });
    //  panic should be caught
    assert!(jh.await.is_err());
    // cancel consumer
    channel.basic_cancel(BasicCancelArguments::new(&ctag)).await.unwrap();

[gftea]

current api can be improved, but it will be breaking changes, and we need to plan it in v2.0.0
so at present, I recommend you use above example to handle panic

[gftea]

@cdbennett I would close this is no more inquiry

[cdbennett]

Sounds good. I didn't have a chance yet to try out your example of catching panic from task. Thanks for your help.