If a consumer panics, the library keeps trying to deliver future messages to it, but the consumer seems to have crashed or hung up, and it never receives the messages.
What is the expected behavior if a consumer panics? Should the consumer be removed entirely? Or should it continue to function normally?
It seems that panics are being caught somehow, since the app does not abort and exit. Maybe the Tokio runtime is catching it? The result is that we are somewhere in between aborting the app, and catching the panic, a place where the app will continue to run but the consumer just stops receiving any messages.
I have experienced web server frameworks like Axum and Actix Web, which catch and recover from panics in user handlers gracefully, so should we aim for similar behavior in amqprs?
In this test, we have a consumer app that subscribes to messages, and intentionally panics when the 4th message is received.
$ RUST_LOG=TRACE cargo run --bin consumer
...
// started
2023-03-07T20:58:34.592109Z TRACE amqprs::net::split_connection: 92 bytes read from network
2023-03-07T20:58:34.592169Z TRACE amqprs::net::split_connection: RECV on channel 1: Deliver(MethodHeader { class_id: 60, method_id: 60 }, Deliver { consumer_tag: ShortStr(16, "amqprs_reconnect"), delivery_tag: 1, redelivered: false, exchange: ShortStr(9, "amq.topic"), routing_key: ShortStr(0, "") })
2023-03-07T20:58:34.592198Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851418214405864 }
2023-03-07T20:58:34.592237Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentHeader(ContentHeader { common: ContentHeaderCommon { class: 60, weight: 0, body_size: 13 }, basic_properties: BasicProperties { property_flags: [0, 0], content_type: None, content_encoding: None, headers: None, delivery_mode: None, priority: None, correlation_id: None, reply_to: None, expiration: None, message_id: None, timestamp: None, message_type: None, user_id: None, app_id: None, cluster_id: None } })
2023-03-07T20:58:34.592442Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851418214648260 }
2023-03-07T20:58:34.592466Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentBody(ContentBody { inner: [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33] })
2023-03-07T20:58:34.592482Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851418214691050 }
2023-03-07T20:58:34.592561Z INFO consumer: consume delivery Deliver: consumer_tag = amqprs_reconnect, delivery_tag = 1, redelivered = false, exchange = amq.topic, routing_key = on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]', content size: 13
2023-03-07T20:58:34.592579Z INFO consumer: panic countdown: 3
2023-03-07T20:58:34.592587Z INFO consumer: ack to delivery Deliver: consumer_tag = amqprs_reconnect, delivery_tag = 1, redelivered = false, exchange = amq.topic, routing_key = on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]'
2023-03-07T20:58:34.592626Z TRACE amqprs::net::split_connection: SENT on channel 1: Ack(MethodHeader { class_id: 60, method_id: 80 }, Ack { delivery_tag: 1, mutiple: false })
2023-03-07T20:58:34.592675Z TRACE amqprs::net::writer_handler: connection 'AMQPRS000@localhost:13300/ [open]' heartbeat deadline is updated to Instant { t: 851388214882059 }
// ^^ sent 1 (ok)
...
// ^^ sent 2 (ok)
2023-03-07T20:58:48.353648Z TRACE amqprs::net::split_connection: 8 bytes read from network
2023-03-07T20:58:48.353712Z TRACE amqprs::net::split_connection: RECV on channel 0: HeartBeat(HeartBeat)
2023-03-07T20:58:48.353746Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851431976260929 }
2023-03-07T20:58:48.353777Z DEBUG amqprs::net::reader_handler: received heartbeat on connection 'AMQPRS000@localhost:13300/ [open]'
2023-03-07T20:58:53.282089Z TRACE amqprs::net::split_connection: 92 bytes read from network
2023-03-07T20:58:53.282138Z TRACE amqprs::net::split_connection: RECV on channel 1: Deliver(MethodHeader { class_id: 60, method_id: 60 }, Deliver { consumer_tag: ShortStr(16, "amqprs_reconnect"), delivery_tag: 3, redelivered: false, exchange: ShortStr(9, "amq.topic"), routing_key: ShortStr(0, "") })
2023-03-07T20:58:53.282166Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851436904794243 }
2023-03-07T20:58:53.282197Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentHeader(ContentHeader { common: ContentHeaderCommon { class: 60, weight: 0, body_size: 13 }, basic_properties: BasicProperties { property_flags: [0, 0], content_type: None, content_encoding: None, headers: None, delivery_mode: None, priority: None, correlation_id: None, reply_to: None, expiration: None, message_id: None, timestamp: None, message_type: None, user_id: None, app_id: None, cluster_id: None } })
2023-03-07T20:58:53.282273Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851436904900711 }
2023-03-07T20:58:53.282294Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentBody(ContentBody { inner: [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33] })
2023-03-07T20:58:53.282309Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851436904938418 }
2023-03-07T20:58:53.282373Z INFO consumer: consume delivery Deliver: consumer_tag = amqprs_reconnect, delivery_tag = 3, redelivered = false, exchange = amq.topic, routing_key = on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]', content size: 13
2023-03-07T20:58:53.282392Z INFO consumer: panic countdown: 1
2023-03-07T20:58:53.282400Z INFO consumer: ack to delivery Deliver: consumer_tag = amqprs_reconnect, delivery_tag = 3, redelivered = false, exchange = amq.topic, routing_key = on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]'
2023-03-07T20:58:53.282436Z TRACE amqprs::net::split_connection: SENT on channel 1: Ack(MethodHeader { class_id: 60, method_id: 80 }, Ack { delivery_tag: 3, mutiple: false })
2023-03-07T20:58:53.282479Z TRACE amqprs::net::writer_handler: connection 'AMQPRS000@localhost:13300/ [open]' heartbeat deadline is updated to Instant { t: 851406905106624 }
// ^^ sent 3 (ok)
// Next received message will trigger our intentional panic for testing:
2023-03-07T20:59:01.952081Z TRACE amqprs::net::split_connection: 92 bytes read from network
2023-03-07T20:59:01.952133Z TRACE amqprs::net::split_connection: RECV on channel 1: Deliver(MethodHeader { class_id: 60, method_id: 60 }, Deliver { consumer_tag: ShortStr(16, "amqprs_reconnect"), delivery_tag: 4, redelivered: false, exchange: ShortStr(9, "amq.topic"), routing_key: ShortStr(0, "") })
2023-03-07T20:59:01.952162Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851445574985087 }
2023-03-07T20:59:01.952193Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentHeader(ContentHeader { common: ContentHeaderCommon { class: 60, weight: 0, body_size: 13 }, basic_properties: BasicProperties { property_flags: [0, 0], content_type: None, content_encoding: None, headers: None, delivery_mode: None, priority: None, correlation_id: None, reply_to: None, expiration: None, message_id: None, timestamp: None, message_type: None, user_id: None, app_id: None, cluster_id: None } })
2023-03-07T20:59:01.952468Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851445575289391 }
2023-03-07T20:59:01.952494Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentBody(ContentBody { inner: [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33] })
2023-03-07T20:59:01.952510Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851445575334271 }
2023-03-07T20:59:01.952575Z INFO consumer: consume delivery Deliver: consumer_tag = amqprs_reconnect, delivery_tag = 4, redelivered = false, exchange = amq.topic, routing_key = on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]', content size: 13
2023-03-07T20:59:01.952597Z INFO consumer: panic time!
thread 'tokio-runtime-worker' panicked at 'testing consumer handling of panics', src/bin/consumer.rs:229:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
// ^^ panicked
2023-03-07T20:59:20.136793Z TRACE amqprs::net::split_connection: 92 bytes read from network
2023-03-07T20:59:20.136853Z TRACE amqprs::net::split_connection: RECV on channel 1: Deliver(MethodHeader { class_id: 60, method_id: 60 }, Deliver { consumer_tag: ShortStr(16, "amqprs_reconnect"), delivery_tag: 5, redelivered: false, exchange: ShortStr(9, "amq.topic"), routing_key: ShortStr(0, "") })
2023-03-07T20:59:20.136892Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851463760119248 }
2023-03-07T20:59:20.136937Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentHeader(ContentHeader { common: ContentHeaderCommon { class: 60, weight: 0, body_size: 13 }, basic_properties: BasicProperties { property_flags: [0, 0], content_type: None, content_encoding: None, headers: None, delivery_mode: None, priority: None, correlation_id: None, reply_to: None, expiration: None, message_id: None, timestamp: None, message_type: None, user_id: None, app_id: None, cluster_id: None } })
2023-03-07T20:59:20.137036Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851463760264706 }
2023-03-07T20:59:20.137071Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentBody(ContentBody { inner: [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33] })
2023-03-07T20:59:20.137090Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851463760323153 }
2023-03-07T20:59:20.137159Z ERROR amqprs::api::channel::dispatcher: failed to dispatch message to consumer amqprs_reconnect on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]'
2023-03-07T20:59:23.283800Z TRACE amqprs::net::split_connection: SENT on channel 0: HeartBeat(HeartBeat)
2023-03-07T20:59:23.283980Z DEBUG amqprs::net::writer_handler: sent heartbeat over connection 'AMQPRS000@localhost:13300/ [open]'
// ^^ sent 4 (failed to dispatch)
2023-03-07T20:59:29.469691Z TRACE amqprs::net::split_connection: 92 bytes read from network
2023-03-07T20:59:29.469741Z TRACE amqprs::net::split_connection: RECV on channel 1: Deliver(MethodHeader { class_id: 60, method_id: 60 }, Deliver { consumer_tag: ShortStr(16, "amqprs_reconnect"), delivery_tag: 6, redelivered: false, exchange: ShortStr(9, "amq.topic"), routing_key: ShortStr(0, "") })
2023-03-07T20:59:29.469769Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851473093210483 }
2023-03-07T20:59:29.469800Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentHeader(ContentHeader { common: ContentHeaderCommon { class: 60, weight: 0, body_size: 13 }, basic_properties: BasicProperties { property_flags: [0, 0], content_type: None, content_encoding: None, headers: None, delivery_mode: None, priority: None, correlation_id: None, reply_to: None, expiration: None, message_id: None, timestamp: None, message_type: None, user_id: None, app_id: None, cluster_id: None } })
2023-03-07T20:59:29.469870Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851473093311935 }
2023-03-07T20:59:29.469888Z TRACE amqprs::net::split_connection: RECV on channel 1: ContentBody(ContentBody { inner: [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33] })
2023-03-07T20:59:29.469903Z TRACE amqprs::net::reader_handler: server heartbeat deadline is updated to Instant { t: 851473093346081 }
2023-03-07T20:59:29.469953Z ERROR amqprs::api::channel::dispatcher: failed to dispatch message to consumer amqprs_reconnect on channel 1 [open] of connection 'AMQPRS000@localhost:13300/ [open]'
// ^^ sent 5 (failed to dispatch)