Prepared statements bug. about cdrs-tokio

[krojew]

Can you enable debug logging by setting env: RUST_LOG=debug and send the output? This error should be handled and the query re-prepared.

[npatsakula]

Sort of. I can't send you logs as is (production logs under NDA and I don't have enough time for MRE), but log types is:

DEBUG cdrs_tokio::cluster::control_connection Establishing new control connection...
DEBUG cdrs_tokio::cluster::connection_pool Establishing new connection...
DEBUG cdrs_tokio::cluster::metadata_builder Copying contact point. node_info= NodeInfo { host_id: id, broadcast_rpc_address: host:port, broadcast_address: Some(host:port), datacenter: "DC0", tokens: [Murmur3Token { value: token }, ...], rack: "R0" }
+ error that a mentioned before

[krojew]

There is no "Re-preparing statement." debug log?

[npatsakula]

Nope:(

[krojew]

Can you share limited code where you make a query and it fails?

[npatsakula]

Something like this:

let query = session
        .prepare("SELECT * FROM some_table WHERE value = (?) ;")
        .await
        .unwrap();

let query_values = query_values!("value" => value as u64)

let data = session
        .exec_with_values(&query, query_values)
        .await
        .unwrap()

[npatsakula]

It's look super odd because of all machinery around 0x2500 checks: we should see Re-preparing statement. log even if it's fail on second send_frame..

I spent more time playing with logs and comment about results

[krojew]

You can try adding dbg!(&result); in sessions.rs:214 and see what the actual error is.

[krojew]

Any luck capturing that debug dump of the error?

[npatsakula]

Sorry, now I don't have time for this:( I can get back to work on it on next week.

[krojew]

No problem.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cibingeorge]

I ran into similar issues. Looks like the ServerError gets converted to general error somewhere. Let me see if i can find out why.

2022-07-08T14:27:27.671377Z DEBUG cdrs_tokio::cluster::connection_pool: Establishing new connection...
2022-07-08T14:27:27.674190Z ERROR cdrs_tokio::transport: Transport error! error=Server error: ErrorBody { error_code: 9472, message: "No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }
2022-07-08T14:27:27.674227Z DEBUG cdrs_tokio::cluster::session: Result from exec_with_params=Err(General("Server error: ErrorBody { error_code: 9472, message: \"No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.\", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }"))  <-- This is my debug log before error check.

[cibingeorge]

I was testing on the sha just before the protocol v5 implementation.

The issue in my case is caused by

cdrs-tokio/cdrs-tokio/src/transport.rs

Line 298 in e1de5ff

response_handler_map.signal_general_error(&error.to_string());

which converts server error into a general error. So the reprepare doesnt happen.

I am not sure why the error here need to be converted to general error. I patched up to not convert Server errors, and the prepares work as expected.

diff --git a/cdrs-tokio/src/transport.rs b/cdrs-tokio/src/transport.rs
index 3b96ce8..c2314b5 100644
--- a/cdrs-tokio/src/transport.rs
+++ b/cdrs-tokio/src/transport.rs
@@ -295,7 +295,7 @@ impl AsyncTransport {
             error!(%error, "Transport error!");
 
             is_broken.store(true, Ordering::Relaxed);
-            response_handler_map.signal_general_error(&error.to_string());
+            response_handler_map.signal_general_error(&error);
 
             if let Some(error_handler) = error_handler {
                 let _ = error_handler.send(error).await;
@@ -419,9 +419,16 @@ impl ResponseHandlerMap {
         }
     }
 
-    pub fn signal_general_error(&self, error: &str) {
+    pub fn signal_general_error(&self, error: &Error) {
         for (_, handler) in self.stream_handlers.lock().unwrap().drain() {
-            let _ = handler.send(Err(Error::General(error.to_string())));
+            match error {
+                Error::Server(err) => {
+                    let _ = handler.send(Err(Error::Server(err.to_owned())));
+                },
+                _ => {
+                    let _ = handler.send(Err(Error::General(error.to_string())));
+                }
+            }
         }
     }

I am not very familiar with the code. So not sure if this is the right fix. You could jst return the error as it is.
@krojew What do you think?

[krojew]

Thanks for debugging the issue! You are right - there's no point in converting the error. I'll add a fix next Monday.

[cibingeorge]

I found another bug in reprepare. The reprepare runs on the node which is different from the original node which errored out. So reprepare works but the subsequent query fails with the same error. I added some extra logging in the code to trace the ip where the request is made.

2022-07-10T12:38:48.382011Z DEBUG actix::model::cache_request: fetching product worker=470 partition=e512291d-584e-4121-bfe6-fc71faaccc50 id=9780140186772
2022-07-10T12:38:48.382945Z DEBUG cdrs_tokio::cluster::session: query_plan=[Node { broadcast_rpc_address: 172.31.131.15:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(170052ac-e12c-4810-a5bd-383a7a1c7adb), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.215.156:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ae6f113b-13d7-4d88-9d01-945dcae599b8), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.163.168:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(27d712e1-b9fe-457c-94b0-85d030856951), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.214.238:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(aa8e4162-d781-431c-a87d-f6dc4bf0f8c9), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.59.217:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(1058887f-e279-4ecb-be8f-3528bd762595), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.191.223:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d79cfc80-901a-421b-ad38-2b81107920bf), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.151.198:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(0b2c3343-234a-4f40-b0b0-3f4868711c08), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.145.159:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ffa5e97a-3457-498a-bca2-d9d638cce641), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.180.16:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ef2c4c8b-6aa7-4c17-82f9-07df533c9333), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.173.243:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(6fd93f53-9529-4249-abc4-bc9cc0e04aff), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.216.173:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d9959417-7382-4f56-b985-0ca94e6bf88f), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.209.57:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(4c2f2a6b-c721-4693-a0dd-5618d61b1ade), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.89:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(cfd6fc36-d7b1-425b-b2b4-a2abf495a556), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.199.50:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(80baae93-489e-403c-bfa7-e82721334bc3), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.56.242:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d119ebae-b609-4774-8154-4c5d616f1565), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.223.11:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(7bf4a903-7971-4e8a-9a68-9cd5bb1072cb), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.135:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(46c3b830-381a-4edd-96f0-00b32f0519fe), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.154.21:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(8fdf28bb-1626-4a54-9f53-d0a769f9916b), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.56.128:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(e6b4af0d-e4a4-4e9a-b5ff-f8e0f3ca9409), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.156.241:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(f5df495d-25b2-4cb0-8698-c9e58ab268a4), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.61.130:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(8c6083b0-1970-42fa-b6b0-096f8ce6d2d0), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.211:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(7e4887e7-58c7-4612-a26e-e62a79e6d055), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.61.126:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(176d521c-032c-47f3-8521-18a74567aace), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.151.166:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(feb49464-a20d-4640-9d67-a9622b671139), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.158.174:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(c461b54c-5306-4d28-9bcc-315a125f2c61), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.165.175:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(fe47db22-0e4a-4343-81f6-fb20f10cf5df), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.203.205:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(32feb7c4-3232-4a19-8bb4-99a477f3b8c3), rack: "2c", datacenter: "us-west-2" }]
2022-07-10T12:38:48.382996Z DEBUG cdrs_tokio::cluster::connection_pool: Establishing new connection to 172.31.131.15:9042
2022-07-10T12:38:48.383748Z DEBUG cdrs_tokio::transport: Transport.write_frame to node=172.31.131.15:9042
2022-07-10T12:38:48.384895Z DEBUG cdrs_tokio::transport: Transport.write_frame to node=172.31.131.15:9042
2022-07-10T12:38:48.385771Z ERROR cdrs_tokio::transport: Transport error! error=Server error: ErrorBody { error_code: 9472, message: "No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }
2022-07-10T12:38:48.385804Z DEBUG cdrs_tokio::cluster::session: Result from exec_with_params=Err(Server(ErrorBody { error_code: 9472, message: "No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }))
2022-07-10T12:38:48.385812Z DEBUG cdrs_tokio::cluster::session: Error from exec_with_params error=ErrorBody { error_code: 9472, message: "No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }
2022-07-10T12:38:48.385825Z DEBUG cdrs_tokio::cluster::session: Re-preparing statement.
2022-07-10T12:38:48.385833Z DEBUG cdrs_tokio::cluster::session: query_plan=[Node { broadcast_rpc_address: 172.31.59.217:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(1058887f-e279-4ecb-be8f-3528bd762595), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.215.156:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ae6f113b-13d7-4d88-9d01-945dcae599b8), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.191.223:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d79cfc80-901a-421b-ad38-2b81107920bf), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.151.198:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(0b2c3343-234a-4f40-b0b0-3f4868711c08), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.145.159:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ffa5e97a-3457-498a-bca2-d9d638cce641), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.180.16:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ef2c4c8b-6aa7-4c17-82f9-07df533c9333), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.173.243:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(6fd93f53-9529-4249-abc4-bc9cc0e04aff), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.216.173:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d9959417-7382-4f56-b985-0ca94e6bf88f), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.209.57:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(4c2f2a6b-c721-4693-a0dd-5618d61b1ade), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.89:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(cfd6fc36-d7b1-425b-b2b4-a2abf495a556), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.199.50:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(80baae93-489e-403c-bfa7-e82721334bc3), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.56.242:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d119ebae-b609-4774-8154-4c5d616f1565), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.223.11:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(7bf4a903-7971-4e8a-9a68-9cd5bb1072cb), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.135:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(46c3b830-381a-4edd-96f0-00b32f0519fe), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.154.21:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(8fdf28bb-1626-4a54-9f53-d0a769f9916b), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.163.168:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(27d712e1-b9fe-457c-94b0-85d030856951), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.131.15:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(170052ac-e12c-4810-a5bd-383a7a1c7adb), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.56.128:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(e6b4af0d-e4a4-4e9a-b5ff-f8e0f3ca9409), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.156.241:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(f5df495d-25b2-4cb0-8698-c9e58ab268a4), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.61.130:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(8c6083b0-1970-42fa-b6b0-096f8ce6d2d0), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.211:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(7e4887e7-58c7-4612-a26e-e62a79e6d055), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.61.126:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(176d521c-032c-47f3-8521-18a74567aace), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.151.166:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(feb49464-a20d-4640-9d67-a9622b671139), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.158.174:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(c461b54c-5306-4d28-9bcc-315a125f2c61), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.165.175:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(fe47db22-0e4a-4343-81f6-fb20f10cf5df), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.203.205:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(32feb7c4-3232-4a19-8bb4-99a477f3b8c3), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.214.238:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(aa8e4162-d781-431c-a87d-f6dc4bf0f8c9), rack: "2c", datacenter: "us-west-2" }]
2022-07-10T12:38:48.385868Z DEBUG cdrs_tokio::transport: Transport.write_frame to node=172.31.59.217:9042
2022-07-10T12:38:48.386276Z DEBUG cdrs_tokio::cluster::session: query_plan=[Node { broadcast_rpc_address: 172.31.131.15:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(170052ac-e12c-4810-a5bd-383a7a1c7adb), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.215.156:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ae6f113b-13d7-4d88-9d01-945dcae599b8), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.163.168:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(27d712e1-b9fe-457c-94b0-85d030856951), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.191.223:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d79cfc80-901a-421b-ad38-2b81107920bf), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.151.198:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(0b2c3343-234a-4f40-b0b0-3f4868711c08), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.145.159:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ffa5e97a-3457-498a-bca2-d9d638cce641), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.180.16:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(ef2c4c8b-6aa7-4c17-82f9-07df533c9333), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.173.243:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(6fd93f53-9529-4249-abc4-bc9cc0e04aff), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.216.173:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d9959417-7382-4f56-b985-0ca94e6bf88f), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.209.57:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(4c2f2a6b-c721-4693-a0dd-5618d61b1ade), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.89:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(cfd6fc36-d7b1-425b-b2b4-a2abf495a556), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.199.50:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(80baae93-489e-403c-bfa7-e82721334bc3), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.56.242:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(d119ebae-b609-4774-8154-4c5d616f1565), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.223.11:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(7bf4a903-7971-4e8a-9a68-9cd5bb1072cb), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.135:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(46c3b830-381a-4edd-96f0-00b32f0519fe), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.154.21:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(8fdf28bb-1626-4a54-9f53-d0a769f9916b), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.56.128:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(e6b4af0d-e4a4-4e9a-b5ff-f8e0f3ca9409), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.156.241:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(f5df495d-25b2-4cb0-8698-c9e58ab268a4), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.61.130:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(8c6083b0-1970-42fa-b6b0-096f8ce6d2d0), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.58.211:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(7e4887e7-58c7-4612-a26e-e62a79e6d055), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.61.126:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(176d521c-032c-47f3-8521-18a74567aace), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.151.166:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(feb49464-a20d-4640-9d67-a9622b671139), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.158.174:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(c461b54c-5306-4d28-9bcc-315a125f2c61), rack: "2a", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.165.175:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(fe47db22-0e4a-4343-81f6-fb20f10cf5df), rack: "2b", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.203.205:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(32feb7c4-3232-4a19-8bb4-99a477f3b8c3), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.214.238:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(aa8e4162-d781-431c-a87d-f6dc4bf0f8c9), rack: "2c", datacenter: "us-west-2" }, Node { broadcast_rpc_address: 172.31.59.217:9042, broadcast_address: None, distance: Some(Local), state: Atomic(Up), host_id: Some(1058887f-e279-4ecb-be8f-3528bd762595), rack: "2b", datacenter: "us-west-2" }]
2022-07-10T12:38:48.386311Z DEBUG cdrs_tokio::cluster::connection_pool: Establishing new connection to 172.31.131.15:9042
2022-07-10T12:38:48.387050Z DEBUG cdrs_tokio::transport: Transport.write_frame to node=172.31.131.15:9042
2022-07-10T12:38:48.388190Z DEBUG cdrs_tokio::transport: Transport.write_frame to node=172.31.131.15:9042
2022-07-10T12:38:48.389026Z ERROR cdrs_tokio::transport: Transport error! error=Server error: ErrorBody { error_code: 9472, message: "No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }
2022-07-10T12:38:48.389159Z ERROR HTTP{request_id=1e7abe45-0700-4606-b46f-73d13420c8fc http.method=PUT  http.scheme=http otel.kind="server" exception.message=cdrs_tokio: error fetching data http.status_code=500 otel.status_code="ERROR"}: tracing_actix_web::middleware: Error encountered while processing the incoming HTTP request: CassandraDataFetchError(Server(ErrorBody { error_code: 9472, message: "No prepared statement with ID 722aa1fba791034d7f81d51eb49a02e9 found.", additional_info: Unprepared(UnpreparedError { id: CBytesShort { bytes: Some([114, 42, 161, 251, 167, 145, 3, 77, 127, 129, 213, 30, 180, 154, 2, 233]) } }) }))

Because of the retry logic its complicated to rerun the prepare statement on the right node. The only thing i can think of is to pass in the Prepared query info into the

cdrs-tokio/cdrs-tokio/src/cluster/send_frame.rs

Line 13 in e1de5ff

pub async fn send_frame<T: CdrsTransport + 'static, CM: ConnectionManager<T> + 'static>(

and retry there.

[DevNulPavel]

Hello :-)

I'm using cdrs-tokio = "^6.2", but still receiving error:

Prepared query with ID 37ef1cdb7c0142dc94ca37812bae409e not found (either the query was not prepared on this host (maybe the host has been restarted?) or you have prepared too many queries and it has been evicted from the internal cache)

It looks like problem still exists.

It be be fixed in the next version, am i right?

[krojew]

Sorry, I've been focused on the next version with protocol v5. Since beta 1 is out, I can backport some fixes now.

[krojew]

@DevNulPavel unfortunately, this fix needs cassandra-protocol crate in version 2.0, so can't be backported to cdrs-tokio 6. The changes made by Cassandra when introducing v5 make this difficult to port without introducing artificial major version updates.

[DevNulPavel]

@krojew Thank you for answer 👍😊