test_mv_resurrected_rows_after_decommission_interrupt[with_rbno] fails in debug mode about scylladb

[bhalevy]

@kbr-scylla this happened with consistent-topology-changes (and rbno).
Can you please look into this?

[kbr-scylla]

Nothing unexpected happened here. Decommission process was interrupted too late, at the end:

INFO  2024-06-16 06:39:01,130 [shard 0:strm] raft_topology - streaming completed

the topology coordinator has reached the point of no return -- after streaming is completed we enter a state from which the only way is forward, i.e. finishing the decommission, which causes the node to be banned.

INFO  2024-06-16 06:39:01,134 [shard 0: gms] raft_topology - updating topology state: decommissioning: streaming completed for node 8d337728-7ec3-4376-8363-df83207c3a9b
ERROR 2024-06-16 06:39:01,302 [shard 0:main] rpc - client 127.0.7.4:7000: client connection dropped: recv: Connection reset by peer
ERROR 2024-06-16 06:39:01,303 [shard 0: gms] rpc - client 127.0.7.4:7000: client connection dropped: recv: Connection reset by peer
ERROR 2024-06-16 06:39:01,303 [shard 0: gms] rpc - client 127.0.7.4:7000: client connection dropped: recv: Connection reset by peer
INFO  2024-06-16 06:39:01,336 [shard 0: gms] raft_topology - entered `write both read new` transition state
INFO  2024-06-16 06:39:01,336 [shard 0: gms] raft_topology - executing global topology command barrier_and_drain, excluded nodes: {}
ERROR 2024-06-16 06:39:01,390 [shard 0: gms] raft_topology - drain rpc failed, proceed to fence old writes: std::runtime_error (raft topology: exec_global_command(barrier_and_drain) failed with seastar::rpc::closed_error (connection is closed))
INFO  2024-06-16 06:39:01,393 [shard 0: gms] raft_topology - updating topology state: advance fence version to 12
INFO  2024-06-16 06:39:01,587 [shard 0: gms] raft_topology - executing global topology command barrier, excluded nodes: {}
ERROR 2024-06-16 06:39:01,646 [shard 0: gms] raft_topology - transition_state::write_both_read_new, global_token_metadata_barrier failed, error std::runtime_error (raft topology: exec_global_command(barrier) failed with seastar::rpc::closed_error (connection is closed))
WARN  2024-06-16 06:39:02,279 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
INFO  2024-06-16 06:39:03,019 [shard 0:main] raft_group_registry - marking Raft server 8d337728-7ec3-4376-8363-df83207c3a9b as dead for raft groups
WARN  2024-06-16 06:39:04,281 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:06,282 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:08,283 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:10,284 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
INFO  2024-06-16 06:39:11,650 [shard 0: gms] raft_topology - updating topology state: decommissioning: read fence completed
INFO  2024-06-16 06:39:11,766 [shard 0: gms] raft_topology - entered `left token ring` transition state
INFO  2024-06-16 06:39:11,766 [shard 0: gms] raft_topology - executing global topology command barrier_and_drain, excluded nodes: {}
ERROR 2024-06-16 06:39:11,804 [shard 0: gms] raft_topology - drain rpc failed, proceed to fence old writes: std::runtime_error (raft topology: exec_global_command(barrier_and_drain) failed with seastar::rpc::closed_error (connection is closed))
INFO  2024-06-16 06:39:11,807 [shard 0: gms] raft_topology - updating topology state: advance fence version to 13
INFO  2024-06-16 06:39:11,919 [shard 0: gms] raft_topology - executing global topology command barrier, excluded nodes: {}
ERROR 2024-06-16 06:39:11,986 [shard 0: gms] raft_topology - transition_state::left_token_ring, raft_topology_cmd::command::barrier failed, error std::runtime_error (raft topology: exec_global_command(barrier) failed with seastar::rpc::closed_error (connection is closed))
WARN  2024-06-16 06:39:12,286 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:14,287 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:16,288 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:18,289 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
WARN  2024-06-16 06:39:20,290 [shard 0:strm] gossip - failure_detector_loop: Send echo to node 127.0.7.4, status = failed: seastar::rpc::closed_error (connection is closed)
INFO  2024-06-16 06:39:21,990 [shard 0: gms] raft_topology - updating topology state: report request completion in left_token_ring state
WARN  2024-06-16 06:39:22,098 [shard 0: gms] raft_topology - failed to tell node 8d337728-7ec3-4376-8363-df83207c3a9b to shut down - it may hang. It's safe to shut it down manually now. (Exception: seastar::rpc::closed_error (connection is closed))
INFO  2024-06-16 06:39:22,103 [shard 0: gms] raft_topology - removing node 8d337728-7ec3-4376-8363-df83207c3a9b from group 0 configuration...
INFO  2024-06-16 06:39:22,112 [shard 0: gms] raft_topology - node 8d337728-7ec3-4376-8363-df83207c3a9b removed from group 0 configuration
INFO  2024-06-16 06:39:22,116 [shard 0: gms] raft_topology - updating topology state: finished decommissioning node 8d337728-7ec3-4376-8363-df83207c3a9b

This is why the node gets stuck here when trying to restart:

INFO  2024-06-16 06:39:39,696 [shard 0:strm] raft_group0 - finish_setup_after_join: becoming a voter in the group 0 configuration...

it can not contact the cluster because it's banned.

The test is timing based and therefore flaky.

We need to make sure that we interrupt the node before decommission finishes. @temichus and @aleksbykov you guys dealt with such problems before and unflaked many such tests, so I'm assigning it to you.