topology_custom/test_change_rpc_address.py::test_change_rpc_address takes longer than 50 to stop a server about scylladb

[tchaikov]

in scylla-744's log, we have

INFO  2024-01-16 20:18:31,784 [shard 0:main] init - Shutting down database
INFO  2024-01-16 20:18:31,933 [shard 1:main] large_data - Waiting for 0 background handlers
INFO  2024-01-16 20:18:31,933 [shard 1:main] database - Shutting down commitlog
INFO  2024-01-16 20:18:31,933 [shard 1:main] database - Shutting down commitlog complete
INFO  2024-01-16 20:18:31,933 [shard 1:main] database - Shutting down system dirty memory manager
INFO  2024-01-16 20:18:31,933 [shard 1:main] database - Shutting down dirty memory manager
INFO  2024-01-16 20:18:31,934 [shard 1:main] database - Shutting down memtable controller
INFO  2024-01-16 20:18:31,934 [shard 1:main] database - Closing user sstables manager
INFO  2024-01-16 20:18:31,934 [shard 0:main] large_data - Waiting for 0 background handlers
INFO  2024-01-16 20:18:31,934 [shard 1:main] database - Closing system sstables manager
INFO  2024-01-16 20:18:31,934 [shard 1:main] database - Stopping querier cache
INFO  2024-01-16 20:18:31,934 [shard 0:main] database - Shutting down commitlog
INFO  2024-01-16 20:18:31,934 [shard 1:main] database - Stopping concurrency semaphores
INFO  2024-01-16 20:18:31,934 [shard 0:main] database - Shutting down commitlog complete
INFO  2024-01-16 20:18:31,934 [shard 0:main] database - Shutting down schema commitlog
INFO  2024-01-16 20:18:31,934 [shard 0:main] database - Shutting down schema commitlog complete
INFO  2024-01-16 20:18:31,934 [shard 1:main] database - Joining memtable update action
INFO  2024-01-16 20:18:31,934 [shard 0:main] database - Shutting down system dirty memory manager
INFO  2024-01-16 20:18:31,934 [shard 0:main] database - Shutting down dirty memory manager
INFO  2024-01-16 20:18:31,935 [shard 0:main] database - Shutting down memtable controller
INFO  2024-01-16 20:18:31,935 [shard 0:main] database - Closing user sstables manager
INFO  2024-01-16 20:18:31,935 [shard 0:main] database - Closing system sstables manager
INFO  2024-01-16 20:18:31,935 [shard 0:main] database - Stopping querier cache
INFO  2024-01-16 20:18:31,935 [shard 0:main] database - Stopping concurrency semaphores
INFO  2024-01-16 20:18:31,936 [shard 0:main] database - Joining memtable update action
INFO  2024-01-16 20:18:31,936 [shard 0:main] init - Shutting down database was successful
INFO  2024-01-16 20:18:31,936 [shard 0:main] init - Shutting down wasm
INFO  2024-01-16 20:18:31,937 [shard 0:main] init - Shutting down wasm was successful
INFO  2024-01-16 20:18:31,937 [shard 0:main] init - Shutting down sstables storage manager
INFO  2024-01-16 20:18:31,937 [shard 0:main] init - Shutting down sstables storage manager was successful
INFO  2024-01-16 20:18:31,937 [shard 0:main] init - Shutting down compaction_manager
INFO  2024-01-16 20:18:31,938 [shard 0:main] task_manager - Stopping module compaction
INFO  2024-01-16 20:18:31,938 [shard 0:main] task_manager - Unregistered module compaction
INFO  2024-01-16 20:18:31,938 [shard 1:main] task_manager - Stopping module compaction
INFO  2024-01-16 20:18:31,938 [shard 1:main] task_manager - Unregistered module compaction
INFO  2024-01-16 20:18:31,939 [shard 0:main] init - Shutting down compaction_manager was successful
INFO  2024-01-16 20:18:31,939 [shard 0:main] init - Shutting down task_manager
INFO  2024-01-16 20:18:31,939 [shard 0:main] init - Shutting down task_manager was successful
INFO  2024-01-16 20:18:31,939 [shard 0:main] init - Shutting down service_memory_limiter
INFO  2024-01-16 20:18:31,939 [shard 0:main] init - Shutting down service_memory_limiter was successful
INFO  2024-01-16 20:18:31,939 [shard 0:main] init - Shutting down sst_dir_semaphore
INFO  2024-01-16 20:18:31,940 [shard 0:main] init - Shutting down sst_dir_semaphore was successful
INFO  2024-01-16 20:18:31,940 [shard 0:main] init - Shutting down migration manager notifier
INFO  2024-01-16 20:18:31,940 [shard 0:main] init - Shutting down migration manager notifier was successful
INFO  2024-01-16 20:18:31,941 [shard 0:main] init - Shutting down prometheus API server
INFO  2024-01-16 20:18:31,943 [shard 0:main] init - Shutting down prometheus API server was successful
INFO  2024-01-16 20:18:31,943 [shard 0:main] init - Shutting down API server
INFO  2024-01-16 20:18:31,945 [shard 0:main] init - Shutting down API server was successful
INFO  2024-01-16 20:18:31,945 [shard 0:main] init - Shutting down sighup
INFO  2024-01-16 20:18:31,948 [shard 0:main] init - Shutting down sighup was successful
INFO  2024-01-16 20:18:31,948 [shard 0:main] init - Shutting down configurables
INFO  2024-01-16 20:18:31,948 [shard 0:main] init - Shutting down configurables was successful
INFO  2024-01-16 20:18:31,949 [shard 0:main] init - Scylla version 5.5.0~dev-0.20240116.72635cc6623b shutdown complete.
------ Ending test debug/topology_custom.test_change_rpc_address.2::test_change_rpc_address ------

while in scylla-745, we have

INFO  2024-01-16 20:18:57,971 [shard 0:main] init - Shutting down messaging service was successful
INFO  2024-01-16 20:18:57,971 [shard 0:main] init - Shutting down service level controller
INFO  2024-01-16 20:18:57,972 [shard 0:main] init - Shutting down service level controller was successful
INFO  2024-01-16 20:18:57,972 [shard 0:main] init - Shutting down tracing instance
INFO  2024-01-16 20:18:57,973 [shard 0:main] init - Shutting down tracing instance was successful
INFO  2024-01-16 20:18:57,973 [shard 0:main] init - Shutting down storage proxy API
INFO  2024-01-16 20:18:57,974 [shard 0:main] init - Shutting down storage proxy API was successful
INFO  2024-01-16 20:18:57,974 [shard 0:main] init - Shutting down database
------ Ending test debug/topology_custom.test_change_rpc_address.2::test_change_rpc_address ------

so, scylla-745 was stuck when "Shutting down database".