Increasing RF doesn't copy mutations around properly about scylladb

[xemul]

@ptrsmrn , please, squash this patch into 6b6fc94 . After it the tablet_transition_kind::rf_change becomes full mirror of tablet_transition_kind::rebuild so I'd vote for removing the former and use only the latter throughout the code

[xemul]

That's because of rf-change skips streaming

diff --git a/service/storage_service.cc b/service/storage_service.cc
index bd504e8ffa..063236f317 100644
--- a/service/storage_service.cc
+++ b/service/storage_service.cc
@@ -5743,6 +5743,7 @@ future<> storage_service::stream_tablet(locator::global_tablet_id tablet) {
                 case locator::tablet_transition_kind::migration: return streaming::stream_reason::tablet_migration;
                 case locator::tablet_transition_kind::intranode_migration: return streaming::stream_reason::tablet_migration;
                 case locator::tablet_transition_kind::rebuild: return streaming::stream_reason::rebuild;
+                case locator::tablet_transition_kind::rf_change: return streaming::stream_reason::rebuild;
                 default:
                     throw std::runtime_error(fmt::format("stream_tablet(): Invalid tablet transition: {}", trinfo->transition));
             }
diff --git a/service/topology_coordinator.cc b/service/topology_coordinator.cc
index f95761616c..1d7cf70e49 100644
--- a/service/topology_coordinator.cc
+++ b/service/topology_coordinator.cc
@@ -1136,12 +1136,9 @@ class topology_coordinator : public endpoint_lifecycle_subscriber {
                         }
                     }
                     if (do_barrier()) {
-                        auto next_transition_stage = (trinfo.transition == locator::tablet_transition_kind::rf_change)
-                                                     ? locator::tablet_transition_stage::end_migration
-                                                     : locator::tablet_transition_stage::write_both_read_old;
-                        rtlogger.debug("Will set tablet {} stage to {}", gid, next_transition_stage);
+                        rtlogger.debug("Will set tablet {} stage to {}", gid, locator::tablet_transition_stage::write_both_read_old);
                         updates.emplace_back(get_mutation_builder()
-                            .set_stage(last_token, next_transition_stage)
+                            .set_stage(last_token, locator::tablet_transition_stage::write_both_read_old)
                             // Create session a bit earlier to avoid adding barrier
                             // to the streaming stage to create sessions on replicas.
                             .set_session(last_token, session_id(utils::UUID_gen::get_time_UUID()))

[xemul]

Now we have "extra" mutations on the target node

10:40:28.051 INFO> Found mutations for 29 key on ['c03347c9-f350-41f9-9684-48f7da9c5767', 'eed835b5-225d-4b7b-82df-74302660e266', 'eed835b5-225d-4b7b-82df-74302660e266', '5bd6a459-82ad-474b-b94f-7c06dd9b900d'] hosts, but expected only 3 o
f them

That's because streaming really generates two sstables

INFO  2024-05-23 10:40:27,025 [shard 0:strm] range_streamer - Tablet rf_change starts, nr_ranges_remaining=2
INFO  2024-05-23 10:40:27,025 [shard 0:strm] range_streamer - Tablet rf_change with [127.216.2.1, 127.216.2.3] for keyspace=test started, nodes_to_stream=2
INFO  2024-05-23 10:40:27,025 [shard 0:strm] range_streamer - Tablet rf_change with 127.216.2.1 for keyspace=test, streaming [0, 1) out of 1 ranges
INFO  2024-05-23 10:40:27,025 [shard 0:strm] stream_session - [Stream #b8dc3810-18d7-11ef-9908-464fb66455bd] Executing streaming plan for Tablet rf_change-test-index-0 with peers={127.216.2.1}, master
INFO  2024-05-23 10:40:27,025 [shard 0:strm] range_streamer - Tablet rf_change with 127.216.2.3 for keyspace=test, streaming [0, 1) out of 1 ranges
INFO  2024-05-23 10:40:27,025 [shard 0:strm] stream_session - [Stream #b8dc3811-18d7-11ef-9908-464fb66455bd] Executing streaming plan for Tablet rf_change-test-index-0 with peers={127.216.2.3}, master
INFO  2024-05-23 10:40:27,032 [shard 0:strm] sstable - SSTable with generation 3ggd_0lbf_05sc02bs1n8vbbu3st of test.test was sealed successfully (origin: ).
INFO  2024-05-23 10:40:27,032 [shard 0:strm] stream_session - [Stream #b8dc3810-18d7-11ef-9908-464fb66455bd] Write to sstable for ks=test, cf=test, estimated_partitions=0, received_partitions=128
INFO  2024-05-23 10:40:27,032 [shard 0:strm] stream_session - [Stream #b8dc3810-18d7-11ef-9908-464fb66455bd] Streaming plan for Tablet rf_change-test-index-0 succeeded, peers={127.216.2.1}, tx=0 KiB, 0.00 KiB/s, rx=26 KiB, 3420.75 KiB/s
INFO  2024-05-23 10:40:27,032 [shard 0:strm] range_streamer - Finished 1 out of 2 ranges for rebuild, finished percentage=0.5
INFO  2024-05-23 10:40:27,032 [shard 0:strm] range_streamer - Tablet rf_change with 127.216.2.1 for keyspace=test succeeded, took 0.00766701 seconds
INFO  2024-05-23 10:40:27,042 [shard 0:strm] sstable - SSTable with generation 3ggd_0lbf_07xhs2bs1n8vbbu3st of test.test was sealed successfully (origin: ).
INFO  2024-05-23 10:40:27,042 [shard 0:strm] stream_session - [Stream #b8dc3811-18d7-11ef-9908-464fb66455bd] Write to sstable for ks=test, cf=test, estimated_partitions=0, received_partitions=128
INFO  2024-05-23 10:40:27,042 [shard 0:strm] stream_session - [Stream #b8dc3811-18d7-11ef-9908-464fb66455bd] Streaming plan for Tablet rf_change-test-index-0 succeeded, peers={127.216.2.3}, tx=0 KiB, 0.00 KiB/s, rx=26 KiB, 1476.25 KiB/s
INFO  2024-05-23 10:40:27,042 [shard 0:strm] range_streamer - Finished 2 out of 2 ranges for rebuild, finished percentage=1
INFO  2024-05-23 10:40:27,042 [shard 0:strm] range_streamer - Tablet rf_change with 127.216.2.3 for keyspace=test succeeded, took 0.017892428 seconds
INFO  2024-05-23 10:40:27,042 [shard 0:strm] range_streamer - Tablet rf_change succeeded, took 0 seconds, nr_ranges_remaining=0

[xemul]

🎉 , this is how streaming works -- two sources -> two sstables. The mutations checking code would need to be adjusted

diff --git a/test/topology_custom/test_tablets.py b/test/topology_custom/test_tablets.py
index 8cb47b311b..bcb1ae9b7d 100644
--- a/test/topology_custom/test_tablets.py
+++ b/test/topology_custom/test_tablets.py
@@ -110,6 +110,8 @@ async def test_tablet_rf_change(manager: ManagerClient, direction):
     cfg = {'enable_user_defined_functions': False,
            'experimental_features': ['tablets']}
     servers = await manager.servers_add(3, config=cfg)
+    for s in servers:
+        await manager.api.disable_tablet_balancing(s.ip_addr)
 
     cql = manager.get_cql()
     res = await cql.run_async("SELECT data_center FROM system.local")
@@ -146,7 +148,7 @@ async def test_tablet_rf_change(manager: ManagerClient, direction):
     logger.info(f"Checking {rf_to} re-allocated replicas")
     await check_allocated_replica(rf_to)
 
-    fragments = { pk: [] for pk in random.sample(range(128), 17) }
+    fragments = { pk: set() for pk in random.sample(range(128), 17) }
     for s in servers:
         host_id = await manager.get_host_id(s.server_id)
         host = await wait_for_cql_and_get_hosts(cql, [s], time.time() + 30)
@@ -155,7 +157,7 @@ async def test_tablet_rf_change(manager: ManagerClient, direction):
             res = await cql.run_async(f"SELECT partition_region FROM MUTATION_FRAGMENTS(test.test) WHERE pk={k}", host=host[0])
             for fragment in res:
                 if fragment.partition_region == 0: # partition start
-                    fragments[k].append(host_id)
+                    fragments[k].add(host_id)
     logger.info("Checking fragments")
     for k in fragments:
         assert len(fragments[k]) == rf_to, f"Found mutations for {k} key on {fragments[k]} hosts, but expected only {rf_to} of them"

[ptrsmrn]

@ptrsmrn , please, squash this patch into 6b6fc94 . After it the tablet_transition_kind::rf_change becomes full mirror of tablet_transition_kind::rebuild so I'd vote for removing the former and use only the latter throughout the code

@xemul ok, squashed. WRT merging transition kinds, we can make this decision in follow-up PRs.

[xemul]

@xemul ok, squashed. WRT merging transition kinds, we can make this decision in follow-up PRs.

I doubt that. Those kinds are put into raft logs and, eventually, system tables, so newer versions of scylla will need to understand those values

[ptrsmrn]

@xemul ok, squashed. WRT merging transition kinds, we can make this decision in follow-up PRs.

I doubt that. Those kinds are put into raft logs and, eventually, system tables, so newer versions of scylla will need to understand those values

Fair enough, I removed the rf_change and replaced it rebuild. What I had in mind was that no one will use this code in production, and we can change whatever we want if the code has not been released (at least I think so).

[ptrsmrn]

@xemul can we close this issue? The CI for #16723 passes.