5.4.6/dtest-debug test_remove_field_with_cdc[Single_cluster] failure: Error from server: code=0000 [Server error] message="std::exception about scylladb

[mykaul]

cdc_remove_node3.log
cdc_remove_node2.log
cdc_remove_node1.log
cdc_remove_dtest-gw2.log

[mykaul]

Past failure of a different variant of the test (add): https://jenkins.scylladb.com/job/scylla-5.4/job/dtest-debug/17/artifact/logs-full.debug.015/

[mykaul]

dtest-debug, 5.2.18(!), similar CDC test failure (different test)!:

15:23:48,540 555     dtest_setup                    DEBUG    dtest_setup.py      :60   | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Allocated cluster ID 17: /jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap
15:23:48,549 555     cdc_test                       DEBUG    cdc_test.py         :188  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: cluster_config=ClusterConfig(size=[3], replication="{'class': 'SimpleStrategy', 'replication_factor': 3}")
15:23:48,550 555     cdc_test                       DEBUG    cdc_test.py         :412  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Setup a cluster: size=[3] replication={'class': 'SimpleStrategy', 'replication_factor': 3} with_preimage=True
15:23:48,550 555     cdc_test                       DEBUG    cdc_test.py         :63   | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Starting node 1
15:23:49,638 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: start_nodes: no_wait=False wait_for_binary_proto=True wait_other_notice=False force_wait_for_cluster_start=True
15:23:49,645 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: node1: Starting scylla: args=['/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node1/bin/scylla', '--options-file', '/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node1/conf/scylla.yaml', '--log-to-stdout', '1', '--api-address', '127.0.17.1', '--collectd-hostname', 'd38dbdb9d254.node1', '--smp', '2', '--memory', '1024M', '--abort-on-seastar-bad-alloc', '--abort-on-lsa-bad-alloc', '1', '--abort-on-internal-error', '1', '--developer-mode', 'true', '--default-log-level', 'info', '--blocked-reactor-notify-ms', '5000', '--collectd', '0', '--overprovisioned', '--prometheus-address', '127.0.17.1', '--unsafe-bypass-fsync', '1', '--kernel-page-cache', '1', '--commitlog-use-o-dsync', '0', '--max-networking-io-control-blocks', '1000'] wait_other_notice=False wait_for_binary_proto=True
15:24:24,700 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: node1: Starting scylla-jmx: args=['/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node1/bin/symlinks/scylla-jmx', '-Dapiaddress=127.0.17.1', '-Djavax.management.builder.initial=com.scylladb.jmx.utils.APIBuilder', '-Djava.rmi.server.hostname=127.0.17.1', '-Dcom.sun.management.jmxremote', '-Dcom.sun.management.jmxremote.host=127.0.17.1', '-Dcom.sun.management.jmxremote.port=7199', '-Dcom.sun.management.jmxremote.rmi.port=7199', '-Dcom.sun.management.jmxremote.local.only=false', '-Xmx256m', '-XX:+UseSerialGC', '-Dcom.sun.management.jmxremote.authenticate=false', '-Dcom.sun.management.jmxremote.ssl=false', '-jar', '/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node1/bin/scylla-jmx-1.0.jar']
15:24:25,056 555     cdc_test                       DEBUG    cdc_test.py         :74   | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Starting node 2
15:24:25,136 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: node2: Starting scylla: args=['/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node2/bin/scylla', '--options-file', '/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node2/conf/scylla.yaml', '--log-to-stdout', '1', '--api-address', '127.0.17.2', '--collectd-hostname', 'd38dbdb9d254.node2', '--smp', '2', '--memory', '1024M', '--abort-on-seastar-bad-alloc', '--abort-on-lsa-bad-alloc', '1', '--abort-on-internal-error', '1', '--developer-mode', 'true', '--default-log-level', 'info', '--blocked-reactor-notify-ms', '5000', '--collectd', '0', '--overprovisioned', '--prometheus-address', '127.0.17.2', '--unsafe-bypass-fsync', '1', '--kernel-page-cache', '1', '--commitlog-use-o-dsync', '0', '--max-networking-io-control-blocks', '1000'] wait_other_notice=False wait_for_binary_proto=True
15:25:35,584 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: node2: Starting scylla-jmx: args=['/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node2/bin/symlinks/scylla-jmx', '-Dapiaddress=127.0.17.2', '-Djavax.management.builder.initial=com.scylladb.jmx.utils.APIBuilder', '-Djava.rmi.server.hostname=127.0.17.2', '-Dcom.sun.management.jmxremote', '-Dcom.sun.management.jmxremote.host=127.0.17.2', '-Dcom.sun.management.jmxremote.port=7199', '-Dcom.sun.management.jmxremote.rmi.port=7199', '-Dcom.sun.management.jmxremote.local.only=false', '-Xmx256m', '-XX:+UseSerialGC', '-Dcom.sun.management.jmxremote.authenticate=false', '-Dcom.sun.management.jmxremote.ssl=false', '-jar', '/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node2/bin/scylla-jmx-1.0.jar']
15:25:35,936 555     cdc_test                       DEBUG    cdc_test.py         :74   | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Starting node 3
15:25:35,983 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: node3: Starting scylla: args=['/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node3/bin/scylla', '--options-file', '/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node3/conf/scylla.yaml', '--log-to-stdout', '1', '--api-address', '127.0.17.3', '--collectd-hostname', 'd38dbdb9d254.node3', '--smp', '2', '--memory', '1024M', '--abort-on-seastar-bad-alloc', '--abort-on-lsa-bad-alloc', '1', '--abort-on-internal-error', '1', '--developer-mode', 'true', '--default-log-level', 'info', '--blocked-reactor-notify-ms', '5000', '--collectd', '0', '--overprovisioned', '--prometheus-address', '127.0.17.3', '--unsafe-bypass-fsync', '1', '--kernel-page-cache', '1', '--commitlog-use-o-dsync', '0', '--max-networking-io-control-blocks', '1000'] wait_other_notice=False wait_for_binary_proto=True
15:26:58,613 555     ccm                            DEBUG    cluster.py          :711  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: node3: Starting scylla-jmx: args=['/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node3/bin/symlinks/scylla-jmx', '-Dapiaddress=127.0.17.3', '-Djavax.management.builder.initial=com.scylladb.jmx.utils.APIBuilder', '-Djava.rmi.server.hostname=127.0.17.3', '-Dcom.sun.management.jmxremote', '-Dcom.sun.management.jmxremote.host=127.0.17.3', '-Dcom.sun.management.jmxremote.port=7199', '-Dcom.sun.management.jmxremote.rmi.port=7199', '-Dcom.sun.management.jmxremote.local.only=false', '-Xmx256m', '-XX:+UseSerialGC', '-Dcom.sun.management.jmxremote.authenticate=false', '-Dcom.sun.management.jmxremote.ssl=false', '-jar', '/jenkins/workspace/scylla-5.2/dtest-debug/scylla/.dtest/dtest-xrfzvaap/test/node3/bin/scylla-jmx-1.0.jar']
15:26:58,984 555     cassandra.policies             INFO     policies.py         :289  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.17.1:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
15:26:58,984 555     cassandra.cluster              INFO     dtest_setup.py      :416  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: New Cassandra host <Host: 127.0.17.3:9042 datacenter1> discovered
15:26:58,985 555     cassandra.cluster              INFO     dtest_setup.py      :416  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: New Cassandra host <Host: 127.0.17.2:9042 datacenter1> discovered
15:26:59,264 555     cdc_test                       DEBUG    cdc_test.py         :418  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Wait for the last generation to become active
15:26:59,272 555     cdc_test                       DEBUG    cdc_test.py         :115  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Current generation timestamp: 2024-04-19 15:26:26.504000
15:27:00,272 555     cdc_test                       DEBUG    cdc_test.py         :137  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Token ring length: 768
15:27:00,272 555     dtest_class                    DEBUG    dtest_class.py      :164  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: (Waiting until metadata is updated (1.0002694129943848 s)
15:27:00,272 555     cdc_test                       DEBUG    cdc_test.py         :723  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Checking invariants on generation
15:27:00,272 555     cdc_test                       DEBUG    cdc_test.py         :724  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Checking if generation token ranges refine vnodes
15:27:00,333 555     cdc_test                       DEBUG    cdc_test.py         :730  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Checking that all vnodes have a stream
15:27:00,336 555     cdc_test                       DEBUG    cdc_test.py         :425  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Create a table with CDC enabled, and start writing to it
15:27:00,336 555     cdc_test                       INFO     cdc_test.py         :561  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: {'class': 'SimpleStrategy', 'replication_factor': 3}
15:27:00,352 555     cassandra.policies             INFO     policies.py         :289  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.17.1:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
15:27:00,352 555     cassandra.cluster              INFO     dtest_setup.py      :416  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: New Cassandra host <Host: 127.0.17.3:9042 datacenter1> discovered
15:27:00,353 555     cassandra.cluster              INFO     dtest_setup.py      :416  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: New Cassandra host <Host: 127.0.17.2:9042 datacenter1> discovered
15:27:18,830 555     cdc_test                       DEBUG    cdc_test.py         :431  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Alter schema: ALTER TABLE ks.cf ALTER b TYPE blob
15:27:20,734 555     cdc_test                       DEBUG    cdc_test.py         :434  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: Write for 15 more seconds, and stop writing
15:27:35,854 555     errors                         ERROR    conftest.py         :203  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: test failed: 
self = <cdc_test.TestCdc object at 0x7f46355f16d0>
request = <FixtureRequest for <Function test_change_field_type_with_cdc_and_preimage[Single_cluster]>>
cluster_config = ClusterConfig(size=[3], replication="{'class': 'SimpleStrategy', 'replication_factor': 3}")

    def test_change_field_type_with_cdc_and_preimage(self, request, cluster_config):
>       self.schema_change_template(request, "ALTER TABLE ks.cf ALTER b TYPE blob",
                                    cluster_size=cluster_config.size, replication=cluster_config.replication, with_preimage=True)

cdc_test.py:453: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cdc_test.py:436: in schema_change_template
    write_count = finish_writing()
cdc_test.py:606: in finisher
    (confirmed, unconfirmed) = fut.result()
/usr/lib64/python3.11/concurrent/futures/_base.py:449: in result
    return self.__get_result()
/usr/lib64/python3.11/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
/usr/lib64/python3.11/concurrent/futures/thread.py:58: in run
    result = self.fn(*self.args, **self.kwargs)
cdc_test.py:583: in run_writes
    session.execute(stmt, (i, str(worker_id)))
cassandra/cluster.py:2611: in cassandra.cluster.Session.execute
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   cassandra.protocol.ServerError: <Error from server: code=0000 [Server error] message="std::exception">

cassandra/cluster.py:4829: ServerError
15:27:35,854 555     errors                         ERROR    conftest.py         :203  | test_change_field_type_with_cdc_and_preimage[Single_cluster]: test failed: 
self = <cdc_test.TestCdc object at 0x7f46355f16d0>
request = <FixtureRequest for <Function test_change_field_type_with_cdc_and_preimage[Single_cluster]>>
cluster_config = ClusterConfig(size=[3], replication="{'class': 'SimpleStrategy', 'replication_factor': 3}")

    def test_change_field_type_with_cdc_and_preimage(self, request, cluster_config):
>       self.schema_change_template(request, "ALTER TABLE ks.cf ALTER b TYPE blob",
                                    cluster_size=cluster_config.size, replication=cluster_config.replication, with_preimage=True)

It's unclear to me why we always get the server error exception.