raft topology: documentation: a node which failed to bootstrap (join the cluster) is permanently banned about scylladb

[kbr-scylla]

As part of this information, we can instruct the user what to do --- i.e. the same as we do in tests --- clear the node's work directory before restarting (so we basically start a completely new node). It can still use the same IP though.

[kbr-scylla]

cc @annastuchlik @tzach

[tzach]

we ban the node (using its host ID) if it fails to join the cluster.

• Why can this might happen?
• How can the user identify this issue?
• How can the user fix this issuer (change host id? how?)

[kbr-scylla]

Why can this might happen?

Any failed bootstrap. For example: the node crashed, or user killed/shutdown it while it was bootstrapping, or there was a network partition that caused failure in streaming/repair phase.

How can the user identify this issue?

The booting node will fail to boot and shutdown. Or it will hang, and topology coordinator logs will show that it decided to abort the procedure. Node will be marked as LEFT in system.topology

How can the user fix this issuer (change host id? how?)

Purge the node (remove workdir). The user can join a new node (if they want they can use the same machine and IP, but it will be a new node due to fresh workdir)

[tzach]

Node will be marked as LEFT in system.topology

SELECT * from system.topology, any node in status LEFT can not rejoin the clutser.
But how do I know the node hostid to correlate the two?

Purge the node (remove workdir).

You mean this procedure?
https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/safely-removing-joining-node.html

[kbr-scylla]

But how do I know the node hostid to correlate the two?

From joining node's logs, it prints Setting local host id to X somewhere at the beginning of startup, and there is also the /storage_service/hostid/local REST endpoint

You mean this procedure?

yes