Hi,
Can you make a documentation about how to recover the Consul cluster after the leader is stopped/damaged? Something happened to my Consul cluster because I can't start it now.
Now, if I launch the leader my log is:
2020-03-19T23:18:04.575Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:06.755Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:06.787Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:13.924Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:18:13.956Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:00:47.139Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:00:47.395Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:16.995Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:00:54.311Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:17.027Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:00:54.565Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:00:57.379Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:00:57.635Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:00:57.908Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:00:58.177Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:18:17.575Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:00.451Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:01:00.707Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:07.621Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:07.877Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:10.691Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:17.767Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:10.947Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:11.443Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:11.790Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:13.763Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:01:14.019Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:21.017Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:20.067Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:21.189Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:24.067Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:20.099Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:01:24.259Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:24.770Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:25.133Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:27.139Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:27.323Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:27.331Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:27.354Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:34.310Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:30.403Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:34.583Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:37.379Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:01:37.635Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:30.435Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:31.291Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:38.336Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:38.361Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:38.713Z [ERROR] agent.server: failed to reconcile member: member="{a71217dd3a4b 10.0.7.245 8301 map[acls:0 bootstrap:1 build:1.7.2:9ea1a204 dc:dc1 id:46c3d011-333a-6d9e-be06-d3d01b93cfbe port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="error removing server with duplicate ID "46c3d011-333a-6d9e-be06-d3d01b93cfbe": Need at least one voter in configuration: {[{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300} {Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}]}", 2020-03-21T11:01:40.451Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:31.321Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:40.707Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:33.475Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:47.621Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:47.878Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:50.691Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:01:50.947Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:01:51.682Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:18:33.507Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:01:51.691Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:40.644Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:01:53.763Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:40.676Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:01:54.019Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:00.933Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:01.189Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:04.003Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:04.259Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:43.715Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:04.786Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:04.999Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:07.075Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:07.331Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:14.324Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:14.571Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:18:43.751Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:17.379Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:43.919Z [ERROR] agent.server: failed to reconcile member: member="{ff5f48876b6b 10.0.7.244 8301 map[acls:0 bootstrap:1 build:1.7.2:9ea1a204 dc:dc1 id:46c3d011-333a-6d9e-be06-d3d01b93cfbe port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="error removing server with duplicate ID "46c3d011-333a-6d9e-be06-d3d01b93cfbe": Need at least one voter in configuration: {[{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300} {Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}]}", 2020-03-21T11:02:17.635Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:18.020Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:18.269Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:20.451Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:44.437Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:20.711Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:27.624Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:27.877Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:30.691Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:44.544Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:30.947Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:46.787Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:31.584Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:31.887Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:33.763Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:46.823Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:34.019Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:18:54.026Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:38.714Z [ERROR] agent.server: failed to reconcile member: member="{a71217dd3a4b 10.0.7.245 8301 map[acls:0 bootstrap:1 build:1.7.2:9ea1a204 dc:dc1 id:46c3d011-333a-6d9e-be06-d3d01b93cfbe port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="error removing server with duplicate ID "46c3d011-333a-6d9e-be06-d3d01b93cfbe": Need at least one voter in configuration: {[{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300} {Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}]}", 2020-03-19T23:18:54.043Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:41.032Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:41.251Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:44.099Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:44.323Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:44.567Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:57.091Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:44.816Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:47.171Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:47.395Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:54.340Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:18:57.123Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:02:54.565Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:02:57.411Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:57.588Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:57.635Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:02:58.134Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:02:58.568Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:00.483Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:18:57.761Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:00.707Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:00.163Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:03:07.730Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:03:07.966Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:10.787Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:00.195Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:03:11.043Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:03:11.625Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:03:11.788Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:13.859Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:03:14.115Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:07.332Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:21.029Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:03:21.285Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:24.099Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:03:24.355Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-21T11:03:24.760Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-21T11:03:24.997Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-21T11:03:27.171Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.245:0->10.0.7.219:8300: connect: no route to host", 2020-03-21T11:03:27.427Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.245:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:07.364Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:10.407Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:10.435Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:11.312Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:11.364Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:13.475Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:13.507Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:20.648Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:20.676Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:23.715Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:23.747Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:24.333Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:24.350Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:26.787Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:26.819Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:34.015Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:34.051Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:37.091Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:37.123Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:37.770Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:37.991Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:40.167Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:40.195Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:43.919Z [ERROR] agent.server: failed to reconcile member: member="{ff5f48876b6b 10.0.7.244 8301 map[acls:0 bootstrap:1 build:1.7.2:9ea1a204 dc:dc1 id:46c3d011-333a-6d9e-be06-d3d01b93cfbe port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="error removing server with duplicate ID "46c3d011-333a-6d9e-be06-d3d01b93cfbe": Need at least one voter in configuration: {[{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300} {Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}]}", 2020-03-19T23:19:47.332Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:47.364Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:50.403Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:50.435Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:19:51.213Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:19:51.343Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:19:53.479Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:19:53.507Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:20:00.645Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:20:00.676Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:20:03.715Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:20:03.747Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:20:04.437Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:20:04.630Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:20:06.787Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:20:06.819Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:20:14.012Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:20:14.078Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:20:14.228Z [INFO] agent: Caught: signal=terminated, 2020-03-19T23:20:14.228Z [INFO] agent: Gracefully shutting down agent..., 2020-03-19T23:20:14.228Z [INFO] agent.server: server starting leave, 2020-03-19T23:20:14.229Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: ff5f48876b6b.dc1 10.0.7.244, 2020-03-19T23:20:14.229Z [INFO] agent.server: Handled event for server in area: event=member-leave server=ff5f48876b6b.dc1 area=wan, 2020-03-19T23:20:14.229Z [INFO] agent.server.router.manager: shutting down, 2020-03-19T23:20:17.091Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 4304cc1e-9b82-6dfa-a6a7-6783efb8a605 10.0.7.218:8300}" error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:20:17.155Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 6a68833f-4b17-d99d-769b-b94bd63bef90 10.0.7.219:8300}" error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:20:17.231Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: ff5f48876b6b 10.0.7.244, 2020-03-19T23:20:17.232Z [INFO] agent.server: Removing LAN server: server="ff5f48876b6b (Addr: tcp/10.0.7.244:8300) (DC: dc1)", 2020-03-19T23:20:17.233Z [WARN] agent.server: deregistering self should be done by follower: name=ff5f48876b6b, 2020-03-19T23:20:17.731Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4304cc1e-9b82-6dfa-a6a7-6783efb8a605 fallback=10.0.7.218:8300 error="Could not find address for server id 4304cc1e-9b82-6dfa-a6a7-6783efb8a605", 2020-03-19T23:20:17.846Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=6a68833f-4b17-d99d-769b-b94bd63bef90 fallback=10.0.7.219:8300 error="Could not find address for server id 6a68833f-4b17-d99d-769b-b94bd63bef90", 2020-03-19T23:20:18.000Z [ERROR] agent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found", 2020-03-19T23:20:20.000Z [ERROR] agent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found", 2020-03-19T23:20:20.163Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.218:8300 error="dial tcp 10.0.7.244:0->10.0.7.218:8300: connect: no route to host", 2020-03-19T23:20:20.227Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.0.7.219:8300 error="dial tcp 10.0.7.244:0->10.0.7.219:8300: connect: no route to host", 2020-03-19T23:20:20.233Z [INFO] agent.server: Waiting to drain RPC traffic: drain_time=5s, 2020-03-19T23:20:22.000Z [ERROR] agent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found", 2020-03-19T23:20:24.000Z [ERROR] agent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found", 2020-03-19T23:20:24.000Z [ERROR] agent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found",
I have not launched the replicas, because after I launched them, the whole cluster breaks down and nothing worked. Now the leader is full with errors in logs, but the system looks "half-functional". Right now, the certifications on my sites are from the Traefik not from Letsencrypt, so they are not valid.
Thank you!