Problem: `consul-agent` service fails to start about cortx-hare

[andriytk]

closed

[andriytk]

We are not going to generate consul configuration file from cfgen anymore (see commit 96ea059). So the bootstrap script will specify explicitly the IP addresses for the agents on each node via the --bind parameter.

[vvv]

assigned to @andriy.tkachuk

[vvv]

Yes, cfgen shall be able to put the IP address into consul-conf.dhall (which will then be converted into consul-conf.json).

cc @andriy.tkachuk

[vvv]

Can we provide preferred address @ /opt/seagate/consul/consul-srv-conf.json ?

@rajanikant.chirmade Define “preferred address”.

[rajanikantchirmade]

Can we provide preferred address @ /opt/seagate/consul/consul-srv-conf.json ?

Yes it is working if bind address added to confirm.

[vagrant@cmu hare]$ sudo systemctl start consul-agent
[vagrant@cmu hare]$ sudo systemctl status consul-agent
● consul-agent.service - Consul agent for Hare
   Loaded: loaded (/data/hare/systemd/consul-agent.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-08-28 10:28:00 UTC; 4s ago
 Main PID: 6392 (consul)
   CGroup: /system.slice/consul-agent.service
           ├─6392 /usr/local/bin/consul agent -config-dir=/opt/seagate/consul -data-dir=/tmp/consul -bootstrap-expect=1 -ui
           ├─6403 bash /opt/seagate/consul/check-confd
           ├─6405 bash /opt/seagate/consul/check-confd
           ├─6406 consul kv get processes/0x7200000000000001:0x1
           └─6407 jq .state -r

Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] consul: Adding LAN server cmu (Addr: tcp/172.28.128.3:8300) (DC: dc1)
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] consul: Handled member-join event for server "cmu.dc1" in area "wan"
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [WARN] agent/proxy: running as root, will not start managed proxies
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] agent: Started DNS server 10.0.2.15:8600 (tcp)
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] agent: Started HTTP server on 10.0.2.15:8500 (tcp)
Aug 28 10:28:00 cmu consul[6392]: 2019/08/28 10:28:00 [INFO] agent: started state syncer
Aug 28 10:28:03 cmu consul[6392]: ==> Newer Consul version available: 1.6.0 (currently running: 1.5.1)
[vagrant@cmu hare]$ git diff  /opt/seagate/consul/consul-srv-conf.json 
diff --git a/consul-srv-conf.json b/consul-srv-conf.json
index 30a5ee7..62484d8 100644
--- a/consul-srv-conf.json
+++ b/consul-srv-conf.json
@@ -1,4 +1,5 @@
 {
+  "bind_addr": "172.28.128.3",
   "server" : true,
   "addresses" : {
     "grpc" : "{{GetPrivateIP}}",
[vagrant@cmu hare]$ 

[rajanikantchirmade]

@vvv

diff --git a/systemd/consul-agent.service b/systemd/consul-agent.service
index 887723e..d430fa0 100644
--- a/systemd/consul-agent.service
+++ b/systemd/consul-agent.service
@@ -5,7 +5,7 @@ After=network-online.target
 ConditionFileNotEmpty=/opt/seagate/consul/consul-srv-conf.json
 
 [Service]
-ExecStart=/usr/local/bin/consul agent -config-dir=/opt/seagate/consul -data-dir=/tmp/consul -bootstrap-expect=1 -ui
+ExecStart=/usr/local/bin/consul agent -bind=172.28.128.3 -config-dir=/opt/seagate/consul -data-dir=/tmp/consul -bootstrap-expect=1 -ui
 ExecReload=/usr/local/bin/consul reload
 KillMode=process
 Restart=on-failure

[andriytk]

Right, it's because there are several network interfaces in your vm configured and only one of them should be selected for the consul agent to work on. We should think about how to handle such configurations. Maybe specify the IP address explicitly in the consul configuration file via cfgen...

[vvv]

cc @andriy.tkachuk

[vvv]

@rajanikant.chirmade Can you show the diff please?

git diff -- systemd/consul-agent.service

[vvv]

changed the description

[vvv]

changed title from Problem: {-consul agent-} fails to start to Problem: {+consul-agent service+} fails to start

[vvv]

changed the description

[vvv]

changed the description

[vvv]

changed the description

[vvv]

changed title from Problem: consul{--agent failed-} to start to Problem: consul{+ agent fails+} to start

[rajanikantchirmade]

changed the description

[rajanikantchirmade]

changed the description

[rajanikantchirmade]

changed the description

[rajanikantchirmade]

changed the description

[rajanikantchirmade]

changed the description