Fails to start under VMWare Fusion about kubernetes-vagrant-coreos-cluster

[pires]

Apr 10 06:14:22 master wget[1024]: wget: unable to resolve host address 'storage.googleapis.com'

So, no internet access on that VM?

[rubenv]

It does have it. If I ssh into it and restart setup-network-environment.service, it runs wget properly.

[rubenv]

core@master ~ $ systemctl status network-online.target
● network-online.target - Network is Online
   Loaded: loaded (/usr/lib64/systemd/system/network-online.target; static; vendor preset: disabled)
   Active: active since Fri 2015-04-10 13:21:38 UTC; 14s ago
     Docs: man:systemd.special(7)
           http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Apr 10 13:21:38 master systemd[1]: Reached target Network is Online.
Apr 10 13:21:38 master systemd[1]: Starting Network is Online.
core@master ~ $ systemctl status setup-network-environment.service
● setup-network-environment.service - Setup Network Environment
   Loaded: loaded (/etc/systemd/system/setup-network-environment.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2015-04-10 13:21:38 UTC; 24s ago
     Docs: https://github.com/kelseyhightower/setup-network-environment
  Process: 1018 ExecStartPre=/usr/bin/wget -N -P /opt/bin https://storage.googleapis.com/k8s/setup-network-environment (code=exited, status=4)
  Process: 1016 ExecStartPre=/usr/bin/mkdir -p /opt/bin (code=exited, status=0/SUCCESS)

Apr 10 13:21:38 master systemd[1]: Starting Setup Network Environment...
Apr 10 13:21:38 master wget[1018]: --2015-04-10 13:21:38--  https://storage.googleapis.com/k8s/setup-network-environment
Apr 10 13:21:38 master wget[1018]: Resolving storage.googleapis.com... failed: Temporary failure in name resolution.
Apr 10 13:21:38 master wget[1018]: wget: unable to resolve host address 'storage.googleapis.com'
Apr 10 13:21:38 master systemd[1]: setup-network-environment.service: control process exited, code=exited status=4
Apr 10 13:21:38 master systemd[1]: Failed to start Setup Network Environment.
Apr 10 13:21:38 master systemd[1]: Unit setup-network-environment.service entered failed state.
Apr 10 13:21:38 master systemd[1]: setup-network-environment.service failed.
core@master ~ $ ping google.be
PING google.be (64.15.124.150) 56(84) bytes of data.
64 bytes from cache.google.com (64.15.124.150): icmp_seq=1 ttl=128 time=9.15 ms
64 bytes from cache.google.com (64.15.124.150): icmp_seq=2 ttl=128 time=11.0 ms
^C
--- google.be ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 9.158/10.099/11.041/0.946 ms

Perhaps network-online.target is not a strong-enough guarantee that the web is reachable?

[rubenv]

core@master ~ $ sudo systemctl restart setup-network-environment
core@master ~ $ systemctl status setup-network-environment.service
● setup-network-environment.service - Setup Network Environment
   Loaded: loaded (/etc/systemd/system/setup-network-environment.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2015-04-10 13:25:12 UTC; 1s ago
     Docs: https://github.com/kelseyhightower/setup-network-environment
  Process: 1325 ExecStart=/opt/bin/setup-network-environment (code=exited, status=0/SUCCESS)
  Process: 1324 ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment (code=exited, status=0/SUCCESS)
  Process: 1322 ExecStartPre=/usr/bin/wget -N -P /opt/bin https://storage.googleapis.com/k8s/setup-network-environment (code=exited, status=0/SUCCESS)
  Process: 1319 ExecStartPre=/usr/bin/mkdir -p /opt/bin (code=exited, status=0/SUCCESS)
 Main PID: 1325 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/setup-network-environment.service

Apr 10 13:25:12 master wget[1322]: 2200K .......... .......... .......... .......... .......... 87% 14.3M 0s
Apr 10 13:25:12 master wget[1322]: 2250K .......... .......... .......... .......... .......... 89% 5.96M 0s
Apr 10 13:25:12 master wget[1322]: 2300K .......... .......... .......... .......... .......... 91% 4.82M 0s
Apr 10 13:25:12 master wget[1322]: 2350K .......... .......... .......... .......... .......... 93% 6.97M 0s
Apr 10 13:25:12 master wget[1322]: 2400K .......... .......... .......... .......... .......... 95% 3.70M 0s
Apr 10 13:25:12 master wget[1322]: 2450K .......... .......... .......... .......... .......... 97% 5.58M 0s
Apr 10 13:25:12 master wget[1322]: 2500K .......... .......... .......... .......... .......... 99% 7.41M 0s
Apr 10 13:25:12 master wget[1322]: 2550K ........                                              100%  131M=0.6s
Apr 10 13:25:12 master wget[1322]: 2015-04-10 13:25:12 (4.44 MB/s) - '/opt/bin/setup-network-environment' saved [2620224/2620224]
Apr 10 13:25:12 master systemd[1]: Started Setup Network Environment.
core@master ~ $ 

Sadly, I have no idea how to trigger the remaining bootup after pushing it through manually.

[pires]

setup-network-environment is not related to VM networkingconfiguration, but rather some CoreOS environment stuff - otherwise, how could you download /opt/bin https://storage.googleapis.com/k8s/setup-network-environment? This must be something with VMware provisioning.

[pires]

@AntonioMeireles care to join the discussion?

[AntonioMeireles]

humm... (please do note that i have a way to reproduce this, so bellow just shots in the dark)

at sight it seems that for some reason setup-network-environement.service is just run too soon.

@rubenv can you post please here the contents of /etc/network-environment both when things go wrong and after the manual restart of setup-network-environement.service that induces the expected behaviour ?

[rubenv]

Upon initial boot, there is no /etc/network-environment. After restarting the service:

LO_IPV4=127.0.0.1
ENS33_IPV4=192.168.247.136
DEFAULT_IPV4=192.168.247.136
ENS34_IPV4=172.17.8.173

[AntonioMeireles]

@rubenv by parts ...

a) destroy existing VMs (vagrant destroy -f)
b) recreate everything
c) vssh master
wait for things to fail.

• systemctl status setup-network-environement<< paste results.
• journalctl -u setup-network-environement --no-pager --lines=500 << paste results
• systemctl list-units << paste ...
• cat /etc/network-environment << paste results (may or not be empty depends on above)

d)
- systemctl restart setup-network-environement
- and after all the output again of the command above in c) (by same sequence)

[rubenv]

Here you go:

Initially: https://gist.githubusercontent.com/rubenv/caade42ba89e9f64945a/raw/b89844e5f604665530bb13adc91c13b1db05b18e/gistfile1.txt

Boot it: https://gist.githubusercontent.com/rubenv/96759eedca6cfe48f2c1/raw/8bbd664ab8e3c3e786c1b6c37e043cedc4221fe6/gistfile1.txt

Status: https://gist.githubusercontent.com/rubenv/a6e798e5d8ca518439a3/raw/d91609b7c36a721e7177d561bfa44f011c0e48bf/gistfile1.txt
Note the failed unit at login.

For reference, ip a: https://gist.githubusercontent.com/rubenv/335c2ad3cc11a6c185ea/raw/4a35485ed317af5292c98022e3002feeff75a12a/gistfile1.txt

Journal: https://gist.githubusercontent.com/rubenv/ff65f88dc3f650e47461/raw/0edb00a479a91a112e3c16a7661d8d9478dff904/gistfile1.txt

Units: https://gist.githubusercontent.com/rubenv/d7eb37804cd5a70cfc08/raw/5a50cd685a8549d405a5ef1d670fdb4fbe0b9029/gistfile1.txt

At this point:

core@master ~ $ cat /etc/network-environment
cat: /etc/network-environment: No such file or directory

---

Restarting: sudo systemctl restart setup-network-environment

Status: https://gist.githubusercontent.com/rubenv/0363b84abe66ec782d8e/raw/e8c642468c16fc74203fe168c6efabf23569d44c/gistfile1.txt

Journal: https://gist.githubusercontent.com/rubenv/5ab29049b99729b8acbf/raw/170755f6608b9c21436f2c681c453a0e10162d15/gistfile1.txt

Units: https://gist.githubusercontent.com/rubenv/d26acb1d83fd0b284d5e/raw/b2450d9f5f3475b3812f5b089f2640d11af9d1af/gistfile1.txt

At this point we get:

core@master ~ $ cat /etc/network-environment 
LO_IPV4=127.0.0.1
ENS33_IPV4=192.168.247.135
DEFAULT_IPV4=192.168.247.135
ENS34_IPV4=172.17.8.176

---

And for completeness, the full Journal: https://gist.githubusercontent.com/rubenv/ddde2f962556b0868243/raw/5a9c5793c877d700968ab3035341f62d539d30f0/gistfile1.txt

And the output of dmesg: https://gist.githubusercontent.com/rubenv/ced75057ae46fc61d358/raw/e316baeb67d90a75abd2e59ea1158932c291e8eb/gistfile1.txt

Phew :-). Hope this is of any help!

[AntonioMeireles]

ah, for some reason the system is taking too long to get dns setup. dunno yet why.

[pires]

Can we just force DNS set-up? Something like echo 8.8.8.8 > /etc/resolv.conf?

[pires]

@rubenv care to try latest master? I think #80 may have fixed this, just need to specify FLANNEL_IF=enXX where XX is the interface name provisioned by Fusion. Kudos to @ericgray for that.

[rubenv]

I'm sorry, I haven't had the time yet to test this. Sadly my VMWare Fusion + Vagrant is also acting up (Hashicorp is looking into it). Will have a look at it, at some point.